|
| 1 | +# Kubernetes - enviromments |
| 2 | + |
| 3 | +As already mentioned `CTFv1` creates `k8s` environments programmatically from existing building blocks that currently include: |
| 4 | +* `anvil` |
| 5 | +* `blockscout` (cdk8s) |
| 6 | +* `chainlink node` |
| 7 | +* `geth` |
| 8 | +* `goc` (cdk8s) |
| 9 | +* `grafana` |
| 10 | +* `influxdb` |
| 11 | +* `kafka` |
| 12 | +* `mock-adapter` |
| 13 | +* `mockserver` |
| 14 | +* `reorg controller` |
| 15 | +* `schema registry` |
| 16 | +* `solana validator` |
| 17 | +* `starknet validator` |
| 18 | +* `wiremock` |
| 19 | + |
| 20 | +Unless marked otherwise, they are all based on `Helm` charts. |
| 21 | + |
| 22 | +> [!NOTE] |
| 23 | +> Creation of new environment and modification of existing ones is explained in detail [here](../k8s/TUTORIAL.md), so we won't repeat it here, |
| 24 | +> but instead focus on practical example of creating a new `k8s` test that creates its own basic environment. |
| 25 | +> |
| 26 | +> **It is highly recommended that you read it before continuing**. |
| 27 | +
|
| 28 | +We will focus on creating a basic testing environment compromised of: |
| 29 | +* 6 `chainlink nodes` |
| 30 | +* 1 blockchain node (`go-ethereum` aka `geth`) |
| 31 | + |
| 32 | +Let's start! |
| 33 | + |
| 34 | +# Step 1: Create Chainlink node TOML config |
| 35 | +In real-world scenario you should dynamically create or load Chainlink node configuration to match your needs. |
| 36 | +Here, for simplification, we will use a hardcoded config that will work for our case. |
| 37 | +```go |
| 38 | +func TestSimpleDONWithLinkContract(t *testing.T) { |
| 39 | + tomlConfig := `[Feature] |
| 40 | +FeedsManager = true |
| 41 | +LogPoller = true |
| 42 | +UICSAKeys = true |
| 43 | +
|
| 44 | +[Database] |
| 45 | +MaxIdleConns = 20 |
| 46 | +MaxOpenConns = 40 |
| 47 | +MigrateOnStartup = true |
| 48 | +
|
| 49 | +[Log] |
| 50 | +Level = "debug" |
| 51 | +JSONConsole = true |
| 52 | +
|
| 53 | +[Log.File] |
| 54 | +MaxSize = "0b" |
| 55 | +
|
| 56 | +[WebServer] |
| 57 | +AllowOrigins = "*" |
| 58 | +HTTPWriteTimeout = "3m0s" |
| 59 | +HTTPPort = 6688 |
| 60 | +SecureCookies = false |
| 61 | +SessionTimeout = "999h0m0s" |
| 62 | +
|
| 63 | +[WebServer.RateLimit] |
| 64 | +Authenticated = 2000 |
| 65 | +Unauthenticated = 1000 |
| 66 | +
|
| 67 | +[WebServer.TLS] |
| 68 | +HTTPSPort = 0 |
| 69 | +
|
| 70 | +[OCR] |
| 71 | +Enabled = true |
| 72 | +
|
| 73 | +[P2P] |
| 74 | +
|
| 75 | +[P2P.V2] |
| 76 | +ListenAddresses = ["0.0.0.0:6690"] |
| 77 | +
|
| 78 | +[[EVM]] |
| 79 | +ChainID = "1337" |
| 80 | +AutoCreateKey = true |
| 81 | +FinalityDepth = 1 |
| 82 | +MinContractPayment = "0" |
| 83 | +
|
| 84 | +[EVM.GasEstimator] |
| 85 | +PriceMax = "200 gwei" |
| 86 | +LimitDefault = 6000000 |
| 87 | +FeeCapDefault = "200 gwei" |
| 88 | +
|
| 89 | +[[EVM.Nodes]] |
| 90 | +Name = "Simulated Geth-0" |
| 91 | +WSURL = "ws://geth:8546" |
| 92 | +HTTPURL = "http://geth:8544"` |
| 93 | +``` |
| 94 | +
|
| 95 | +This configuration uses log poller and OCRv2 and connects to a single EVM chain with id `1337` that can be reached through RPC node with following URLS: |
| 96 | +* `ws://geth:8546` |
| 97 | +* `http://geth:8544` |
| 98 | + |
| 99 | +These are standard `geth` ports, and `geth` is the default name of our `go-ethereum` k8s service. We will connect to a "simulated" blockchain, which |
| 100 | +is a private ephemeral/on-demand blockchain composed of a single node. |
| 101 | + |
| 102 | +Now, let's build the chart that describes our chainlink `k8s` deployment: |
| 103 | +```go |
| 104 | +chainlinkImageCfg := &ctf_config.ChainlinkImageConfig{ |
| 105 | + Image: ptr.Ptr("public.ecr.aws/chainlink/chainlink"), |
| 106 | + Version: ptr.Ptr("2.19.0"), |
| 107 | +} |
| 108 | +
|
| 109 | +var overrideFn = func(_ interface{}, target interface{}) { |
| 110 | + ctf_config.MustConfigOverrideChainlinkVersion(chainlinkImageCfg, target) |
| 111 | +} |
| 112 | +
|
| 113 | +cd := chainlink.NewWithOverride(0, map[string]any{ |
| 114 | + "replicas": 6, // number of nodes |
| 115 | + "toml": tomlConfig, |
| 116 | + "db": map[string]any{ |
| 117 | + "stateful": true, // stateful DB by default for soak tests |
| 118 | + }, |
| 119 | +}, chainlinkImageCfg, overrideFn) |
| 120 | +``` |
| 121 | +
|
| 122 | +Here, we use a hardcoded image and version for the Chainlink node, but in real test you would like to make it configurable. This setup |
| 123 | +will launch 6 nodes, with stateful set dbs. It does look complex, but for various legacy reasons after removing support for some env vars |
| 124 | +this is how setting image name and version looks like. |
| 125 | +
|
| 126 | +# Step 2: Label resources |
| 127 | +For the purpose of better expenses tracking in the next step we will create necessary `chain.link` labels that every k8s resource needs to have. We will |
| 128 | +use existing convenience functions: |
| 129 | +```go |
| 130 | +productName := "data-feedsv1.0" |
| 131 | +nsLabels, err := environment.GetRequiredChainLinkNamespaceLabels(productName, "soak") |
| 132 | +if err != nil { |
| 133 | + t.Fatal("Error creating required chain.link labels for namespace", err) |
| 134 | +} |
| 135 | +
|
| 136 | +workloadPodLabels, err := environment.GetRequiredChainLinkWorkloadAndPodLabels(productName, "soak") |
| 137 | +if err != nil { |
| 138 | + t.Fatal("Error creating required chain.link labels for workload and pod", err) |
| 139 | +} |
| 140 | +``` |
| 141 | +
|
| 142 | +> [!NOTE] |
| 143 | +> As explained [here](../k8s/labels.md) there are two environment variables that need to be set |
| 144 | +> to satisfy labelling requirements: |
| 145 | +> - `CHAINLINK_ENV_USER` - name of person running the test |
| 146 | +> - `CHAINLINK_USER_TEAM` - name of the team, for which the test is run |
| 147 | +
|
| 148 | +# Step 3: Create environment config |
| 149 | +This step is pretty straightforward: |
| 150 | +```go |
| 151 | +baseEnvironmentConfig := &environment.Config{ |
| 152 | + TTL: time.Hour * 2, |
| 153 | + NamespacePrefix: "my-namespace-prefix", |
| 154 | + Test: t, |
| 155 | + PreventPodEviction: true, |
| 156 | + Labels: nsLabels, // pass labels created in previous step |
| 157 | + WorkloadLabels: workloadPodLabels, // pass labels created in previous step |
| 158 | + PodLabels: workloadPodLabels, // pass labels created in previous step |
| 159 | +} |
| 160 | +``` |
| 161 | +Just three explanations are necessary here: |
| 162 | +* `TTL` is the amount of time after which the namespace will by automatically removed |
| 163 | +* `NamespacePrefix` is the preffix to which unique hash will be attached to ensure name uniqueness |
| 164 | +* `PreventPodEviction` will prevent our pods from being evicted or restarted by `k8s` |
| 165 | +
|
| 166 | +# Step 4: Define blockchain network |
| 167 | +For simplicity, we will use a hardcoded "simulated" EVM network, which should more accurately be called |
| 168 | +an ephemeral private blockchain. In real case scenario you would use existing convenienice functions |
| 169 | +for dynamically selecting the network, to which nodes should connect as it could be either a "simulated" one or an existing network (public or private). |
| 170 | +In the latter case your code should skip adding the `ethereum` chart that represents `go-ethereum`-based blockchain node, as it |
| 171 | +be connecting an already available service. |
| 172 | +
|
| 173 | +```go |
| 174 | +nodeNetwork := blockchain.SimulatedEVMNetwork |
| 175 | +
|
| 176 | +ethProps := ðereum.Props{ |
| 177 | + NetworkName: nodeNetwork.Name, |
| 178 | + Simulated: nodeNetwork.Simulated, |
| 179 | + WsURLs: nodeNetwork.URLs, |
| 180 | + HttpURLs: nodeNetwork.HTTPURLs, |
| 181 | +} |
| 182 | +``` |
| 183 | +There's no default network name or URLs set for `ethereum` chart, so you need to set these as a minimum. |
| 184 | + |
| 185 | +# Step 5: Build the environment |
| 186 | +Now that we have all the building blocks lets put them together and build the environment: |
| 187 | +```go |
| 188 | +testEnv := environment.New(baseEnvironmentConfig). |
| 189 | + AddHelm(ethereum.New(ethProps)). // blockchain node |
| 190 | + AddHelm(cd) // chainlink node |
| 191 | +
|
| 192 | +err = testEnv.Run() |
| 193 | +if err != nil { |
| 194 | + t.Fatal("Error running environment", err) |
| 195 | +} |
| 196 | +``` |
| 197 | + |
| 198 | +# Step 6: Create new blockchain client |
| 199 | +With our environment created, let's create blockchain client, which will connect to our EVM node and later on deploy |
| 200 | +a contract. We will use [Seth](../../libs/seth.md) for that purpose: |
| 201 | +```go |
| 202 | +// if test is running inside K8s, nothing to do, default network urls are correct |
| 203 | +if !testEnv.Cfg.InsideK8s { |
| 204 | + // Test is running locally, use forwarded URL of Geth blockchain node |
| 205 | + wsURLs := testEnv.URLs[blockchain.SimulatedEVMNetwork.Name] |
| 206 | + httpURLs := testEnv.URLs[blockchain.SimulatedEVMNetwork.Name+"_http"] |
| 207 | + if len(wsURLs) == 0 || len(httpURLs) == 0 { |
| 208 | + t.Fatal("Forwarded Geth URLs should not be empty") |
| 209 | + } |
| 210 | + nodeNetwork.URLs = wsURLs |
| 211 | + nodeNetwork.HTTPURLs = httpURLs |
| 212 | +} |
| 213 | +
|
| 214 | +sethClient, err := seth.NewClientBuilder(). |
| 215 | + WithRpcUrl(nodeNetwork.URLs[0]). |
| 216 | + WithPrivateKeys([]string{nodeNetwork.PrivateKeys[0]}). |
| 217 | + Build() |
| 218 | +if err != nil { |
| 219 | + t.Fatal("Error creating Seth client", err) |
| 220 | +} |
| 221 | +``` |
| 222 | +Notice the URL rewriting for our `nodeNetwork`. That's required, because by default, that network uses the name |
| 223 | +of `geth` service in the `k8s` as it's URI. That works inside `k8s`, but not when your test is executing |
| 224 | +on local environment, as is currently the case. |
| 225 | +
|
| 226 | +`Environment` is capable of forwarding `k8s` ports to local machine and does that for some of applications automatically. |
| 227 | +`Geth` running in "simulated" mode is one of these and adds forwarded ports to the `URLs` map, so we can just grab them from it. |
| 228 | +
|
| 229 | +# Step 7: Deploy LINK contract |
| 230 | +Finally, let's deploy a LINK contract and assert that it's total supply isn't 0: |
| 231 | +```go |
| 232 | +linkTokenAbi, err := link_token_interface.LinkTokenMetaData.GetAbi() |
| 233 | +if err != nil { |
| 234 | + t.Fatal("Error getting LinkToken ABI", err) |
| 235 | +} |
| 236 | +linkDeploymentData, err := sethClient.DeployContract(sethClient.NewTXOpts(), "LinkToken", *linkTokenAbi, common.FromHex(link_token_interface.LinkTokenMetaData.Bin)) |
| 237 | +if err != nil { |
| 238 | + t.Fatal("Error deploying LinkToken contract", err) |
| 239 | +} |
| 240 | +linkToken, err := link_token_interface.NewLinkToken(linkDeploymentData.Address, sethClient.Client) |
| 241 | +if err != nil { |
| 242 | + t.Fatal("Error creating LinkToken contract instance", err) |
| 243 | +} |
| 244 | +
|
| 245 | +totalSupply, err := linkToken.TotalSupply(sethClient.NewCallOpts()) |
| 246 | +if err != nil { |
| 247 | + t.Fatal("Error getting total supply of LinkToken", err) |
| 248 | +} |
| 249 | +if totalSupply.Cmp(big.NewInt(0)) <= 0 { |
| 250 | + t.Fatal("Total supply of LinkToken should be greater than 0") |
| 251 | +} |
| 252 | +``` |
| 253 | +In a real world scenario that could be the end of the setup phase. Well, you should probably deploy a couple more contracts, |
| 254 | +maybe the data feeds? And then, generate some load, ideally using [WASP](../../libs/wasp/overview.md). |
| 255 | + |
| 256 | +Let's say that is what you really want. And that on top of that you would like your test to run for 2 days without having to |
| 257 | +keep your local machine up and running, or having to deal with CI limitations (6h maximum action duration in Github Actions). |
| 258 | +
|
| 259 | +In the [next chapter](./remote_runner.md) you'll learn how to achieve that. |
| 260 | + |
| 261 | +> [!NOTE] |
| 262 | +> You can find this example [here](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/lib/k8s/examples/link/link_test.go). |
0 commit comments