Skip to content

Commit 65f1613

Browse files
skudasovskudasov
committed
gas and reorg tests, docs
1 parent 8094529 commit 65f1613

File tree

14 files changed

+468
-252
lines changed

14 files changed

+468
-252
lines changed

‎book/src/SUMMARY.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@
1212
- [NodeSet Compat Environment](./framework/nodeset_compatibility.md)
1313
- [Creating your own components](./developing/developing_components.md)
1414
- [Asserting Logs](./developing/asserting_logs.md)
15-
- [Fork Testing](./framework/fork.md)
1615
- [Quick Contracts Deployment](./framework/quick_deployment.md)
1716
- [Verifying Contracts](./framework/verify.md)
1817
- [NodeSet with External Blockchain]()
@@ -55,7 +54,8 @@
5554
- [Testing Maturity Model](framework/testing.md)
5655
- [Smoke]()
5756
- [Performance]()
58-
- [Chaos]()
57+
- [Chaos](./framework/chaos/chaos.md)
58+
- [Fork Testing](./framework/fork.md)
5959
- [Interactive](framework/interactive.md)
6060
- [Libraries](./libraries.md)
6161
- [Seth](./libs/seth.md)

‎book/src/framework/chaos/chaos.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Chaos Testing
2+
3+
We offer Docker and Kubernetes boilerplates designed to test the resilience of `NodeSet` and `Blockchain`, which you can customize and integrate into your pipeline.
4+
5+
6+
## Goals
7+
8+
We recommend structuring your tests as a linear suite that applies various chaos experiments and verifies the outcomes using a [load](../../libs/wasp.md) testing suite. Focus on critical user metrics, such as:
9+
10+
- The ratio of successful responses to failed responses
11+
- The nth percentile of response latency
12+
13+
Next, evaluate observability:
14+
15+
- Ensure proper alerts are triggered during failures (manual or automated)
16+
- Verify the service recovers within the expected timeframe (manual or automated)
17+
18+
In summary, the **primary** focus is on meeting user expectations and maintaining SLAs, while the **secondary** focus is on observability and making operational part smoother.
19+
20+
21+
## Docker
22+
23+
For Docker, we utilize [Pumba](https://github.com/alexei-led/pumba) to conduct chaos experiments, including:
24+
25+
- Container reboots
26+
- Network simulations (such as delays, packet loss, corruption, etc., using the tc tool)
27+
- Stress testing for CPU and memory usage
28+
29+
Additionally, we offer a [resources](../../framework/components/resources.md) API that allows you to test whether your software can operate effectively in low-resource environments.
30+
31+
You can also use [fake](../../framework/components/mocking.md) package to create HTTP chaos experiments.
32+
33+
Given the complexity of `Kubernetes`, we recommend starting with `Docker` first. Identifying faulty behavior in your services early—such as cascading latency—can prevent more severe issues when scaling up. Addressing these problems at a smaller scale can save significant time and effort later.
34+
35+
Check `NodeSet` + `Blockchain` template [here]().
36+
37+
## Kubernetes
38+
39+
We utilize a subset of [ChaosMesh](https://chaos-mesh.org/) experiments that can be safely executed on an isolated node group. These include:
40+
41+
- [Pod faults](https://chaos-mesh.org/docs/simulate-pod-chaos-on-kubernetes/)
42+
43+
- [Network faults](https://chaos-mesh.org/docs/simulate-network-chaos-on-kubernetes/) – We focus on delay and partition experiments, as others may impact pods outside the dedicated node group.
44+
45+
- [HTTP faults](https://chaos-mesh.org/docs/simulate-http-chaos-on-kubernetes/)
46+
47+
Check `NodeSet` + `Blockchain` template [here]().
48+
49+
## Blockchain
50+
51+
We also offer a set of blockchain-specific experiments, which typically involve API calls to blockchain simulators to execute certain actions. These include:
52+
53+
- Adjusting gas prices
54+
55+
- Introducing chain reorganizations (setting a new head)
56+
57+
- Utilizing developer APIs (e.g., Anvil)
58+
59+
Check [gas]() and [reorg]() examples.

framework/examples/myproject/chaos.toml renamed to framework/examples/myproject/chaos/chaos.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11

22
[blockchain_a]
3-
type = "anvil"
43
docker_cmd_params = ["-b", "1"]
4+
type = "anvil"
55

66
[data_provider]
77
port = 9111

‎framework/examples/myproject/chaos/chaos_blockchain_evm_gas_test.go‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
package chaos
2+
3+
import (
4+
"context"
5+
"github.com/ethereum/go-ethereum/ethclient"
6+
"github.com/smartcontractkit/chainlink-testing-framework/framework"
7+
"github.com/smartcontractkit/chainlink-testing-framework/framework/clclient"
8+
"github.com/smartcontractkit/chainlink-testing-framework/framework/components/blockchain"
9+
"github.com/smartcontractkit/chainlink-testing-framework/framework/components/fake"
10+
ns "github.com/smartcontractkit/chainlink-testing-framework/framework/components/simple_node_set"
11+
"github.com/smartcontractkit/chainlink-testing-framework/framework/rpc"
12+
"github.com/stretchr/testify/require"
13+
"math/big"
14+
"testing"
15+
"time"
16+
)
17+
18+
func printBlockBaseFee(t *testing.T, url string) {
19+
ec, err := ethclient.Dial(url)
20+
require.NoError(t, err)
21+
bn, err := ec.BlockNumber(context.Background())
22+
require.NoError(t, err)
23+
b, err := ec.BlockByNumber(context.Background(), big.NewInt(int64(bn)))
24+
require.NoError(t, err)
25+
t.Logf("Current block base fee: %d", b.BaseFee())
26+
}
27+
28+
type CfgGas struct {
29+
BlockchainA *blockchain.Input `toml:"blockchain_a" validate:"required"`
30+
MockerDataProvider *fake.Input `toml:"data_provider" validate:"required"`
31+
NodeSet *ns.Input `toml:"nodeset" validate:"required"`
32+
}
33+
34+
func TestBlockchainGasChaos(t *testing.T) {
35+
in, err := framework.Load[CfgGas](t)
36+
require.NoError(t, err)
37+
38+
// Can replace deployments with CRIB here
39+
40+
bc, err := blockchain.NewBlockchainNetwork(in.BlockchainA)
41+
require.NoError(t, err)
42+
_, err = fake.NewFakeDataProvider(in.MockerDataProvider)
43+
require.NoError(t, err)
44+
out, err := ns.NewSharedDBNodeSet(in.NodeSet, bc)
45+
require.NoError(t, err)
46+
47+
c, err := clclient.New(out.CLNodes)
48+
require.NoError(t, err)
49+
50+
// !! This value must match anvil block speed to set gas values for every block !!
51+
blockEvery := 1 * time.Second
52+
waitBetweenTests := 1 * time.Minute
53+
54+
gasControlFunc := func(t *testing.T, r *rpc.RPCClient, url string) {
55+
startGasPrice := big.NewInt(2e9)
56+
// ramp
57+
for i := 0; i < 10; i++ {
58+
printBlockBaseFee(t, url)
59+
t.Logf("Setting block base fee: %d", startGasPrice)
60+
err := r.AnvilSetNextBlockBaseFeePerGas(startGasPrice)
61+
require.NoError(t, err)
62+
startGasPrice = startGasPrice.Add(startGasPrice, big.NewInt(1e9))
63+
time.Sleep(blockEvery)
64+
}
65+
// hold
66+
for i := 0; i < 10; i++ {
67+
printBlockBaseFee(t, url)
68+
t.Logf("Setting block base fee: %d", startGasPrice)
69+
err := r.AnvilSetNextBlockBaseFeePerGas(startGasPrice)
70+
require.NoError(t, err)
71+
}
72+
// release
73+
for i := 0; i < 10; i++ {
74+
printBlockBaseFee(t, url)
75+
time.Sleep(blockEvery)
76+
}
77+
}
78+
79+
testCases := []struct {
80+
name string
81+
chainURL string
82+
increase *big.Int
83+
waitBetweenTests time.Duration
84+
gasFunc func(t *testing.T, r *rpc.RPCClient, url string)
85+
validate func(t *testing.T, c []*clclient.ChainlinkClient)
86+
}{
87+
{
88+
name: "Slow and low",
89+
chainURL: bc.Nodes[0].HostHTTPUrl,
90+
waitBetweenTests: 30 * time.Second,
91+
increase: big.NewInt(1e9),
92+
gasFunc: gasControlFunc,
93+
validate: func(t *testing.T, c []*clclient.ChainlinkClient) {
94+
// add more clients and validate
95+
},
96+
},
97+
{
98+
name: "Fast and degen",
99+
chainURL: bc.Nodes[0].HostHTTPUrl,
100+
waitBetweenTests: 30 * time.Second,
101+
increase: big.NewInt(5e9),
102+
gasFunc: gasControlFunc,
103+
validate: func(t *testing.T, c []*clclient.ChainlinkClient) {
104+
// add more clients and validate
105+
},
106+
},
107+
}
108+
109+
// Start WASP load test here, apply average load profile that you expect in production!
110+
// Configure timeouts and validate all the test cases until the test ends
111+
112+
// Run test cases
113+
for _, tc := range testCases {
114+
t.Run(tc.name, func(t *testing.T) {
115+
t.Log(tc.name)
116+
printBlockBaseFee(t, tc.chainURL)
117+
r := rpc.New(tc.chainURL, nil)
118+
tc.gasFunc(t, r, tc.chainURL)
119+
tc.validate(t, c)
120+
time.Sleep(waitBetweenTests)
121+
})
122+
}
123+
}

‎framework/examples/myproject/chaos/chaos_blockchain_evm_reorg_test.go‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
package chaos
2+
3+
import (
4+
"github.com/smartcontractkit/chainlink-testing-framework/framework"
5+
"github.com/smartcontractkit/chainlink-testing-framework/framework/clclient"
6+
"github.com/smartcontractkit/chainlink-testing-framework/framework/components/blockchain"
7+
"github.com/smartcontractkit/chainlink-testing-framework/framework/components/clnode"
8+
"github.com/smartcontractkit/chainlink-testing-framework/framework/components/fake"
9+
ns "github.com/smartcontractkit/chainlink-testing-framework/framework/components/simple_node_set"
10+
"github.com/smartcontractkit/chainlink-testing-framework/framework/rpc"
11+
"github.com/stretchr/testify/require"
12+
"testing"
13+
"time"
14+
)
15+
16+
type CfgReorgTwoChains struct {
17+
BlockchainA *blockchain.Input `toml:"blockchain_a" validate:"required"`
18+
BlockchainB *blockchain.Input `toml:"blockchain_b" validate:"required"`
19+
MockerDataProvider *fake.Input `toml:"data_provider" validate:"required"`
20+
NodeSet *ns.Input `toml:"nodeset" validate:"required"`
21+
}
22+
23+
func TestBlockchainReorgChaos(t *testing.T) {
24+
in, err := framework.Load[CfgReorgTwoChains](t)
25+
require.NoError(t, err)
26+
27+
// Can replace deployments with CRIB here
28+
29+
bcA, err := blockchain.NewBlockchainNetwork(in.BlockchainA)
30+
require.NoError(t, err)
31+
bcB, err := blockchain.NewBlockchainNetwork(in.BlockchainB)
32+
require.NoError(t, err)
33+
// create network configs for 2 EVM networks
34+
srcNetworkCfg, err := clnode.NewNetworkCfg(&clnode.EVMNetworkConfig{
35+
MinIncomingConfirmations: 1,
36+
MinContractPayment: "0.00001 link",
37+
ChainID: bcA.ChainID,
38+
EVMNodes: []*clnode.EVMNode{
39+
{
40+
SendOnly: false,
41+
Order: 100,
42+
},
43+
},
44+
}, bcA)
45+
dstNetworkConfig, err := clnode.NewNetworkCfg(&clnode.EVMNetworkConfig{
46+
MinIncomingConfirmations: 1,
47+
MinContractPayment: "0.00001 link",
48+
ChainID: bcA.ChainID,
49+
EVMNodes: []*clnode.EVMNode{
50+
{
51+
SendOnly: false,
52+
Order: 100,
53+
},
54+
},
55+
}, bcB)
56+
// override the configuration to connect with 2 networks
57+
in.NodeSet.NodeSpecs[0].Node.TestConfigOverrides = srcNetworkCfg + dstNetworkConfig
58+
// create a node set
59+
nodesOut, err := ns.NewSharedDBNodeSet(in.NodeSet, bcA)
60+
require.NoError(t, err)
61+
62+
c, err := clclient.New(nodesOut.CLNodes)
63+
require.NoError(t, err)
64+
65+
testCases := []struct {
66+
name string
67+
wait time.Duration
68+
chainURL string
69+
reorgDepth int
70+
validate func(c []*clclient.ChainlinkClient) error
71+
}{
72+
{
73+
name: "Reorg src with depth: 1",
74+
wait: 30 * time.Second,
75+
chainURL: bcA.Nodes[0].HostHTTPUrl,
76+
reorgDepth: 1,
77+
validate: func(c []*clclient.ChainlinkClient) error {
78+
// add clients and validate
79+
return nil
80+
},
81+
},
82+
{
83+
name: "Reorg dst with depth: 1",
84+
wait: 30 * time.Second,
85+
chainURL: bcB.Nodes[0].HostHTTPUrl,
86+
reorgDepth: 1,
87+
validate: func(c []*clclient.ChainlinkClient) error {
88+
return nil
89+
},
90+
},
91+
{
92+
name: "Reorg src with depth: 5",
93+
wait: 30 * time.Second,
94+
chainURL: bcA.Nodes[0].HostHTTPUrl,
95+
reorgDepth: 5,
96+
validate: func(c []*clclient.ChainlinkClient) error {
97+
return nil
98+
},
99+
},
100+
{
101+
name: "Reorg dst with depth: 5",
102+
wait: 30 * time.Second,
103+
chainURL: bcB.Nodes[0].HostHTTPUrl,
104+
reorgDepth: 5,
105+
validate: func(c []*clclient.ChainlinkClient) error {
106+
return nil
107+
},
108+
},
109+
}
110+
111+
// Start WASP load test here, apply average load profile that you expect in production!
112+
// Configure timeouts and validate all the test cases until the test ends
113+
114+
// Run test cases
115+
for _, tc := range testCases {
116+
t.Run(tc.name, func(t *testing.T) {
117+
t.Log(tc.name)
118+
r := rpc.New(tc.chainURL, nil)
119+
err := r.GethSetHead(tc.reorgDepth)
120+
require.NoError(t, err)
121+
time.Sleep(tc.wait)
122+
err = tc.validate(c)
123+
require.NoError(t, err)
124+
})
125+
}
126+
}

0 commit comments

Comments
 (0)