Merge branch 'main' into fund-testnet

Author: skudasov

book/src/libs/seth.md

@@ -851,25 +851,27 @@ Additionally, when the client is build anc `cfg.ReadOnly = true` is set, we will
 
 ## ABI Finder
 
-In order to be able to decode and trace transactions and calls between smart contracts we need their ABIs. Unfortunately it might happen that two or more contracts have methods with the same signatures, which might result in incorrect tracing. To make that problem less severe we have decided to add a single point of entry for contract deployment in Seth as that way we always know what contract is deployed at which address and thus avoid incorrect tracing due to potentially ambiguous method signatures.
+In order to be able to decode and trace transactions and calls between smart contracts we need their ABIs. Unfortunately, it might happen that two or more contracts have methods with the same signatures, which might result in incorrect tracing. To make that problem less severe we have decided to add a single point of entry for contract deployment in Seth to track what contract is deployed at which address and thus minimise incorrect tracing due to potentially ambiguous method signatures.
 
-1.  We don’t know what contract (ABI) is located at a given address. Should be the case, when the contract either wasn’t uploaded via Seth or we haven’t supplied Seth with a contract map as part of its configuration (more on that later).
+There are two possible starting points:
 
-    a. We sequentially iterate over all known ABIs (map: `name -> ABI_name`) checking whether it has a method with a given signature. Once we get a first match we will upsert that (`address -> ABI_name`) data into the contract map and return the ABI.
+1.  We don’t know what contract (ABI) is located at a given address. That is the case, when the contract either wasn’t uploaded via Seth or we haven’t supplied Seth with a contract map as part of its configuration (more on that later).
 
-        The caveat here is that if the method we are searching for belongs is present in more than one ABI we might associate the address with an incorrect address (we will use the first match).
+    a. We sequentially iterate over all known ABIs (map: `name -> ABI_name`) checking whether it has a method with a given signature. Once we get a first match we upsert that (`address -> ABI_name`) data into the contract map and return the ABI.
 
-    b. If no match is found we will return an error.
+        The caveat here is that if the method we are searching for is present in more than one ABI we might associate the address with an incorrect address (we use the first match).
 
-2.  We know what ABI is located at a given address. It should be the case, when we have either uploaded the contract via Seth, provided Seth with a contract map or already traced a transaction to that address and found an ABI with matching method signature.
+    b. If no match is found we return an error.
 
-    a. We fetch the corresponding ABIand check if it indeed contains the method we are looking for (as mentioned earlier in some cases it might not be the case).
+2.  We know what ABI is located at a given address. That is the case, when we have either uploaded the contract via Seth, provided Seth with a contract map or already traced a transaction to that address and found an ABI with matching method signature.
+
+    a. We fetch the corresponding ABI and check if it indeed contains the method we are looking for (as mentioned earlier in some cases it might not be the case, because we associated the ABI with address using a different method that it shared with some other contract).
 
     b. If it does, we return the ABI.
 
-    c. If it doesn’t we iterate over all known ABIs, in the same way as in 1a. If we find a match we update the (`address -> ABI_name`) association in the contract map and return the ABI.
+    c. If it doesn’t we iterate again over all known ABIs, in the same way as in `1a`. If we find a match we update the (`address -> ABI_name`) association in the contract map and return the ABI.
 
-        It is possible that this will happen multiple times in case we have multiple contracts with multiple identical methods, but given a sufficiently diverse set of methods that were called we should eventually arrive at a fully correct contract map.
+        It is possible that this will happen multiple times, if we have multiple contracts with multiple identical methods, but given a sufficiently diverse set of methods that were called we should eventually arrive at a fully correct contract map.
 
     d. If no match is found we will return an error.
 
@@ -879,16 +881,16 @@ In order to be able to decode and trace transactions and calls between smart con
 
 ## Contract Map
 
-We support in-memory contract map and a TOML file contract map that keeps the association of (`address -> ABI_name`). The latter map is only used for non-simulated networks. Every time we deploy a contract we save (`address -> ABI_name`) entry in the in-memory map.If the network is not a simulated one we also save it in a file. That file can later be pointed to in Seth configuration and we will load the contract map from it (**currently without validating whether we have all the ABIs mentioned in the file**).
+We support in-memory contract map and a TOML file-based one that keeps the association of (`address -> ABI_name`). The latter map is only used for non-simulated networks. Every time we deploy a contract we save (`address -> ABI_name`) entry in the in-memory map. If the network is not a simulated one we also save it in a file. That file can later be pointed to in Seth configuration and we will load the contract map from it (**currently without validating whether we have all the ABIs mentioned in the file**).
 
-When saving contract deployment information we will either generate filename for you (if you didn’t configure Seth to use a particular file) using the pattern of `deployed_contracts_${network_name}_${timestamp}.toml` or use the filename provided in Seth TOML configuration file.
+When saving contract deployment information we can either generate filename for you (if you didn’t configure Seth to use a particular file) using the pattern of `deployed_contracts_${network_name}_${timestamp}.toml` or use the filename provided in Seth TOML configuration file.
 
-It has to be noted that the file contract map is currently updated only, when new contracts are deployed. There’s no mechanism for updating it if we found the mapping invalid (which might be the case if you manually created the entry in the file).
+It has to be noted that the file-based contract map is currently updated only, when new contracts are deployed. There’s no mechanism for updating it if we found the mapping invalid (which might be the case if you manually created the entry in the file).
 
 ## Contract Store
 
-Seth can be used with the contract store, but it would have very limited usage as transaction decoding and tracing cannot work with ABIs. Thus, when you initialise Seth with contract store it is necessary that it can load at least one ABI.
+Contract store is a component that stores ABIs and contracts' bytecodes. In theory, Seth can be used without it, but it would have very limited usage as transaction decoding and tracing cannot work without ABIs. Thus in practice, we enforce a non-empty Contract Store durin Seth initialisation.
 
 Another use of Contract Store is simplified contract deployment. For that we also need the contract's bytecode. The contract store can be used to store the bytecode of the contract and then deploy it using the `DeployContractFromContractStore(auth *bind.TransactOpts, name string, backend bind.ContractBackend, params ...interface{})` method. When Seth is intialisied with the contract store and no bytecode files (`*.bin`) are provided, it will log a warning, but initialise successfully nonetheless.
 
-If bytecode file wasn't provided you need to use `DeployContract(auth *bind.TransactOpts, name string, abi abi.ABI, bytecode []byte, backend bind.ContractBackend, params ...interface{})` method, which expects you to provide contract name (best if equal to the name of the ABI file), bytecode and the ABI.
+If bytecode file wasn't provided, you need to use `DeployContract(auth *bind.TransactOpts, name string, abi abi.ABI, bytecode []byte, backend bind.ContractBackend, params ...interface{})` method, which expects you to provide contract name (best if equal to the name of the ABI file), bytecode and the ABI.

framework/clclient/client.go

@@ -557,15 +557,20 @@ func (c *ChainlinkClient) UpdateEthKeyMaxGasPriceGWei(keyId string, gWei int) (*
 }
 
 // ReadPrimaryETHKey reads updated information about the Chainlink's primary ETH key
-func (c *ChainlinkClient) ReadPrimaryETHKey() (*ETHKeyData, error) {
+func (c *ChainlinkClient) ReadPrimaryETHKey(chainId string) (*ETHKeyData, error) {
 	ethKeys, err := c.MustReadETHKeys()
 	if err != nil {
 		return nil, err
 	}
 	if len(ethKeys.Data) == 0 {
 		return nil, fmt.Errorf("Error retrieving primary eth key on node %s: No ETH keys present", c.URL())
 	}
-	return &ethKeys.Data[0], nil
+	for _, data := range ethKeys.Data {
+		if data.Attributes.ChainID == chainId {
+			return &data, nil
+		}
+	}
+	return nil, fmt.Errorf("error retrieving primary eth key on node %s: No ETH keys present for chain %s", c.URL(), chainId)
 }
 
 // ReadETHKeyAtIndex reads updated information about the Chainlink's ETH key at given index

framework/components/simple_node_set/fund.go

@@ -10,6 +10,7 @@ import (
 	"github.com/ethereum/go-ethereum/core/types"
 	"github.com/ethereum/go-ethereum/crypto"
 	"github.com/ethereum/go-ethereum/ethclient"
+	er "github.com/pkg/errors"
 	"github.com/smartcontractkit/chainlink-testing-framework/framework"
 	"github.com/smartcontractkit/chainlink-testing-framework/framework/clclient"
 	"math/big"
@@ -18,7 +19,7 @@ import (
 func SendETH(client *ethclient.Client, privateKeyHex string, toAddress string, amount *big.Float) error {
 	privateKey, err := crypto.HexToECDSA(privateKeyHex)
 	if err != nil {
-		return fmt.Errorf("failed to parse private key: %v", err)
+		return er.Wrap(err, "failed to parse private key")
 	}
 	wei := new(big.Int)
 	amountWei := new(big.Float).Mul(amount, big.NewFloat(1e18))
@@ -33,29 +34,29 @@ func SendETH(client *ethclient.Client, privateKeyHex string, toAddress string, a
 
 	nonce, err := client.PendingNonceAt(context.Background(), fromAddress)
 	if err != nil {
-		return fmt.Errorf("failed to fetch nonce: %v", err)
+		return er.Wrap(err, "failed to fetch nonce")
 	}
 
 	gasPrice, err := client.SuggestGasPrice(context.Background())
 	if err != nil {
-		return fmt.Errorf("failed to fetch gas price: %v", err)
+		return er.Wrap(err, "failed to fetch gas price")
 	}
 	gasLimit := uint64(21000) // Standard gas limit for ETH transfer
 
 	tx := types.NewTransaction(nonce, common.HexToAddress(toAddress), wei, gasLimit, gasPrice, nil)
 
 	chainID, err := client.NetworkID(context.Background())
 	if err != nil {
-		return fmt.Errorf("failed to fetch chain ID: %v", err)
+		return er.Wrap(err, "failed to fetch chain ID")
 	}
 	signedTx, err := types.SignTx(tx, types.NewEIP155Signer(chainID), privateKey)
 	if err != nil {
-		return fmt.Errorf("failed to sign transaction: %v", err)
+		return er.Wrap(err, "failed to sign transaction")
 	}
 
 	err = client.SendTransaction(context.Background(), signedTx)
 	if err != nil {
-		return fmt.Errorf("failed to send transaction: %v", err)
+		return er.Wrap(err, "failed to send transaction")
 	}
 	framework.L.Info().Msgf("Transaction sent: %s", signedTx.Hash().Hex())
 	_, err = bind.WaitMined(context.Background(), client, signedTx)
@@ -67,13 +68,17 @@ func FundNodes(c *ethclient.Client, nodes []*clclient.ChainlinkClient, pkey stri
 	if ethAmount == 0 {
 		return errors.New("funds_eth is 0, set some value in config, ex.: funds_eth = 30.0")
 	}
+	chainID, err := c.ChainID(context.Background())
+	if err != nil {
+		return er.Wrap(err, "failed to fetch chain ID")
+	}
 	for _, cl := range nodes {
-		ek, err := cl.ReadPrimaryETHKey()
+		ek, err := cl.ReadPrimaryETHKey(chainID.String())
 		if err != nil {
 			return err
 		}
 		if err := SendETH(c, pkey, ek.Attributes.Address, big.NewFloat(ethAmount)); err != nil {
-			return fmt.Errorf("failed to fund CL node %s: %w", ek.Attributes.Address, err)
+			return er.Wrapf(err, "failed to fund CL node %s", ek.Attributes.Address)
 		}
 	}
 	return nil

tools/flakeguard/cmd/aggregate_results.go

@@ -121,7 +121,8 @@ var AggregateResultsCmd = &cobra.Command{
 
 			// Remove logs from test results for the report without logs
 			for i := range failedReportWithLogs.Results {
-				failedReportWithLogs.Results[i].Outputs = nil
+				failedReportWithLogs.Results[i].PassedOutputs = nil
+				failedReportWithLogs.Results[i].FailedOutputs = nil
 				failedReportWithLogs.Results[i].PackageOutputs = nil
 			}
 
@@ -137,7 +138,8 @@ var AggregateResultsCmd = &cobra.Command{
 
 		// Remove logs from test results for the aggregated report
 		for i := range aggregatedReport.Results {
-			aggregatedReport.Results[i].Outputs = nil
+			aggregatedReport.Results[i].PassedOutputs = nil
+			aggregatedReport.Results[i].FailedOutputs = nil
 			aggregatedReport.Results[i].PackageOutputs = nil
 		}
 

tools/flakeguard/reports/data.go

@@ -36,7 +36,9 @@ type TestResult struct {
 	Failures       int
 	Successes      int
 	Skips          int
-	Outputs        []string
+	Outputs        map[string][]string `json:"-"` // Temporary storage for outputs during test run
+	PassedOutputs  map[string][]string // Outputs for passed runs
+	FailedOutputs  map[string][]string // Outputs for failed runs
 	Durations      []time.Duration
 	PackageOutputs []string
 	TestPath       string
@@ -191,7 +193,18 @@ func aggregateFromReports(reports ...*TestReport) (*TestReport, error) {
 func mergeTestResults(a, b TestResult) TestResult {
 	a.Runs += b.Runs
 	a.Durations = append(a.Durations, b.Durations...)
-	a.Outputs = append(a.Outputs, b.Outputs...)
+	if a.PassedOutputs == nil {
+		a.PassedOutputs = make(map[string][]string)
+	}
+	if a.FailedOutputs == nil {
+		a.FailedOutputs = make(map[string][]string)
+	}
+	for runID, outputs := range b.PassedOutputs {
+		a.PassedOutputs[runID] = append(a.PassedOutputs[runID], outputs...)
+	}
+	for runID, outputs := range b.FailedOutputs {
+		a.FailedOutputs[runID] = append(a.FailedOutputs[runID], outputs...)
+	}
 	a.PackageOutputs = append(a.PackageOutputs, b.PackageOutputs...)
 	a.Successes += b.Successes
 	a.Failures += b.Failures

tools/flakeguard/reports/data_test.go

@@ -346,11 +346,13 @@ func TestAggregateOutputs(t *testing.T) {
 		TestRunCount: 1,
 		Results: []TestResult{
 			{
-				TestName:       "TestOutput",
-				TestPackage:    "pkg1",
-				Runs:           1,
-				Successes:      1,
-				Outputs:        []string{"Output from report1 test run"},
+				TestName:    "TestOutput",
+				TestPackage: "pkg1",
+				Runs:        1,
+				Successes:   1,
+				PassedOutputs: map[string][]string{
+					"run1": {"Output from report1 test run"},
+				},
 				PackageOutputs: []string{"Package output from report1"},
 			},
 		},
@@ -361,11 +363,13 @@ func TestAggregateOutputs(t *testing.T) {
 		TestRunCount: 1,
 		Results: []TestResult{
 			{
-				TestName:       "TestOutput",
-				TestPackage:    "pkg1",
-				Runs:           1,
-				Successes:      1,
-				Outputs:        []string{"Output from report2 test run"},
+				TestName:    "TestOutput",
+				TestPackage: "pkg1",
+				Runs:        1,
+				Successes:   1,
+				PassedOutputs: map[string][]string{
+					"run2": {"Output from report2 test run"},
+				},
 				PackageOutputs: []string{"Package output from report2"},
 			},
 		},
@@ -382,18 +386,22 @@ func TestAggregateOutputs(t *testing.T) {
 
 	result := aggregatedReport.Results[0]
 
-	// Expected outputs
-	expectedOutputs := []string{
-		"Output from report1 test run",
-		"Output from report2 test run",
+	expectedOutputs := map[string][]string{
+		"run1": {
+			"Output from report1 test run",
+		},
+		"run2": {
+			"Output from report2 test run",
+		},
 	}
+
 	expectedPackageOutputs := []string{
 		"Package output from report1",
 		"Package output from report2",
 	}
 
-	if !reflect.DeepEqual(result.Outputs, expectedOutputs) {
-		t.Errorf("Expected Outputs %v, got %v", expectedOutputs, result.Outputs)
+	if !reflect.DeepEqual(result.PassedOutputs, expectedOutputs) {
+		t.Errorf("Expected Outputs %v, got %v", expectedOutputs, result.PassedOutputs)
 	}
 
 	if !reflect.DeepEqual(result.PackageOutputs, expectedPackageOutputs) {
@@ -407,11 +415,13 @@ func TestAggregateIdenticalOutputs(t *testing.T) {
 		TestRunCount: 1,
 		Results: []TestResult{
 			{
-				TestName:       "TestIdenticalOutput",
-				TestPackage:    "pkg1",
-				Runs:           1,
-				Successes:      1,
-				Outputs:        []string{"Identical output"},
+				TestName:    "TestIdenticalOutput",
+				TestPackage: "pkg1",
+				Runs:        1,
+				Successes:   1,
+				PassedOutputs: map[string][]string{
+					"run1": {"Identical output"},
+				},
 				PackageOutputs: []string{"Identical package output"},
 			},
 		},
@@ -422,11 +432,13 @@ func TestAggregateIdenticalOutputs(t *testing.T) {
 		TestRunCount: 1,
 		Results: []TestResult{
 			{
-				TestName:       "TestIdenticalOutput",
-				TestPackage:    "pkg1",
-				Runs:           1,
-				Successes:      1,
-				Outputs:        []string{"Identical output"},
+				TestName:    "TestIdenticalOutput",
+				TestPackage: "pkg1",
+				Runs:        1,
+				Successes:   1,
+				PassedOutputs: map[string][]string{
+					"run1": {"Identical output"},
+				},
 				PackageOutputs: []string{"Identical package output"},
 			},
 		},
@@ -443,80 +455,24 @@ func TestAggregateIdenticalOutputs(t *testing.T) {
 
 	result := aggregatedReport.Results[0]
 
-	// Expected outputs
-	expectedOutputs := []string{
-		"Identical output",
-		"Identical output",
+	expectedOutputs := map[string][]string{
+		"run1": {"Identical output", "Identical output"},
 	}
+
 	expectedPackageOutputs := []string{
 		"Identical package output",
 		"Identical package output",
 	}
 
-	if !reflect.DeepEqual(result.Outputs, expectedOutputs) {
-		t.Errorf("Expected Outputs %v, got %v", expectedOutputs, result.Outputs)
+	if !reflect.DeepEqual(result.PassedOutputs, expectedOutputs) {
+		t.Errorf("Expected Outputs %v, got %v", expectedOutputs, result.PassedOutputs)
 	}
 
 	if !reflect.DeepEqual(result.PackageOutputs, expectedPackageOutputs) {
 		t.Errorf("Expected PackageOutputs %v, got %v", expectedPackageOutputs, result.PackageOutputs)
 	}
 }
 
-// TestMergeTestResults tests the mergeTestResults function.
-func TestMergeTestResults(t *testing.T) {
-	a := TestResult{
-		TestName:       "TestA",
-		TestPackage:    "pkg1",
-		Runs:           2,
-		Successes:      2,
-		Failures:       0,
-		Skips:          0,
-		Durations:      []time.Duration{time.Second, time.Second},
-		Outputs:        []string{"Output1", "Output2"},
-		PackageOutputs: []string{"PkgOutput1"},
-		Panic:          false,
-		Race:           false,
-		Skipped:        false,
-	}
-
-	b := TestResult{
-		TestName:       "TestA",
-		TestPackage:    "pkg1",
-		Runs:           3,
-		Successes:      2,
-		Failures:       1,
-		Skips:          0,
-		Durations:      []time.Duration{2 * time.Second, 2 * time.Second, 2 * time.Second},
-		Outputs:        []string{"Output3", "Output4", "Output5"},
-		PackageOutputs: []string{"PkgOutput2"},
-		Panic:          true,
-		Race:           false,
-		Skipped:        false,
-	}
-
-	merged := mergeTestResults(a, b)
-
-	expected := TestResult{
-		TestName:       "TestA",
-		TestPackage:    "pkg1",
-		Runs:           5,
-		Successes:      4,
-		Failures:       1,
-		Skips:          0,
-		Durations:      []time.Duration{time.Second, time.Second, 2 * time.Second, 2 * time.Second, 2 * time.Second},
-		Outputs:        []string{"Output1", "Output2", "Output3", "Output4", "Output5"},
-		PackageOutputs: []string{"PkgOutput1", "PkgOutput2"},
-		Panic:          true,
-		Race:           false,
-		Skipped:        false,
-		PassRatio:      0.8,
-	}
-
-	if !reflect.DeepEqual(merged, expected) {
-		t.Errorf("Expected %+v, got %+v", expected, merged)
-	}
-}
-
 // TestAvgDuration tests the avgDuration function.
 func TestAvgDuration(t *testing.T) {
 	durations := []time.Duration{