Chain Agnostic MultiNode + EVM Implementation (#13386)

* Update rpc client

* Update RPCClient

* Update RPCClient

* use Eth Subscription for heads

* Make RpcClient public

* Update node

* Update Multinode

* Fix build + generate

* Udate multinode

* update multinode

* fix tests

* Fix mocks

* Update node_lifecycle_test.go

* Fix all client tests

* Fix tests

* Update client_test.go

* go mod tidy

* fix tests

* Fix tests

* Clean up

* Fix features test mocking

* Fix logging

* Remove logging

* Fix tests

* Fix context

* lint

* Update node_lifecycle_test.go

* Remove unused generics

* Add state locking

* Set block difficulty

* Update node_lifecycle.go

* Fix tests

* Make NodeStates public

* Update multi_node_test.go

* Update Unsubscribe

* Remove HEAD generic from Node interface

* Remove unneeded generics

* Remove unneeded generics from Multinode

* Remove Multinode as interface

* Add PoolChainInfoProvider

* Setup SendOnly nodes

* Test empty context

* Add err to log

* Add rpc newHeads method

* Fix context

* Changeset

* Remove unused mocks

* Create Transaction Sender

* Implement TransactionSender

* Fix transaction sender types

* Implement transaction sender tests

* Ensure MultiNode is running

* Address comments

* Remove ChainClientRPC interface

* Remove unneeded test

* Generate mocks

* Fix tests

* fix tests

* Use UnsubscribeAllExcept

* Fix rpc client tests

* Address comments

* Remove unused code

* Generate private mock

* lint

* Fix locks and unsubscribing

* Update node.go

* fixed flaky headtracker tests

* Update node_lifecycle_test.go

* Update node_lifecycle_test.go

* Update transaction sender

* Update chain_client_test.go

* Remove unused variables

* lint

* enable secondary url test

* Use http if not nil

* Update transaction_sender.go

* Add close method

* Update rpc_client.go

* changeset

* Add Close to transaction sender

* Return struct not interface

* Remove flaky test

* Fix context

* Test if mn closes

* Fix context cancel

* Fix DoAll

* lint generate

* Revert

* Fix mocks

* Update node_lifecycle.go

* lint

* Delete rpc_client.go

* Address comments

* Fix tests

* Update common/client/send_only_node.go

Co-authored-by: Dmytro Haidashenko <[email protected]>

* Address comments

* Address comments

* Rename SubscribeToHeads

* Return code

* Update transaction_sender.go

* Fix merge conflicts

* Handle batch requests

* Delete mock_rpc_client_test.go

* Update common/client/types.go

Co-authored-by: Dmytro Haidashenko <[email protected]>

* Move RPC methods

* Use map for subs

* Don't cancel context

* Update transaction_sender.go

* Fix lint

* Delete late-mails-battle.md

* lint

* lint

* Check Transaction Sender state

* Return Retryable

* Update core/chains/evm/client/chain_client_test.go

Co-authored-by: Jordan Krage <[email protected]>

* Update sub_forwarder.go

* Update config.go

* lint

* Don't export nodeState

* Update changeset

* Use subsSliceMu

* lint

* Update rpc_client.go

* nil ws url for sendonly nodes

* Use head polling

---------

Co-authored-by: Dmytro Haidashenko <[email protected]>
Co-authored-by: Dmytro Haidashenko <[email protected]>
Co-authored-by: Jordan Krage <[email protected]>

Author: 3 people

.changeset/orange-feet-share.md

@@ -0,0 +1,25 @@
+---
+"chainlink": minor
+---
+
+Implemented new chain agnostic MultiNode design along with the corresponding EVM implementation. The chain agnostic components enable Multinode to be integrated with Solana and other non-EVM chains. Previously the Multinode was coupled with EVM specific actions, and was called to execute these actions direclty. With this change, the MultiNode's responsibility has been simplified to focus on RPC selection along with performing health checks. Chain specific actions will instead be executed on the RPC directly after being selected by MultiNode. The Chain Agnostic MultiNode provides improved reliability and metrics for all chain integrations using it.
+
+These are following main components:
+Node: Common component which wraps an RPC with state information, health checks, and an alive loop to handle state changes along with maintaining chain information.
+RPCClient: Chain-specific RPC wrapper which implements required interface for MultiNode along with any chain-specific functionality needed.
+MultiNode: Perform RPCClient selection and performs health checks on all RPCs.
+TransactionSender: Chain agnostic component which broadcasts transactions to all healthy RPCs and aggregates results. A chain-specific error classifier must be implemented.
+
+MultiNode picks the "best" RPC based on one of the configurable criteria:
+- Priority defined in the config.
+- Highest latest block.
+- Round-robin within the same priority level (or using other configurable selection algorithms)
+
+Benefits of Chain Agnostic MultiNode:
+  Reliability: Improved RPC reliability scaleable to all chains
+  Maintainability: Can apply changes across all chain integrations through the use of common code
+  Extendability: Can add new health checks, RPC selection and ranking algorithms
+  Integration Speed: Much faster to integrate MultiNode with new chains
+  Reduced Generics: Significantly less bulky code!
+
+#updated #changed #internal

.mockery.yaml

@@ -14,9 +14,8 @@ packages:
       NodeSelector:
       sendOnlyClient:
       SendOnlyNode:
-      RPC:
+      RPCClient:
       Head:
-      NodeClient:
       PoolChainInfoProvider:
   github.com/smartcontractkit/chainlink/v2/common/headtracker:
     interfaces:
@@ -50,7 +49,6 @@ packages:
   github.com/smartcontractkit/chainlink/v2/core/chains/evm/client:
     interfaces:
       Client:
-      RPCClient:
   github.com/smartcontractkit/chainlink/v2/core/chains/evm/config:
     interfaces:
       GasEstimator: