Complete rewrite

Author: namikmesic

.gitignore

@@ -12,5 +12,6 @@
 *.out
 local*
 
-# Dependency directories (remove the comment below to include it)
-# vendor/
+# Python
+venv
+__pycache__

CHANGELOG.md

@@ -1,13 +1,14 @@
 # Changelog
 
 ## v1.0.0
-
-Initial release
+- Initial release
 
 ## v1.0.1
-
-Reworked logic for totalDifficulty to support chains that don't use it
+- Reworked logic for totalDifficulty to support chains that don't use it
 
 ## v1.0.2
+- Handling response of type `None` in JSON RPC response.
 
-Handling response of type `None` in JSON RPC response.
+## v2.0.0
+- Complete rewrite of evm collector to use web3.py library.
+- Introducing limited support for Conflux, Solana and Cardano

Dockerfile

@@ -10,7 +10,7 @@ COPY --from=build /usr/local/lib/python3.9/site-packages/ \
 ENV LC_ALL C.UTF-8
 WORKDIR /usr/src/app
 COPY src/*.py .
-
+COPY src/collectors/* collectors/
 # https://github.com/GoogleContainerTools/distroless/blob/main/experimental/python3/BUILD#L77
 USER nonroot
 CMD ["exporter.py"]

README.md

@@ -1,24 +1,27 @@
 # EVM RPC Websocket Exporter
-Simple exporter used to scrape different metrics from EVM compatible websocket RPCs.
-
+The exporter is used to scrape metrics from blockchain RPC endpoints. The purpose of this exporter is to perform black-box testing on RPC endpoints.
+c
 ## Metrics
-For list of metrics and explanations please check [collect method](src/exporter.py#L42-L79) in exporter source-code.
-For available labels please see [metric_labels](src/exporter.py#L37-L40) list in exporter source-code.
+Exporter currently supports all EVM-compatible chains. In addition, there is limited support for the following chains:
+- Cardano
+- Conflux
+- Solana
 
 # Disclaimer
-Please note that this tool is in early development stage, and as such should not be used to influence business crictical decisions.
-The project in it's current form suits our short-term needs and will receive limited support. We encourage you to fork the project, and extend it with additional functionality you might need.
-This project was inspired by [blackbox-websocket-exporter](https://github.com/smohsensh/blackbox-websocket-exporter)
+Please note that this tool is in the early development stage and should not be used to influence critical business decisions.
+The project in its current form suits our short-term needs and will receive limited support. We encourage you to fork the project and extend it with additional functionality you might need.
 
 ## Development
-It is recommended that you install [pre-commit](https://pre-commit.com/) so that automated linting and formatting checks are performed before each commit. Run:
+You should install [pre-commit](https://pre-commit.com/) so that automated linting and formatting checks are performed before each commit. 
+
+Run:
 ```bash
 pip install pre-commit
 pre-commit install
 ```
 ### Running locally
 1. Make sure you have python3 installed (>3.9.10)
-2. Set-up your python environment
+2. Set up your python environment
 ```bash
 pip3 install virtualenv
 virtualenv venv

config/exporter_example/config.yml

@@ -1,15 +1,24 @@
-blockchain: "Example Chain"             # Name of blockchain i.e "Ethereum"
-chain_id: 1                             # Chain ID, can be found at https://chainlist.org/
-network_name: "Example"                 # Name of the blokckchain network i.e Rinkeby
-network_type: "Example"                 # Type of the blockchain network, only two values allowed (Mainnet|Testnet)
-connection_timeout_seconds : 5          # Amount in seconds before websocket client drops a connection that has not finished connection handshake and labels the RPC node as unhealhty.
-slo:                                    # Metrics below 'slo' will only affect `ws_rpc_sla_compliant` metric. In case any of the threshold are breached, this metric will be set to 0 for that perticular RPC endpoint.
-  max_pool_deviation_block_count: 10    # Threshold for max allowed deviation in block height, compared to the node with the highest block count in the pool.
-  max_response_latency_miliseconds: 500 # Threshold for max allowed latency.
-endpoints:                              # List of endpoints with their metadata.
-  - url: wss://example-rpc-1.com/ws     # RPC Endpoint (Must start with wss://)
-    provider: Provider1                 # Provider (Must be present in allowed providers list. Please check src/settings.py line 24) The purpose is to make sure we do not have same providers spelled differently
-  - url: wss://example-rpc-2.com/ws
+blockchain: "Example Chain" # Name of blockchain i.e "Ethereum"
+chain_id: 1              # Chain ID, can be found at https://chainlist.org/
+network_name: "Example"  # Name of the blokckchain network i.e Rinkeby
+network_type: "Example"  # Type of the blockchain network, only two values allowed (Mainnet|Testnet)
+connection_parameters:
+  open_timeout: 6 # Timeout when opening websocket connection
+  close_timeout: 1 # Timeout when closing websocket connection
+  response_timeout: 5 # Timeout when waiting for a websocket message response
+  ping_interval: 6 # Liveness ping intervals
+  ping_timeout: 3 # Liveness ping timeout
+collector: "evm" # This will load different collectors based on what mode exporter will run with Supported modes are: "evm", "solana", "conflux", "cardano"
+endpoints: # List of endpoints with their metadata.
+  - ws_url: wss://example-rpc-1.com/ws # RPC Endpoint websocket endpoint (Must start with wss://)
+    https_url: https://example-rpc-1.com/rpc    # RPC Endpoint https endpoint (Must be valid https:// domain)
+    provider: Provider1 # Provider (Must be present in allowed providers list. Please check src/settings.py line 24) The purpose is to make sure we do not have same providers spelled differently                       
+  - ws_url: wss://example-rpc-2.com/ws
+    https_url: https://example-rpc-2.com/rpc
     provider: Provider2
-  - url: wss://example-rpc-3.com/ws
+  - ws_url: wss://example-rpc-3.com/ws
+    https_url: https://example-rpc-3.com/rpc
     provider: Provider3
+
+
+

config/exporter_example/validation.yml

@@ -3,7 +3,3 @@ allowed_providers: # When adding endpoints, these are the only values you will b
   - Provider1
   - Provider2
   - Provider3
-allowed_blockchains: # When setting Blockchain key in your config file, it will be validate against this list.`
-  - Example Chain
-  - Example Chain2
-  - Example Chain3

requirements.txt

@@ -1,4 +1,8 @@
-websockets==10.2
+
 prometheus-client==0.13.1
 pyyaml==6.0
-schema==0.7.5
+schema==0.7.5
+websockets==10.3
+web3==v6.0.0-beta.4
+solana==0.25.1
+conflux-web3==1.0.0b1

src/__init__.py

@@ -0,0 +1,40 @@
+from settings import cfg, logger
+import json
+from collectors.ws import websocket_collector
+from helpers import strip_url
+
+
+class cardano_collector():
+
+    def __init__(self, url, provider):
+        self.labels = ['url', 'provider', 'blockchain', 'network_name', 'network_type', 'evmChainID']
+        self.labels_values = [url, provider, cfg.blockchain, cfg.network_name, cfg.network_type]
+        self.url = url
+        self.ws_collector = websocket_collector(url, provider)
+
+    def _get_block_height(self):
+        blk_height_payload = {
+            "type": "jsonwsp/request",
+            "version": "1.0",
+            "servicename": "ogmios",
+            "methodname": "Query",
+            "args": {
+                "query": "blockHeight"
+            }
+        }
+        try:
+            return json.loads(self.ws_collector.query(blk_height_payload))['result']
+        except KeyError as err:
+            logger.error("Failed to fetch block height for {}, error: {}".format(strip_url(self.url), err))
+
+    def probe(self, metrics):
+        try:
+            alive = self.ws_collector.get_liveliness()
+            if alive:
+                metrics['ws_rpc_health'].add_metric(self.labels_values, True)
+                metrics['ws_rpc_latency'].add_metric(self.labels_values, self.ws_collector.get_latency())
+                metrics['ws_rpc_block_height'].add_metric(self.labels_values, self._get_block_height())
+            else:
+                metrics['ws_rpc_health'].add_metric(self.labels_values, False)
+        except Exception as exc:
+            logger.error("Failed probing {} with error: {}".format(strip_url(self.url), exc))