Metrics to Track for py-libp2p

[seetadev]

1. Metrics to Track for py-libp2p

To monitor a live network, collect metrics at 3 levels:

a) Node-Level (per peer instance)

• CPU / Memory / Network

  • CPUUtilization
  • MemoryUtilization
  • NetworkIn / NetworkOut

• Open Connections

  • Active peer count
  • Connection success/failure rate

• Libp2p-Specific Node Stats

  • Number of connected peers
  • Number of active streams
  • Connection establishment time

---

b) Protocol-Level (DHT, PubSub, Relays)

• DHT

  • Avg. lookup latency (dht_lookup_latency)
  • Success/failure rate of DHT queries (dht_query_success_rate)
  • Routing table size over time (dht_routing_table_size)

• PubSub

  • Messages sent / received per second (pubsub_msg_rate)
  • Message delivery latency (pubsub_msg_latency)
  • Dropped messages (pubsub_msg_drop_count)

• Relay / NAT

  • Relay connection usage (relay_conn_count)
  • NAT traversal success rate (nat_holepunch_success_rate)

---

c) Cluster-Level (whole network)

• Total active peers across the cluster
• Churn rate (joins/leaves per minute)
• Error counts (protocol errors, dropped connections)

---

2. AWS Setup

We’ll use CloudWatch for collection and Grafana for visualization.

Step 1 – Instrument py-libp2p

Add a small metrics exporter in your py-libp2p nodes:

• Use Prometheus Python client (prometheus_client) to expose metrics at /metrics.
• Example:

from prometheus_client import start_http_server, Gauge, Counter
import time

connected_peers = Gauge('libp2p_connected_peers', 'Number of connected peers')
dht_lookup_latency = Gauge('libp2p_dht_lookup_latency', 'Average DHT lookup latency')
pubsub_msg_rate = Counter('libp2p_pubsub_msg_rate', 'Pubsub messages per second')

def update_metrics():
    # Update with real py-libp2p values
    connected_peers.set(len(node.get_connected_peers()))
    dht_lookup_latency.set(get_avg_lookup_latency())
    pubsub_msg_rate.inc()

if __name__ == "__main__":
    start_http_server(8000)
    while True:
        update_metrics()
        time.sleep(5)

Step 2 – Push to CloudWatch

Use the CloudWatch Agent in each container/instance:

• Configure to scrape the Prometheus endpoint (/metrics).
• Push to Custom CloudWatch Metrics namespace (py-libp2p).

---

3. Grafana Dashboard Layout

Use Amazon Managed Grafana (integrated with CloudWatch).

Panels to Create:

Top Row – Network Health

• Active peers (line chart) – libp2p_connected_peers
• Churn rate (bar) – peer joins/leaves

Second Row – DHT Performance

• DHT Lookup Latency (p95, avg) – libp2p_dht_lookup_latency
• DHT Query Success % (gauge) – libp2p_dht_query_success_rate

Third Row – PubSub Throughput

• Messages per second – libp2p_pubsub_msg_rate
• Message Latency – libp2p_pubsub_msg_latency

Fourth Row – Resource Usage

• CPU/Memory Usage per Node (heatmap) – CPUUtilization, MemoryUtilization
• Network Traffic per Region – NetworkIn / NetworkOut

Fifth Row – Failures & Errors

• Relay / NAT connection success rates
• Protocol errors over time

---

4. Alerting

Set CloudWatch alarms for:

• Drop in active peers > X% in Y mins.
• DHT lookup latency > threshold (e.g., 500ms).
• Message loss > 1%.
• CPU/Memory usage exceeding limits.

Integrate with SNS or Slack/Webhook for real-time alerts.

[seetadev]

Also, try: Long-Running Stability Test

Goal:
Check for memory leaks, stale connections, or long-term degradation.

Setup:

Keep 500 peers running continuously for 2 days on cheap spot instances.

Use CloudWatch + Prometheus/Grafana for resource tracking.

Simulate light ongoing activity (occasional pubsub + DHT queries).

[seetadev]

Large-Scale Peer Discovery & DHT Benchmark

Goal:
See how py-libp2p handles large peer networks (500-1000 nodes) and DHT operations under load.

Setup:

Deploy 1,000+ py-libp2p peers on ECS (Fargate) or EKS (Kubernetes).

Use multiple AWS regions to add realistic latency.

Perform parallel DHT lookups (PUT/GET) and peer discovery under churn (nodes joining/leaving).

Metrics to collect:

Average lookup latency (ms).

Routing table convergence time.

Success rate under churn.