Observe Leios experiments with Grafana

[bladyjoker]

Grafana is a defacto industry standard for observability. It features Grafana as a web frontend for viewing data (metrics and logs), loki as log storage, prometheus as metrics storage and alloy as a configuration format for wiring all this together.

Our experiments consists of patched cardano-nodes, immdb-server and potentially toxiproxy running together in various configurations.

The goal of this task is to have a Grafana stack enabled with a Leios dashboard showing the relevant information about Leios experiments, specifically:

• All logging is centrally collected and viewable through our local Grafana instance
  • cardano-node
  • immdb-server
  • journald
• All metrics are centrally collected and viewable through our local Grafana instance
  • cardano-node
• There's a simple Leios Dashboard that includes Leios related logs and metrics

[bladyjoker]

The dashboard stolen from our Playground https://playground.monitoring.aws.iohkdev.io/ as applied to our local dev net

NOTES:

1. Dashboards are generally parametrized under some variables, and these are not aligned between environments (for example the playground one had environment and instance labels, which we don't).

• TODO: Aligning on this would make it much easier to reuse (one more argument for following OpenTelemetry metrics model)

[ch1bo]

@bladyjoker How do you collect logs from cardano-node? Is there some setup / integration code that could be linked so I can have a look how you created that dashboard?