Provide some documentation on tracer's logs and metrics

Author: abailly

compose/README.md

@@ -75,7 +75,7 @@ Docker and `make` are the only requirement to build and run your own testnets.
 
 ### Local
 
-- Build the `cardano-node`, `config` and `sidecar` container images
+- Build the `cardano-node`, `cardano-tracer`, `config` and `sidecar` container images
 
   ```
   make build testnet=example_10.2.1
@@ -86,14 +86,15 @@ Docker and `make` are the only requirement to build and run your own testnets.
 
 ### Antithesis
 
-- Build the `cardano-node`, `config` and `sidecar` container images for the Antithesis container registry
+- Build the `cardano-node`, `cardano-tracer`, `config` and `sidecar` container images for the Antithesis container registry
 
   ```
   make build testnet=example_10.2.1 registry=us-central1-docker.pkg.dev/molten-verve-216720/cardano-repository/
   ```
 
 > [!IMPORTANT]
 > Always supply the registry argument when building for Antithesis.
+> It's not necessary to log into the registry for building, but it is necessary for running antithesis tests.
 
 ## Run
 
@@ -191,7 +192,7 @@ Docker and `make` are the only requirement to build and run your own testnets.
   cat credentials.json | docker login -u _json_key https://us-central1-docker.pkg.dev --password-stdin
   ```
 
-- Push the `cardano-node`, `config` and `sidecar` container images
+- Push the `cardano-node`, `cardano-tracer`, `config` and `sidecar` container images
 
   ```
   make push testnet=example_10.2.1 registry=us-central1-docker.pkg.dev/molten-verve-216720/cardano-repository/
@@ -230,6 +231,67 @@ Docker and `make` are the only requirement to build and run your own testnets.
 > [!TIP]
 > The commands above assume a testnet of 3 pools, increase the `{1..3}` if needed.
 
+### Logging & Metrics
+
+The `tracer` container which runs alongside the Cardano cluster aggregates logs and metrics for each of the running nodes. The details of the configuration are out of scope for this document, please checkout the [cardano-tracer](https://github.com/IntersectMBO/cardano-node/tree/master/cardano-tracer) and [new tracing system](https://developers.cardano.org/docs/get-started/cardano-node/new-tracing-system/new-tracing-system) documentation. The default example configuration aggregates some interesting logs and expose Prometheus metrics on port 4000.
+
+- Query prometheus metrics (each node has its own path for metrics):
+
+  ```bash
+  curl -s http://localhost:4000/p1example-3001 | grep block
+  ```
+
+  ```
+  # TYPE cardano_node_metrics_served_block_latest_int gauge
+  cardano_node_metrics_served_block_latest_int 4
+  # TYPE cardano_node_metrics_blockfetchclient_blocksize_int gauge
+  cardano_node_metrics_blockfetchclient_blocksize_int 862
+  # TYPE cardano_node_metrics_blocksForged_int gauge
+  cardano_node_metrics_blocksForged_int 2
+  cardano_node_metrics_blockfetchclient_blockdelay_real 4.574799e-3
+  # TYPE cardano_node_metrics_blockNum_int gauge
+  cardano_node_metrics_blockNum_int 4
+  # TYPE cardano_node_metrics_served_block_counter counter
+  cardano_node_metrics_served_block_counter 4
+  ```
+
+- List available logs (logs from each node are dumped into a file on a shared volume, the name of the file contains timestamp):
+
+  ```bash
+  docker exec -ti tracer ls -l  /opt/cardano-tracer
+  ```
+
+  ```
+  drwxr-xr-x 2 cardano cardano 4096 Apr 13 07:42 p1.example_3001
+  drwxr-xr-x 2 cardano cardano 4096 Apr 13 07:42 p2.example_3001
+  drwxr-xr-x 2 cardano cardano 4096 Apr 13 07:42 p3.example_3001
+  srwxr-xr-x 1 cardano cardano    0 Apr 13 07:42 tracer.socket
+  ```
+
+  ```bash
+  for i in {1..3}; do docker exec -ti tracer ls -l  /opt/cardano-tracer/p${i}.example_3001/ ; done
+  ```
+
+  ```
+  total 404
+  -rw-r--r-- 1 cardano cardano 411911 Apr 13 07:49 node-2025-04-13T07-42-11.json
+  total 376
+  -rw-r--r-- 1 cardano cardano 379328 Apr 13 07:49 node-2025-04-13T07-42-11.json
+  total 380
+  -rw-r--r-- 1 cardano cardano 384804 Apr 13 07:49 node-2025-04-13T07-42-10.json
+  ```
+
+- Follow logs from a specific node:
+
+  ```
+  $ docker exec -ti tracer tail -f  /opt/cardano-tracer/p2.example_3001/node-2025-04-13T07-42-11.json
+  {"at":"2025-04-13T07:47:57.001449232Z","ns":["Forge","Loop","NodeNotLeader"],"data":{"kind":"TraceNodeNotLeader","slot":417},"sev":"Info","thread":"41","host":"p2.example"}
+  {"at":"2025-04-13T07:47:58.001200722Z","ns":["Forge","Loop","StartLeadershipCheckPlus"],"data":{"chainDensity":3.7974683544303798610197731022708467207849025726318359375e-2,"delegMapSize":3,"kind":"TraceStartLeadershipCheck","slot":418,"utxoSize":3},"sev":"Info","thread":"41","host":"p2.example"}
+  {"at":"2025-04-13T07:47:58.001427133Z","ns":["Forge","StateInfo","StateInfo"],"data":{"credentials":"Cardano","endPeriod":0,"evolution":62,"kind":"KESInfo","startPeriod":0},"sev":"Info","thread":"41","host":"p2.example"}
+  {"at":"2025-04-13T07:47:58.001722819Z","ns":["Forge","Loop","NodeNotLeader"],"data":{"kind":"TraceNodeNotLeader","slot":418},"sev":"Info","thread":"41","host":"p2.example"}
+  ...
+  ```
+
 ## Appendix
 
 - [Antithesis Documentation](https://antithesis.com/docs/)