add control plane telemetry via csv

Author: lukasz-zet

edgeless_telemetry/README.md

@@ -1,37 +1,65 @@
 # Edgeless Telemetry
 
-This is an initial take on telemetry (to be extended / replaced as needed). Uses
-OpenMetrics through this library: https://github.com/prometheus/client_rust.
+Telemetry support for EDGELESS nodes and control plane components.
 
-Dockerfiles for running a prometheus instance and Grafana istance are provided
-with a basic dashboard for a one-node Edgeless s=ystem. Currently used by the
-edgeless_node to provide metrics such as:
-- function_count - number of function instances that are present
-- execution_times_count - number of times a function was executed
-etc.
-- execution_times_bucket - 
+## Components
 
-This has been tested on a local setup with Docker for Mac and edgeless_inabox
-running locally.
+### Data Plane Telemetry (Nodes)
 
-## How to run Prometheus / Grafana as docker containers?
-Make sure you have Docker Compose installed. Navigate to `components` and run:
+Event-based metrics collection for function execution and resource usage. Both targets track the same events but export them differently:
 
-```bash
-docker-compose up --build -d
+**PerformanceTarget** - Collects raw samples from function execution:
+- Stores execution times, transfer times, and log entries per function
+- Periodically sent from nodes to orchestrator via node registration refresh
+- Orchestrator writes to Redis and optionally to CSV file
+- Used for post-processing and analysis of function-level metrics
+
+**PrometheusTarget** - Aggregates metrics for real-time monitoring using e.g. dashboards and alerts:
+- Exposes HTTP endpoint (`metrics_url`) for Prometheus scraping
+- Tracks node-level and function-level metrics in histograms
+- Includes: `function_count`, `execution_times`, `transfer_times`
+- Used for live dashboards (Grafana) and alerting
+
+Both targets receive the same telemetry events from function runtimes:
+- Function lifecycle: instantiate, init, exit
+- Function execution: invocation completed times, transfer times
+- Function logs: correlated to function instances
+
+Uses OpenMetrics through https://github.com/prometheus/client_rust.
+
+### Control Plane Tracer
+
+Lightweight span-based tracer for orchestrator and controller operations. Designed for control plane, not data plane.
+
+Exports traces to CSV format for analyzing orchestrator decision-making and workflow lifecycle.
+
+**API**:
+- `ControlPlaneTracer::new(output_path)` - create tracer (stdout or file)
+- `tracer.start_span(name)` - create root span with correlation ID
+- `span.child(name)` - create child span with parent reference
+- `span.log(level, message)` - log correlated to span
+- Automatic span end on drop (RAII)
+
+**CSV Output Format**:
+```
+timestamp_sec,timestamp_ns,event_type,correlation_id,parent_id,name,level,message
 ```
 
-Then just open `localhost:3000` in your browser and open the `Edgeless default
-dashboard` on the left.
+**Why not OpenTelemetry?**
+
+We implement a minimal subset of OpenTelemetry's tracing functionality tailored for orchestrator 
+observability as we need a simple, low-overhead solution without external dependencies.
+
+For production orchestrator observability with distributed tracing and existing infrastructure integration, use OpenTelemetry instead.
+
+## Running Prometheus / Grafana
 
-## How to add new metric types?
-TODO
+> This dashboard has not been updated for the latest telemetry metrics. Use it as a reference only.
 
-## How to instrument your code?
-Instrumentation is currently added to edgeless_node, check it out to learn more.
+Navigate to `components` and run:
+
+```bash
+docker-compose up --build -d
+```
 
-## Next steps:
-- [x] Grafana dashboard for a cluster of one node
-- [ ] Add function_class as a label for metrics
-- [ ] expand Instructions on how to add metrics
-- [x] default anonymous user
+Open `localhost:3000` and access the Edgeless dashboard.

edgeless_telemetry/examples/control_plane_tracer_usage.rs

@@ -0,0 +1,34 @@
+// Example usage of the control plane tracer for orchestrator metrics
+
+use edgeless_telemetry::control_plane_tracer::ControlPlaneTracer;
+
+fn main() {
+    // create tracer that writes to stdout
+    let tracer = ControlPlaneTracer::new(String::new()).unwrap();
+    
+    // or write to a file
+    // let tracer = ControlPlaneTracer::new("orchestrator_traces.csv".to_string()).unwrap();
+    
+    // start a span for a workflow deployment
+    let deployment_span = tracer.start_span("deploy_workflow");
+    deployment_span.log("info", "starting workflow deployment");
+    
+    {
+        // child span for resource allocation
+        let allocation_span = deployment_span.child("resource_allocation");
+        allocation_span.log("debug", "selecting nodes");
+        allocation_span.log("info", "allocated 3 nodes");
+    }
+    
+    {
+        // child span for function instantiation
+        let instantiation_span = deployment_span.child("function_instantiation");
+        instantiation_span.log("debug", "creating function instances");
+        instantiation_span.log("info", "instantiated 5 functions");
+    }
+    
+    deployment_span.log("info", "workflow deployment completed");
+    
+    // global log without correlation
+    tracer.log("warn", "system load high");
+}