[serving] OTel Metrics Documentation (#6352)

* Updating collecting metrics steps

* remove collector installation steps and separate out shared metrics into a snippet

* combine notes

* update nav to remove stutter

* include webhook metrics

* update common metrics with attributes and include serving metrics

* update notice banners for eventing and serving

* fix typos

* Update docs/eventing/observability/metrics/collecting-metrics.md

Co-authored-by: Calum Murray <[email protected]>

---------

Co-authored-by: Calum Murray <[email protected]>

Author: dprotaso

config/nav.yml

@@ -188,7 +188,7 @@ nav:
         - Configuring logging: serving/observability/logging/config-logging.md
         - Configuring Request logging: serving/observability/logging/request-logging.md
         - Collecting metrics: serving/observability/metrics/collecting-metrics.md
-        - Knative Serving metrics: serving/observability/metrics/serving-metrics.md
+        - Metrics Reference: serving/observability/metrics/serving-metrics.md
       # Serving - troubleshooting
       - Troubleshooting:
         - Debugging application issues: serving/troubleshooting/debugging-application-issues.md
@@ -307,7 +307,7 @@ nav:
         - Collecting logs: eventing/observability/logging/collecting-logs.md
         - Configuring logging: eventing/observability/logging/config-logging.md
         - Collecting metrics: eventing/observability/metrics/collecting-metrics.md
-        - Knative Eventing metrics: eventing/observability/metrics/eventing-metrics.md
+        - Metrics Reference: eventing/observability/metrics/eventing-metrics.md
       - Features:
         - About Eventing features: eventing/features/README.md
         - DeliverySpec.Timeout field: eventing/features/delivery-timeout.md

docs/eventing/observability/metrics/collecting-metrics.md

@@ -6,3 +6,40 @@ function: how-to
 ---
 
 --8<-- "collecting-metrics.md"
+
+### Enabling Metric Collection
+
+1. To enable prometheus metrics collection you will want to update `config-observability` ConfigMap and set the `metrics-protocol` to `prometheus`.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config-observability
+  namespace: knative-eventing
+data:
+    # metrics-protocol field specifies the protocol used when exporting metrics
+    # It supports either 'none' (the default), 'prometheus', 'http/protobuf' (OTLP HTTP), 'grpc' (OTLP gRPC)
+    metrics-protocol: prometheus
+
+    tracing-protocol:      http/protobuf
+    tracing-endpoint:      http://jaeger-collector.observability.svc:4318/v1/traces
+    tracing-sampling-rate: "1"
+```
+
+### Apply the Eventing Service/Pod Monitors
+
+1. Apply the ServiceMonitors/PodMonitors to collect metrics from Knative Eventing Control Plane
+
+    ```bash
+    kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/eventing-monitors.yaml
+    ```
+### Import Grafana dashboards
+
+1. Grafana dashboards can be imported from the [`monitoring` repository](https://github.com/knative-extensions/monitoring).
+
+1. If you are using the Grafana Helm Chart with the dashboard sidecar enabled (the default), you can load the dashboards by applying the following configmaps.
+
+    ```bash
+    kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/configmap-eventing-dashboard.yaml
+    ```

docs/eventing/observability/metrics/collector.yaml

@@ -5,7 +5,15 @@ components:
 function: reference
 ---
 
-# Knative Eventing metrics
+# Knative Eventing Metrics
+
+!!! warning
+
+    The metrics below have not been updated to reflect our migration from 
+    OpenCensus to OpenTelemetry. We are in the process of updating them.
+
+    These metrics may change as we flush out our migration from OpenCensus 
+    to OpenTelemetry.
 
 Administrators can view metrics for Knative Eventing components.
 
@@ -42,11 +50,6 @@ By aggregating the metrics over the http code, events can be separated into two
 | event_count | Number of events dispatched by the in-memory channel | Counter | container_name<br>event_type=<br>namespace_name=<br>response_code<br>response_code_class<br>unique_name | Dimensionless | Stable
 | event_dispatch_latencies | The time spent dispatching an event from a in-memory Channel | Histogram | container_name<br>event_type<br>namespace_name=<br>response_code<br>response_code_class<br>unique_name | Milliseconds | Stable
 
-!!! note
-    A number of metrics eg. controller, Go runtime and others are omitted here as they are common
-    across most components. For more about these metrics check the
-    [Serving metrics API section](../../../serving/observability/metrics/serving-metrics.md).
-
 ## Eventing sources
 
 Eventing sources are created by users who own the related system, so they can trigger applications with events.
@@ -57,3 +60,5 @@ to verify that events have been delivered from the source side, thus verifying t
 |:-|:-|:-|:-|:-|:-|
 | event_count | Number of events sent by the source | Counter | event_source<br>event_type<br>name<br>namespace_name<br>resource_group<br>response_code<br>response_code_class<br>response_error<br>response_timeout | Dimensionless  | Stable |
 | retry_event_count | Number of events sent by the source in retries | Counter | event_source<br>event_type<br>name<br>namespace_name<br>resource_group<br>response_code<br>response_code_class<br>response_error<br>response_timeout | Dimensionless | Stable
+
+--8<-- "observability-shared-metrics.md"

docs/eventing/observability/metrics/eventing-metrics.md

@@ -6,3 +6,46 @@ function: how-to
 ---
 
 --8<-- "collecting-metrics.md"
+
+
+### Enabling Metric Collection
+
+1. To enable prometheus metrics collection you will want to update `config-observability` ConfigMap and set the `metrics-protocol` to `prometheus`. For request-metrics we recommend setting up pushing metrics to prometheus. This requires enabling the Prometheus OLTP receiver. This is already configured in our monitoring example.
+
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config-observability
+  namespace: knative-serving
+data:
+    # metrics-protocol field specifies the protocol used when exporting metrics
+    # It supports either 'none' (the default), 'prometheus', 'http/protobuf' (OTLP HTTP), 'grpc' (OTLP gRPC)
+    metrics-protocol: prometheus
+
+    # request-metrics-protocol
+    request-metrics-protocol: http/protobuf
+    request-metrics-endpoint: http://knative-kube-prometheus-st-prometheus.observability.svc:9090/api/v1/otlp/v1/metrics
+
+    tracing-protocol:      http/protobuf
+    tracing-endpoint:      http://jaeger-collector.observability.svc:4318/v1/traces
+    tracing-sampling-rate: "1"
+
+```
+
+1. Apply the ServiceMonitors/PodMonitors to collect metrics from Knative Serving Control Plane.
+
+    ```bash
+    kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/serving-monitors.yaml
+    ```
+
+### Import Grafana dashboards
+
+1. Grafana dashboards can be imported from the [`monitoring` repository](https://github.com/knative-extensions/monitoring).
+
+1. If you are using the Grafana Helm Chart with the dashboard sidecar enabled (the default), you can load the dashboards by applying the following configmaps.
+
+    ```bash
+    kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/configmap-serving-dashboard.yaml
+    ```