Improve support for OTEL metrics collection

[balasankarc]

Is your feature request related to a problem? Please describe.

Right now, the OTEL metrics support is limited to API pods, and is implemented by each API pod getting an OTEL collector sidecar. This means metrics from only API pods get collected. Content and Worker pods also emit metrics, and they are missed.

Describe the solution you'd like

What if instead of a sidecar pattern, there is a dedicated OTEL collector deployment to which all the pods send their metrics? Was there a specific reason we went with the sidecar pattern, and having N number of OTEL collector containers, instead of a single one?

If we prefer to stick to the sidecar pattern, we need to bring in those sidecars to content and worker pods also so metrics from them can be collected.

[balasankarc]

My workaround right now is not to use Operator's telemetry.enabled field, but manually deploy an OTEL Collector deployment to the cluster, and make use of env_vars to configure Pulp pods (api, content and worker) to enable OTEL and send metrics to that OTEL Collector deployments.

[balasankarc]

/cc @git-hyagi

[git-hyagi]

What if instead of a sidecar pattern, there is a dedicated OTEL collector deployment to which all the pods send their metrics? Was there a specific reason we went with the sidecar pattern, and having N number of OTEL collector containers, instead of a single one?

hmmm this is something that we tried in the past, but I don't remember why we decided to go with the sidecar. @decko do you remember?

[decko]

Sure! We had a limitation on the metrics side. Given the pulp architecture, with different processes running, the metrics got overwritten sometimes. I think we fixed that later by introducing the worker name attribute, so, initially, I see no issues with the centralized collector instance.

Just remember to size the collector properly to receive the metrics from all components and be consumed by Prometheus or push data into another system.