Automatically setup basic otel/prometheus metrics

[negz]

What problem are you facing?

It would be neat if we could automatically wrap Functions with some basic observability (e.g. metrics, tracing). Ideally we'll need a format that could be consistent across Functions built in other in-demand languages - e.g. Python.

How could Crossplane help solve your problem?

There's a bunch of stuff in https://github.com/grpc-ecosystem - perhaps this SDK should start a gRPC server that uses one of the interceptors there to add at least basic Prometheus-style metrics?

[phisco]

for sure I'd go with prometheus metrics, https://github.com/grpc-ecosystem/go-grpc-middleware/tree/main/providers/prometheus feels like the suggested way to do that.

[negz]

@phisco I think there's an otel equivalent that gives traces and metrics. Could be worth considering. To be clear, the metrics are exposed in Prom-style /metrics, it's just a different library doing it. That said, last time I checked the otel metrics packages were pretty unstable API wise.

[phisco]

yeah, not so straightforward: https://github.com/grpc-ecosystem/go-grpc-middleware/blob/71d7422112b1d7fadd4b8bf12a6f33ba6d22e98e/examples/server/main.go#L77 .

[FooBarWidget]

Adding to this, it would be great if functions have pprof enabled by default. We have functions (most notably function-kcl) that use an obscene amount of memory but we don't know why.

[sagarwal16-atlassian]

@phisco @negz please review the PR