Add OpenTelemetry Prometheus metrics with configurable port/host

[klaudworks]

Summary

• Add OpenTelemetry-based Prometheus metrics with configurable s3proxy.metrics.port and s3proxy.metrics.host for Kubernetes ServiceMonitor compatibility
• Implement http.server.request.duration histogram per OpenTelemetry HTTP Metrics Semantic Conventions

Metric

• http.server.request.duration - Histogram of HTTP request duration in seconds
  • Labels: http.request.method, url.scheme, http.response.status_code, s3.operation, s3.bucket

Configuration

s3proxy.metrics.enabled=true
s3proxy.metrics.port=9090
s3proxy.metrics.host=0.0.0.0

Sample Metrics:

# HELP http_server_request_duration_seconds Duration of HTTP server requests
# TYPE http_server_request_duration_seconds histogram
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",otel_scope_name="org.gaul.s3proxy",s3_bucket="kimi-dev-backup",s3_operation="GetObject",url_scheme="http",le="0.005"} 0
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",otel_scope_name="org.gaul.s3proxy",s3_bucket="kimi-dev-backup",s3_operation="GetObject",url_scheme="http",le="0.01"} 0
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",otel_scope_name="org.gaul.s3proxy",s3_bucket="kimi-dev-backup",s3_operation="GetObject",url_scheme="http",le="0.025"} 13
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",otel_scope_name="org.gaul.s3proxy",s3_bucket="kimi-dev-backup",s3_operation="GetObject",url_scheme="http",le="0.05"} 142
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",otel_scope_name="org.gaul.s3proxy",s3_bucket="kimi-dev-backup",s3_operation="GetObject",url_scheme="http",le="0.075"} 251
...
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="200",otel_scope_name="org.gaul.s3proxy",s3_bucket="kimi-dev-backup",s3_operation="GetObject",url_scheme="http"} 549

Related Issues

• Feature: Metrics for s3 operations #945 - Implementation discussion
• Does s3proxy expose any meaningful metrics? #780 - Prior metrics discussion
• Disable Jetty shading to allow OpenTelemetry instrumentation #923 - Prior metrics discussion

[gaul]

@hermandavid does this PR address your use case?

[hermandavid]

Metrics are useful, but this does not much relate to #923 since there are no traces being sent.

Having OpenTelemetry in the project helps a lot though. I can follow up with a PR that reuses existing Jetty OTEL instrumentation and just switch it to the shaded package, so we won't have to touch shading to make it work.

[klaudworks]

This PR just adds the minimal metrics setup for compliance with the OTEL spec. This covers my current observability needs already.

[gaul]

Thank you for your contribution @klaudworks!