OpenMetrics server handles connections sequentially; /metrics can hang and cause widespread scrape timeouts

[QuentinBtd]

Summary

We are seeing intermittent-to-frequent /metrics scrape timeouts with the EKS add-on (aws-network-sonar-agent:v1.1.4-eksbuild.1) even when OpenMetrics is enabled and listening.
After investigation, this appears to be a server-side concurrency issue in the OpenMetrics HTTP server implementation: accepted connections are handled sequentially, not per-connection in separate tasks.

Environment

• EKS add-on image: 602401143452.dkr.ecr.eu-west-3.amazonaws.com/aws-network-sonar-agent:v1.1.4-eksbuild.1
• OpenMetrics env:
  • OPEN_METRICS=on
  • OPEN_METRICS_ADDRESS=0.0.0.0
  • OPEN_METRICS_PORT=9109
• Pods use hostNetwork: true, metrics port exposed as 9109.

Symptoms observed

• TCP connect succeeds to :9109, but HTTP response often never arrives:
  • curl ... http://<pod-ip>:9109/metrics -> timeout (no headers/body)
• vmagent logs repeatedly show:
  • Client.Timeout exceeded while awaiting headers
  • context deadline exceeded
• Killing/recreating a problematic pod often restores /metrics temporarily.
• Port-forward behavior was misleading due to browser cache:
  • normal tab looked OK until forced refresh/disabled cache; then timed out too.
  • incognito/curl timed out consistently.

Why this looks like a code-level concurrency bug

In nfm-controller/src/open_metrics/server.rs, accepted connections are processed sequentially:

• accept_connections(...) calls handle_connection(...).await directly
  • around lines 317-324
• handle_connection(...) runs http1::Builder::new().serve_connection(...)
  • around line 281
    Because each accepted connection is awaited inline (no tokio::spawn per connection), one slow/stuck keep-alive connection can block servicing of subsequent connections at application level.
    This matches what we observe in production:
• many clients can establish TCP,
• but most HTTP requests time out waiting for headers.
  Also, each /metrics request does synchronous work (provider.update_metrics()), and providers call external commands via Command::new(...).output(), which can increase request latency and amplify blocking effects.
  Relevant files:
• nfm-controller/src/open_metrics/server.rs
• nfm-controller/src/open_metrics/providers/interface_metrics_provider.rs
• nfm-controller/src/utils/command_runner.rs

Expected behavior

• Concurrent requests to /metrics should be served independently.
• A slow or stuck request should not block all other scrapes.

Actual behavior

• Requests are effectively serialized.
• Under load or degraded conditions, /metrics becomes unresponsive for many/all scrapers until pod restart.

[nahuelfilipuzzi]

Thanks for sharing this issue.

Will look into this. The reason of handling one request at a time was designed like that to reduce the impact on the host for environments with many pods. We can add a parameter to remove this limitation.

[nahuelfilipuzzi]

See: #81