Skip to content

Conversation

@raghu999
Copy link

@raghu999 raghu999 commented Nov 8, 2025

Description

This PR introduces an Adaptive Request Concurrency (ARC) controller to the exporterhelper.

When enabled via the new sending_queue.arc.enabled flag, this controller dynamically manages the number of concurrent export requests, effectively overriding the static num_consumers setting. It adjusts the concurrency limit based on observed RTT (Round-Trip Time) and backpressure signals (e.g., HTTP 429/503, gRPC ResourceExhausted/Unavailable).

The controller follows an AIMD (Additive Increase, Multiplicative Decrease) pattern to find the optimal concurrency limit, maximizing throughput during healthy operation and automatically backing off upon detecting export failures or RTT spikes.

This feature is disabled by default and introduces no behavior change unless explicitly enabled. It also adds a new set of otelcol_exporter_arc_* metrics (detailed in the documentation) for observing its behavior.

Link to tracking issue

Fixes #14080

Testing

  • Added comprehensive unit tests for the core ARC logic in internal/arc/controller_test.go, covering additive increase, multiplicative decrease (TestAdjustIncreaseAndDecrease), and the cold-start backoff heuristic (TestEarlyBackoffOnColdStart).
  • Added specific unit tests for the new shrinkSem (a custom shrinkable semaphore) to validate its concurrency, prioritization, and shutdown safety.
  • Added a critical test (TestController_Shutdown_UnblocksWaiters) to ensure that any goroutines blocked on Acquire are correctly unblocked with a shutdown error, preventing collector hangs.
  • Added a new integration test in internal/queue_sender_test.go (TestQueueSender_ArcAcquireWaitMetric) that validates the end-to-end flow. It confirms that when the limit is reached, new requests block on Acquire and the exporter_arc_acquire_wait_ms metric records the wait time.
  • Added unit tests for the new internal/experr/back_pressure.go utility to verify its detection logic.

Documentation

  • Updated exporterhelper/README.md to include the new sending_queue.arc block with all its configuration options.
  • Updated exporterhelper/metadata.yaml to define all new otelcol_exporter_arc_* metrics, which are in turn reflected in the generated documentation.md.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Nov 8, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: raghu999 / name: Raghu999 (27537d5)

@raghu999
Copy link
Author

raghu999 commented Nov 8, 2025

# HELP otelcol_exporter_arc_acquire_wait_ms_milliseconds Time a worker waited to acquire an ARC permit. [Alpha]
# TYPE otelcol_exporter_arc_acquire_wait_ms_milliseconds histogram
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="0"} 114
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="5"} 114
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="10"} 114
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="25"} 114
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="50"} 115
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="75"} 115
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="100"} 115
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="250"} 117
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="500"} 125
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="750"} 133
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="1000"} 137
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="2500"} 156
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="5000"} 210
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="7500"} 221
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="10000"} 221
otelcol_exporter_arc_acquire_wait_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="+Inf"} 221
otelcol_exporter_arc_acquire_wait_ms_milliseconds_sum{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version=""} 301819
otelcol_exporter_arc_acquire_wait_ms_milliseconds_count{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version=""} 221
# HELP otelcol_exporter_arc_limit Current ARC dynamic concurrency limit. [Alpha]
# TYPE otelcol_exporter_arc_limit gauge
otelcol_exporter_arc_limit{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version=""} 10
# HELP otelcol_exporter_arc_limit_changes_total Number of times ARC changed its concurrency limit. [Alpha]
# TYPE otelcol_exporter_arc_limit_changes_total counter
otelcol_exporter_arc_limit_changes_total{data_type="traces",direction="up",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version=""} 9
# HELP otelcol_exporter_arc_permits_in_use Number of permits currently acquired. [Alpha]
# TYPE otelcol_exporter_arc_permits_in_use gauge
otelcol_exporter_arc_permits_in_use{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version=""} 10
# HELP otelcol_exporter_arc_rtt_ms_milliseconds Request round-trip-time measured by ARC (from permit acquire to release). [Alpha]
# TYPE otelcol_exporter_arc_rtt_ms_milliseconds histogram
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="0"} 0
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="5"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="10"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="25"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="50"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="75"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="100"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="250"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="500"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="750"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="1000"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="2500"} 11
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="5000"} 109
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="7500"} 205
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="10000"} 206
otelcol_exporter_arc_rtt_ms_milliseconds_bucket{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version="",le="+Inf"} 211
otelcol_exporter_arc_rtt_ms_milliseconds_sum{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version=""} 1.045341e+06
otelcol_exporter_arc_rtt_ms_milliseconds_count{data_type="traces",exporter="otlp/e2e_test",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_schema_url="",otel_scope_version=""} 211
go tool -modfile /Users/rchall201/work/observability/unified-ingest/platform/opentelemetry-collector/internal/tools/go.mod gotestsum --packages="./..." -- -timeout 240s  -race
✓  internal/hosttest (cached)
∅  internal/oteltest
✓  internal/experr (cached)
✓  internal/metadatatest (cached)
✓  internal/metadata (cached)
∅  internal/requesttest
✓  internal/queue (cached)
✓  internal/request (cached)
✓  internal/sender (cached)
✓  internal/sendertest (cached)
✓  internal/queuebatch (cached)
∅  internal/storagetest
✓  internal/sizer (cached)
✓  . (1.415s)
✓  internal/arc (1.918s)
✓  internal (2.517s)

DONE 412 tests in 2.570s

Added the test results and generated metrics

}
}

// Acquire obtains a permit unless ARC is disabled or the context is cancelled.
Copy link
Contributor

@jmacd jmacd Nov 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main feedback is this: we have here a large piece of code, it looks good, but it is very specific and opinionated. I would prefer to see this become an extension implementation. This PR is great, I just think it belongs in //extensions.

See #13902 explaining how extension APIs are added, it would begin with //extensions/extensioncontroller APIs, for example, then the bulk of the code would land in //extensions/arccontrollerextension (could be in contrib).

Could we invent an extension point to let you insert your ARC controller into the send pipeline for all exporters? Then, exporterhelper would only need to load and insert the extensions into the pipeline.

Copy link
Author

@raghu999 raghu999 Nov 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jmacd, thanks for the detailed feedback on moving this to an extension.

My original thinking was that putting ARC in exporterhelper would make it easier to enable by default for any exporter that is using sending queue after ARC is battle-tested.

This isn't just a theoretical feature for us; it's based on our own painful experience of overwhelming and taking down downstream systems. We operate at a large scale, using both Vector and OTel to ingest hundreds of TBs of telemetry data, and we've seen the critical need for this kind of adaptive backpressure.

This experience is what led us to look for proven solutions. The Netflix tech blog (https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581) inspired Vector's implementation (https://vector.dev/blog/adaptive-request-concurrency/). We've seen the wins from this model firsthand in our own Vector deployments, and it's a key reason they enable it by default. My goal is to bring that same proven stability to OTel.

That said, I'm completely fine with moving this to an extension if the team prefers that. I did talk with @atoulme at KubeCon about how the sending_queue manages its goroutine pool, and I believe this implementation fits in correctly.

Since it's a big architectural choice, I will wait for feedback from the other exporterhelper code owners. @bogdandrutu and @dmitryax, I'd strongly request you read the two blog posts above if you have a moment. They provide the full context for why this feature is so important and why I'm proposing it.

I'll hold off on the refactoring until you've all had a chance to weigh in. Please let me know which direction you'd like me to take. I'm happy to go whichever way the project prefers.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @raghu999. We are currently focused on Collector stabilization, which includes relying on the configuration already provided by exporterhelper. The change here adds a significant amount of new logic with its own configuration interface, and even though the intention is great, it does conflict with the stabilization effort.

Did you have a chance to build the Collector with this change and verify that it solves the problems you’re running into? Having more details on how this helps with the challenges from actual experiments would help us figure out if and how this functionality could be incorporated into the Collector.

Copy link
Author

@raghu999 raghu999 Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @dmitryax, totally understand and appreciate the focus on stabilization. My intention with this PR is to support that effort: today the sending_queue can become a stability bottleneck when static concurrency limits are misconfigured. ARC gives the collector a built-in safety valve that adapts to downstream capacity and prevents overload conditions.

To your question: yes, I have already built and verified the implementation.

So far, we’ve tested this with otlpgrpc at a smaller scale to confirm that configuration parameters are picked up correctly and the control logic activates as expected.

For validation:

  • Production: We are currently in a holiday code freeze. Immediately after the Thanksgiving moratorium, I will deploy this to a couple of our production tenants and run it against multiple real downstream exporters. I will share the runtime metrics and results as soon as they’re available.
  • Benchmarks: I am currently adding a benchmark suite (BenchmarkARC_vs_Static) to simulate "flapping" downstream destinations (latency spikes and 429s). This will allow us to reproducibly demonstrate that ARC improves stability compared to static limits under controlled load.

A few notes on code impact:

  • Strict opt-in: All logic is gated behind qCfg.Arc.Enabled (defaults to false). In NewQueueSender, if ARC is disabled, we return the standard QueueBatch struct immediately. The ARC controller initialization and the export function wrapper are completely skipped, ensuring zero runtime overhead or code path changes for existing users.
  • Config isolation: The new arc configuration block is fully self-contained and does not alter any existing queue/batch semantics.
  • Feature Gate option: As suggested by @atoulme during KubeCon, I’m fully open to putting the entire ARC functionality behind an Alpha Feature Gate (e.g., exporter.adaptiveRequestConcurrency). If the team feels that would be the safest path during the stabilization period, I can add that in this PR.

I’ll follow up with the benchmark data shortly. Happy to iterate on the Feature Gate approach if that helps merge this safely during the freeze!

@raghu999 raghu999 requested a review from jmacd November 20, 2025 22:26
@atoulme atoulme added the discussion-needed Community discussion needed label Nov 24, 2025
@atoulme
Copy link
Contributor

atoulme commented Nov 24, 2025

This was brought up at KubeCon. I think we need to discuss this at a SIG meeting.

@codspeed-hq
Copy link

codspeed-hq bot commented Nov 25, 2025

CodSpeed Performance Report

Merging #14144 will improve performances by 35.05%

Comparing raghu999:arc-feature (b541b7a) with main (c44a402)

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

⚡ 1 improvement
✅ 115 untouched

Benchmarks breakdown

Benchmark BASE HEAD Change
BenchmarkSplittingBasedOnItemCountHugeLogs 46.7 ms 34.6 ms +35.05%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

discussion-needed Community discussion needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Adaptive Request Concurrency (ARC) for HTTP and gRPC Exporters

4 participants