[Exporterhelper Exporter] Enable metadata population in exporterhelper when sending_queue is enabled

[gizas]

Description

Feature: The enablement of the sending_queue in the configuration of the otplhttp exporter does not preserve the metadata information of the initial context.

This pr introduces a new configuration called metadata_keys that will be used in the exporter config

Testing

• Added partionetr_test

• Built my local otlp exporter

Configuration of otlphttp:

otlphttp:
        endpoint: https://endpoint.invalid/_otlp
        timeout: {{ .http_request_timeout | quote }}
        sending_queue:
          enabled: true
          num_consumers: {{ .concurrentRequests }}
          wait_for_result: false
          block_on_overflow: true  # false = data loss on queue full
          sizer: requests
          queue_size: {{ .queuedRequests }}
          metadata_keys:
           - project-id

See "ProjectID":"local" that is populated below in my logs

{"log.level":"info","@timestamp":"2025-11-06T15:51:49.955Z","message":"ecpRoutingRoundTripper: after URL modification","resource":{"cloud.availability_zone":"local","k8s.namespace.name":"motel-index-collector","k8s.node.name":"hotel-worker2","k8s.pod.name":"motel-index-collector-local-58484b67c8-l4vrt","k8s.pod.uid":"1a8f6fad-91b6-413a-a929-6d004ca489b3","orchestrator.cluster.name":"default","orchestrator.deploymentslice":"","orchestrator.environment":"default","service.instance.id":"ecf43bce-5aac-444c-b244-8871bbf2d69a","service.name":"motel-index-collector","service.version":"git"},"otelcol.component.id":"ecproutingmiddleware","otelcol.component.kind":"extension","final_url_host":"local.es.svc.cluster.local:9200","final_url_scheme":"http","final_url":"[http://local.es.svc.cluster.local:9200/_otlp/v1/metrics","final_host":"","ProjectID":"local","header_count":6,"ecs.version":"1.6.0"}](http://local.es.svc.cluster.local:9200/_otlp/v1/metrics%22,%22final_host%22:%22%22,%22ProjectID%22:%22local%22,%22header_count%22:6,%22ecs.version%22:%221.6.0%22%7D)

This PR is a feature for exporterhelper, so is not just otlphttp-specific.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.17%. Comparing base (b5aaf8e) to head (2fd73f9).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #14139      +/-   ##
==========================================
- Coverage   92.17%   92.17%   -0.01%     
==========================================
  Files         668      669       +1     
  Lines       41463    41515      +52     
==========================================
+ Hits        38219    38265      +46     
- Misses       2212     2216       +4     
- Partials     1032     1034       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[gizas]

@bogdandrutu (sorry for direct ping) did you by any chance have time to review above pr? Do I have to add someone else specific?

Any hint will be really appreciated

[codspeed-hq]

CodSpeed Performance Report

Merging #14139 will improve performances by 25.7%

_{Comparing gizas:otlphttp_mergge_metadata (2fd73f9) with main (b5aaf8e)}

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

⚡ 1 improvement
✅ 72 untouched

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	BenchmarkBatchMetricProcessor2k	2.1 µs	1.7 µs	+25.7%

[gizas]

On puprose kept the comment for parity with base_exporter.go function

[axw]

Thanks for the updates, just a couple of minor things remaining from me.

[axw]

FWIW, I think we should probably compose them. But we can always do that in a followup - it would be an enhancement over what you have implemented.

[gizas]

Yes it was quicker just to error for now.

Do you want me to create a follow up once this merged?

Some details that we consider are things like:
Which logic to merge first, probably the custom logic?
How to check metadata within each custom partition with overall metadata_keys if those exist?

[dmitryax]

We need to provide an option to split batches using keys from other sources, not only request-context metadata, but also resource attributes, instrumentation name etc. #12795 has more details. This interface would conflict with that functionality. I’m not sure it’s worth having two configuration options that overlap or conflict with each other. I’d prefer to make progress toward #12795 instead. @axw WDYT?

[axw]

Thanks for the reminder @dmitryax, good points. I'd also love to make some progress on that - I'll have try to have a play with some ideas today.

This interface would conflict with that functionality.

Why would it conflict? Could we not have both? i.e. config for partitioning by metadata, as well as say support for partitioner extensions? It should also be possible to compose them, as I suggested in another comment thread, e.g. partition first by metadata, and then partition those batches via an extension.

(Playing devil's advocate a little bit, not necessarily saying that's what we should do.)

[axw]

I spent a while looking into this today, and I'm struggling to see a good path for adding resource/scope/record-level partitioning to exporterhelper.

I previously had a bit of a warped understanding of the Partitioner interface. I had in mind that it was operating on plog.Logs and friends, but it's actually operating on request.Request. That interface totally hides the data type, which makes splitting/merging batches complicated except for in generic ways, i.e. sizer-based splitting/merging.

I started looking into adding partitioning before we even get to batching, but stopped pretty quickly because that's effectively equivalent to processor. So... I'm back to thinking we should:

• Introduce a dedicated processor for splitting batches apart: New component: OTTL-based request partitioning processor opentelemetry-collector-contrib#39199
• Add support for partitioning generic requests by metadata keys (i.e. this PR)

@dmitryax if you have ideas about how we can achieve it all in exporterhelper I'd be keen to hear them.