Skip to content

Conversation

@Dylan-M
Copy link
Owner

@Dylan-M Dylan-M commented Jun 9, 2025

Description

Link to tracking issue

Fixes

Testing

Documentation

Dylan-M pushed a commit that referenced this pull request Jun 18, 2025
…elemetry#40559)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

Prometheus remote write exporter add extra logs for the RW2 code path.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes

Partially implements
open-telemetry#33661
(when merging PR please don't close the tracing issue)

<!--Describe what testing was performed and which tests were added.-->
#### Testing
* [x] e2e ran with prometheus locally


Sample of the logs running against both Prometheus `v3.4.1` and
`v2.53.1` (which doesn't support RW2).

![image](https://github.com/user-attachments/assets/ff27ed02-87fa-4e5e-b7cf-65dcaf3c1907)

```
2025-06-09T17:15:44.390+0200	info	[email protected]/server.go:886	[core] [Server #1 ListenSocket open-telemetry#2]ListenSocket created	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "grpc_log": true}
2025-06-09T17:15:52.173+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:15:53.968+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:15:57.164+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:15:58.966+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:16:02.164+0200	debug	prometheusremotewriteexporter/exporter.go:425	X-Prometheus-Remote-Write-Samples-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "samples_written": "22"}
2025-06-09T17:16:02.164+0200	debug	prometheusremotewriteexporter/exporter.go:428	X-Prometheus-Remote-Write-Histograms-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "histograms_written": "0"}
2025-06-09T17:16:02.164+0200	debug	prometheusremotewriteexporter/exporter.go:431	X-Prometheus-Remote-Write-Exemplars-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "exemplars_written": "0"}
2025-06-09T17:16:03.967+0200	debug	prometheusremotewriteexporter/exporter.go:425	X-Prometheus-Remote-Write-Samples-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "samples_written": "290"}
2025-06-09T17:16:03.967+0200	debug	prometheusremotewriteexporter/exporter.go:428	X-Prometheus-Remote-Write-Histograms-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "histograms_written": "0"}
```

cc @krajorama @ywwg
@Dylan-M Dylan-M self-assigned this Jun 18, 2025
@Dylan-M Dylan-M marked this pull request as ready for review August 5, 2025 12:44
Dylan-M pushed a commit that referenced this pull request Aug 5, 2025
…elemetry#40559)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

Prometheus remote write exporter add extra logs for the RW2 code path.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes

Partially implements
open-telemetry#33661
(when merging PR please don't close the tracing issue)

<!--Describe what testing was performed and which tests were added.-->
#### Testing
* [x] e2e ran with prometheus locally


Sample of the logs running against both Prometheus `v3.4.1` and
`v2.53.1` (which doesn't support RW2).

![image](https://github.com/user-attachments/assets/ff27ed02-87fa-4e5e-b7cf-65dcaf3c1907)

```
2025-06-09T17:15:44.390+0200	info	[email protected]/server.go:886	[core] [Server #1 ListenSocket open-telemetry#2]ListenSocket created	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "grpc_log": true}
2025-06-09T17:15:52.173+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:15:53.968+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:15:57.164+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:15:58.966+0200	warn	prometheusremotewriteexporter/exporter.go:420	X-Prometheus-Remote-Write-Samples-Written header is missing from the response, suggesting that the endpoint doesn't support RW2.	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "url": "http://localhost:9091/api/v1/write"}
2025-06-09T17:16:02.164+0200	debug	prometheusremotewriteexporter/exporter.go:425	X-Prometheus-Remote-Write-Samples-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "samples_written": "22"}
2025-06-09T17:16:02.164+0200	debug	prometheusremotewriteexporter/exporter.go:428	X-Prometheus-Remote-Write-Histograms-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "histograms_written": "0"}
2025-06-09T17:16:02.164+0200	debug	prometheusremotewriteexporter/exporter.go:431	X-Prometheus-Remote-Write-Exemplars-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "exemplars_written": "0"}
2025-06-09T17:16:03.967+0200	debug	prometheusremotewriteexporter/exporter.go:425	X-Prometheus-Remote-Write-Samples-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "samples_written": "290"}
2025-06-09T17:16:03.967+0200	debug	prometheusremotewriteexporter/exporter.go:428	X-Prometheus-Remote-Write-Histograms-Written	{"resource": {"service.instance.id": "383f7fdd-08e5-4e4c-85fe-c1fe3002d1ba", "service.name": "otelcontribcol", "service.version": "0.127.0-dev"}, "otelcol.component.id": "prometheusremotewrite/prometheus", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "histograms_written": "0"}
```

cc @krajorama @ywwg
Dylan-M pushed a commit that referenced this pull request Oct 3, 2025
…b.uid (open-telemetry#42641)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR adds support to expose `k8s.cronjob.uid` as resource metadata
when a `Job` is owned by a `CronJob`.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue

Fixes open-telemetry#42557

<!--Describe what testing was performed and which tests were added.-->
#### Testing

Local tested with `telemetrygen` and is working as expected.

```
[pod/k8sevents-receiver-opentelemetry-collector-6fd9966559-brlb6/opentelemetry-collector] {"level":"debug","ts":"2025-09-11T16:29:11.588Z","caller":"[email protected]/processor.go:159","msg":"getting the pod","resource":{"service.instance.id":"9631e38b-aec3-439f-8178-d96fc8368e1e","service.name":"otelcontribcol","service.version":"0.135.0-dev"},"otelcol.component.id":"k8sattributes","otelcol.component.kind":"processor","otelcol.pipeline.id":"traces","otelcol.signal":"traces","pod":{"Name":"otel-log-cronjob-29293469-lw97x","Address":"10.244.0.70","PodUID":"7960681c-5a24-4287-8bea-e2cf506500ee","Attributes":{"k8s.cronjob.name":"otel-log-cronjob","k8s.cronjob.uid":"082b1c42-e393-46bc-9d51-b20a3700d1ab","k8s.job.name":"otel-log-cronjob-29293469","k8s.job.uid":"fbd853b8-7f63-44d8-ace1-8b48c89e3041"},"StartTime":"2025-09-11T16:29:00Z","Ignore":false,"Namespace":"default","NodeName":"","DeploymentUID":"","StatefulSetUID":"","DaemonSetUID":"","JobUID":"fbd853b8-7f63-44d8-ace1-8b48c89e3041","HostNetwork":false,"Containers":{"ByID":null,"ByName":null},"DeletedAt":"0001-01-01T00:00:00Z"}}
[pod/k8sevents-receiver-opentelemetry-collector-6fd9966559-brlb6/opentelemetry-collector] {"level":"info","ts":"2025-09-11T16:29:11.588Z","msg":"Traces","resource":{"service.instance.id":"9631e38b-aec3-439f-8178-d96fc8368e1e","service.name":"otelcontribcol","service.version":"0.135.0-dev"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"traces","resource spans":1,"spans":2}
[pod/k8sevents-receiver-opentelemetry-collector-6fd9966559-brlb6/opentelemetry-collector] {"level":"info","ts":"2025-09-11T16:29:11.588Z","msg":"ResourceSpans #0\nResource SchemaURL: https://opentelemetry.io/schemas/1.4.0\nResource attributes:\n     -> k8s.container.name: Str(telemetrygen)\n     -> service.name: Str(telemetrygen)\n     -> k8s.pod.ip: Str(10.244.0.70)\n     -> k8s.cronjob.name: Str(otel-log-cronjob)\n     -> k8s.cronjob.uid: Str(082b1c42-e393-46bc-9d51-b20a3700d1ab)\n     -> k8s.job.uid: Str(fbd853b8-7f63-44d8-ace1-8b48c89e3041)\n     -> k8s.job.name: Str(otel-log-cronjob-29293469)\nScopeSpans #0\nScopeSpans SchemaURL: \nInstrumentationScope telemetrygen \nSpan #0\n    Trace ID       : 3c7381c14a37814676b00a7d961cb219\n    Parent ID      : 4f8780d5148a9c1c\n    ID             : 17e9da9533dc93ca\n    Name           : okey-dokey-0\n    Kind           : Server\n    Start time     : 2025-09-11 16:29:09.583785469 +0000 UTC\n    End time       : 2025-09-11 16:29:09.583908469 +0000 UTC\n    Status code    : Unset\n    Status message : \nAttributes:\n     -> net.peer.ip: Str(1.2.3.4)\n     -> peer.service: Str(telemetrygen-client)\nSpan #1\n    Trace ID       : 3c7381c14a37814676b00a7d961cb219\n    Parent ID      : \n    ID             : 4f8780d5148a9c1c\n    Name           : lets-go\n    Kind           : Client\n    Start time     : 2025-09-11 16:29:09.583785469 +0000 UTC\n    End time       : 2025-09-11 16:29:09.583908469 +0000 UTC\n    Status code    : Unset\n    Status message : \nAttributes:\n     -> net.peer.ip: Str(1.2.3.4)\n     -> peer.service: Str(telemetrygen-server)\n","resource":{"service.instance.id":"9631e38b-aec3-439f-8178-d96fc8368e1e","service.name":"otelcontribcol","service.version":"0.135.0-dev"},"otelcol.component.id":"debug","otelcol.component.kind":"exporter","otelcol.signal":"traces"}
```

Added also the tests to guarantee the proper functionality.

---------

Signed-off-by: Paulo Dias <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants