How to get proper logs in OTEL Collector to debug ourselves

[seilerre]

Hey all,

I keep running into this, so maybe I just missed a paramount information somewhere. My problem many times has now been, that I can't properly debug OTEL. I would expect to be able to switch on DEBUG logs (which I did) and for example see, what is going wrong in OTEL. Right now I got the following case:

opentelemetry-collector:
  image:
    repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s
    tag: 0.123.1
  mode: "deployment"
  config:
    receivers:
      otlp:
        protocols:
          http:
            endpoint: ${env:MY_POD_IP}:4318
            include_metadata: true
            cors:
              allowed_origins:
                - "*"
              max_age: 7200
    processors:
      memory_limiter:
        check_interval: 5s
        limit_mib: 500
        spike_limit_mib: 100
      batch:
        timeout: 5s
        send_batch_size: 1
    exporters:
      debug:
        verbosity: detailed
    service:
      telemetry:
        logs:
          development: true
          level: DEBUG
      pipelines:
        logs:
          receivers: [otlp]
          processors: []
          exporters: [debug]
        traces:
          receivers: [otlp]
          processors: []
          exporters: [debug]
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [debug]

Requests to all endpoints deliver PartialSuccess

 ./test.sh
HTTP/2 200
date: Wed, 23 Apr 2025 11:09:26 GMT
content-type: application/json
content-length: 21
vary: Origin
content-security-policy: frame-ancestors 'none'; object-src 'none'
cross-origin-resource-policy: same-site
referrer-policy: same-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-frame-options: DENY

{"partialSuccess":{}}
HTTP/2 200
date: Wed, 23 Apr 2025 11:09:26 GMT
content-type: application/json
content-length: 21
vary: Origin
content-security-policy: frame-ancestors 'none'; object-src 'none'
cross-origin-resource-policy: same-site
referrer-policy: same-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-frame-options: DENY

{"partialSuccess":{}}
HTTP/2 200
date: Wed, 23 Apr 2025 11:09:26 GMT
content-type: application/json
content-length: 21
vary: Origin
content-security-policy: frame-ancestors 'none'; object-src 'none'
cross-origin-resource-policy: same-site
referrer-policy: same-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-frame-options: DENY

{"partialSuccess":{}}%

But my logs are completely empty:

 ❯ k logs otel-frontend-opentelemetry-collector-5c9984769-wzsr8 -f
2025-04-23T11:09:08.400Z        info    [email protected]/service.go:197 Setting up own telemetry...
2025-04-23T11:09:08.400Z        info    builders/builders.go:26 Development component. May change in the future.
2025-04-23T11:09:08.400Z        debug   builders/builders.go:24 Stable component.
2025-04-23T11:09:08.400Z        debug   [email protected]/otlp.go:58        created signal-agnostic logger
2025-04-23T11:09:08.400Z        info    builders/builders.go:26 Development component. May change in the future.
2025-04-23T11:09:08.400Z        debug   builders/builders.go:24 Stable component.
2025-04-23T11:09:08.400Z        info    builders/builders.go:26 Development component. May change in the future.
2025-04-23T11:09:08.400Z        debug   builders/builders.go:24 Stable component.
2025-04-23T11:09:08.400Z        debug   builders/extension.go:48        Beta component. May change in the future.
2025-04-23T11:09:08.401Z        info    [email protected]/service.go:264 Starting otelcol-k8s... {"Version": "0.123.1", "NumCPU": 8}
2025-04-23T11:09:08.401Z        info    extensions/extensions.go:41     Starting extensions...
2025-04-23T11:09:08.401Z        debug   Logger core does not support injecting component attributes
2025-04-23T11:09:08.401Z        info    extensions/extensions.go:45     Extension is starting...
2025-04-23T11:09:08.401Z        info    [email protected]/healthcheckextension.go:32        Starting health_check extension {"config": {"Endpoint":"100.110.52.238:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2025-04-23T11:09:08.401Z        info    extensions/extensions.go:62     Extension started.
2025-04-23T11:09:08.401Z        info    [email protected]/server.go:690      [core] [Server #1]Server created        {"grpc_log": true}
2025-04-23T11:09:08.401Z        info    [email protected]/otlp.go:116       Starting GRPC server    {"endpoint": "100.110.52.238:4317"}
2025-04-23T11:09:08.401Z        info    [email protected]/otlp.go:173       Starting HTTP server    {"endpoint": "100.110.52.238:4318"}
2025-04-23T11:09:08.401Z        info    healthcheck/handler.go:132      Health Check state change       {"status": "ready"}
2025-04-23T11:09:08.401Z        info    [email protected]/service.go:287 Everything is ready. Begin running and processing data.
2025-04-23T11:09:08.402Z        info    [email protected]/server.go:886      [core] [Server #1 ListenSocket #2]ListenSocket created  {"grpc_log": true}

Why does OTEL log PartialSuccess, telling me that everything should be fine, but not any debug log? Why is my debug exporter not doing anything?