[exporter/elasticsearchexporter/integrationtest]: Report for failed tests on main

[github-actions]

Auto-generated report for __run_3 job build.

Link to failed build: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/17469693955

Test Failures

• TestExporter/logs/batcher_enabled_es_intermittent_doc_error

=== RUN   TestExporter/logs/batcher_enabled_es_intermittent_doc_error
    exporter_test.go:57: test otel collector configuration: 
        receivers:
          otlp:
            protocols:
              grpc:
                endpoint: "127.0.0.1:33477"
        exporters:
          elasticsearch:
            endpoints: [http://127.0.0.1:33477]
            logs_index: logs-test-idx
            metrics_index: metrics-test-idx
            traces_index: traces-test-idx
            sending_queue:
              enabled: true
              block_on_overflow: true
            mapping:
              mode: otel
            retry:
              enabled: true
              initial_interval: 100ms
              max_interval: 500ms
              max_retries: 10000
              retry_on_status: [429, 503]
            timeout: 10m
        
            batcher:
              flush_timeout: 1s
              enabled: true
        
          debug:
            verbosity: basic
        
        processors:
          
        
        extensions:
          
        
        service:
          telemetry:
            metrics:
              level: none
            logs:
              level: INFO
              sampling:
                enabled: false
          extensions: []
          pipelines:
            logs:
              receivers: [otlp]
              processors: []
              exporters: [elasticsearch]
        
2025/09/04 16:07:21 Starting mock backend...
2025-09-04T16:07:21.570Z	info	otelconftelemetry/telemetry.go:59	Internal metrics telemetry disabled	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}}
2025-09-04T16:07:21.570Z	warn	elasticsearchexporter/config.go:509	batcher has been deprecated, and will be removed in a future version. Use sending_queue instead.	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}, "otelcol.component.id": "elasticsearch", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-09-04T16:07:21.570Z	info	service@v0.134.0/service.go:211	Starting otelcol...	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}, "Version": "latest", "NumCPU": 4}
2025-09-04T16:07:21.570Z	info	extensions/extensions.go:41	Starting extensions...	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}}
2025-09-04T16:07:21.571Z	error	graph/graph.go:439	Failed to start component	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}, "error": "listen tcp 127.0.0.1:33477: bind: address already in use", "type": "Receiver", "id": "otlp"}
2025-09-04T16:07:21.571Z	info	service@v0.134.0/service.go:248	Starting shutdown...	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}}
2025-09-04T16:07:21.571Z	info	extensions/extensions.go:69	Stopping extensions...	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}}
2025-09-04T16:07:21.571Z	info	service@v0.134.0/service.go:262	Shutdown complete.	{"resource": {"service.instance.id": "b7162469-8452-4f22-a2c7-19441d4bc0dc", "service.name": "otelcol", "service.version": "latest"}}
2025/09/04 16:07:22 unable to start, otelcol state is 3
    test_case.go:291: unable to start, otelcol state is 3
2025/09/04 16:07:22 Starting load generator at 1000 items/sec.
2025/09/04 16:07:22 Stopped generator. Sent:         0 logs (0/sec)
2025/09/04 16:07:22 cannot start pipelines: failed to start "otlp" receiver: listen tcp 127.0.0.1:33477: bind: address already in use
2025/09/04 16:07:22 Stopping mock backend...
2025/09/04 16:07:22 Stopped backend. Received:         0 items (0/sec)
--- FAIL: TestExporter/logs/batcher_enabled_es_intermittent_doc_error (1.00s)

• TestExporter

=== RUN   TestExporter
--- FAIL: TestExporter (159.20s)

Note: Information about any subsequent build failures that happen while
this issue is open, will be added as comments with more information to this issue.

[github-actions]

Pinging code owners for exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[github-actions]

Link to latest failed build: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/17490737765

Test Failures

• TestExporter/traces/es_intermittent_doc_error

=== RUN   TestExporter/traces/es_intermittent_doc_error
    exporter_test.go:57: test otel collector configuration: 
        receivers:
          otlp:
            protocols:
              grpc:
                endpoint: "127.0.0.1:49817"
        exporters:
          elasticsearch:
            endpoints: [http://127.0.0.1:49818]
            logs_index: logs-test-idx
            metrics_index: metrics-test-idx
            traces_index: traces-test-idx
            sending_queue:
              enabled: true
              block_on_overflow: true
            mapping:
              mode: otel
            retry:
              enabled: true
              initial_interval: 100ms
              max_interval: 500ms
              max_retries: 10000
              retry_on_status: [429, 503]
            timeout: 10m
        
            flush:
              interval: 1s
        
          debug:
            verbosity: basic
        
        processors:
          
        
        extensions:
          
        
        service:
          telemetry:
            metrics:
              level: none
            logs:
              level: INFO
              sampling:
                enabled: false
          extensions: []
          pipelines:
            traces:
              receivers: [otlp]
              processors: []
              exporters: [elasticsearch]
        
2025/09/05 11:07:11 Starting mock backend...
2025-09-05T11:07:11.375Z	info	otelconftelemetry/telemetry.go:59	Internal metrics telemetry disabled	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}}
2025-09-05T11:07:11.375Z	info	service@v0.134.0/service.go:211	Starting otelcol...	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}, "Version": "latest", "NumCPU": 4}
2025-09-05T11:07:11.375Z	info	extensions/extensions.go:41	Starting extensions...	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}}
2025-09-05T11:07:11.376Z	info	otlpreceiver@v0.134.0/otlp.go:121	Starting GRPC server	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "127.0.0.1:49817"}
2025-09-05T11:07:11.376Z	info	service@v0.134.0/service.go:234	Everything is ready. Begin running and processing data.	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}}
2025/09/05 11:07:12 Starting load generator at 1000 items/sec.
2025/09/05 11:07:14  | Sent:     1,970 traces (986/sec) | Received:     1,000 items (333/sec)
2025-09-05T11:07:21.765Z	error	internal/queue_sender.go:52	Exporting failed. Dropping data.	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}, "otelcol.component.id": "elasticsearch", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "context deadline exceeded", "dropped_items": 10}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue_sender.go:52
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queuebatch/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue.(*asyncQueue[...]).Start.func1
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue/async_queue.go:49
2025/09/05 11:07:21  | Sent:     2,140 traces (227/sec) | Received:     1,000 items (96/sec)
2025-09-05T11:07:22.043Z	error	internal/queue_sender.go:52	Exporting failed. Dropping data.	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}, "otelcol.component.id": "elasticsearch", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "context deadline exceeded", "dropped_items": 10}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue_sender.go:52
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queuebatch/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue.(*asyncQueue[...]).Start.func1
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue/async_queue.go:49
2025-09-05T11:07:22.096Z	error	internal/queue_sender.go:52	Exporting failed. Dropping data.	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}, "otelcol.component.id": "elasticsearch", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "context deadline exceeded", "dropped_items": 10}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue_sender.go:52
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queuebatch/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue.(*asyncQueue[...]).Start.func1
	C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue/async_queue.go:49
2025/09/05 11:07:24  | Sent:     2,150 traces (177/sec) | Received:     2,101 items (160/sec)
2025/09/05 11:07:24 Stopped generator. Sent:     2,170 traces (178/sec)
2025/09/05 11:07:26  | Sent:     2,170 traces (155/sec) | Received:     2,151 items (143/sec)
2025/09/05 11:07:29  | Sent:     2,170 traces (127/sec) | Received:     2,151 items (119/sec)
2025/09/05 11:07:32  | Sent:     2,170 traces (108/sec) | Received:     2,151 items (102/sec)
2025/09/05 11:07:34 Time out waiting for backend should receive all sent items
    test_case.go:291: Time out waiting for backend should receive all sent items
2025-09-05T11:07:34.861Z	info	otelcol@v0.134.0/collector.go:366	Received shutdown request	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}}
2025-09-05T11:07:34.861Z	info	service@v0.134.0/service.go:248	Starting shutdown...	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}}
2025-09-05T11:07:34.863Z	info	extensions/extensions.go:69	Stopping extensions...	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}}
2025-09-05T11:07:34.863Z	info	service@v0.134.0/service.go:262	Shutdown complete.	{"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}}
2025/09/05 11:07:34 Stopping mock backend...
2025/09/05 11:07:34 Stopped backend. Received:     2,151 items (91/sec)
--- FAIL: TestExporter/traces/es_intermittent_doc_error (23.74s)

• TestExporter

=== RUN   TestExporter
--- FAIL: TestExporter (187.23s)

[lahsivjar]

This test seems to capture 2 different issues:

    exporter_test.go:57: test otel collector configuration: 
        receivers:
          otlp:
            protocols:
              grpc:
                endpoint: "127.0.0.1:33477"
        exporters:
          elasticsearch:
            endpoints: [http://127.0.0.1:33477]

The above, from the original issue description, is a conflict due to testutil::GetAvailablePort returning the same port for the OTLP receiver and mock Elasticsearch (33477). This should be fixed by #43121.

OTOH, the issue in this comment is different as it seems to timeout while waiting for the expected docs (the port conflict issue doesn't happen for this run as can be seen that receiver and mock ES have different ports). This might point to a bug in the Elasticsearch exporter code - more investigation is needed here.

[lahsivjar]

The answer for why for #42499 (comment) are actually in the logs:

2025-09-05T11:07:21.765Z error internal/queue_sender.go:52 Exporting failed. Dropping data. {"resource": {"service.instance.id": "2e988e37-c1d6-4451-a319-72ce0bdebdad", "service.name": "otelcol", "service.version": "latest"}, "otelcol.component.id": "elasticsearch", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "context deadline exceeded", "dropped_items": 10}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue_sender.go:52
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume
C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queuebatch/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue.(*asyncQueue[...]).Start.func1
C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/exporter/exporterhelper@v0.134.0/internal/queue/async_queue.go:49

The exporter simply timed out while trying to send data to the mock Elasticsearch - due to this the data got dropped and never went to the mock ES where assertions are made. The issue is happening with the current default for Elasticsearch exporter which will use the inbuilt async indexer. #42767 PR is changing the default to sending_queue, so the issue might get resolved once it is merged. I think we can close the issue when the PR #42767 is merged and investigate future errors if/when they occur.

[mx-psi]

Closing since #42767 was merged