Skip to content

[Bug]: runtime error: invalid memory address or nil pointer dereference #8002

New issue
New issue
Open
Open
[Bug]: runtime error: invalid memory address or nil pointer dereference#8002
Labels
area/storagebugstorage/elasticsearch
@Echo-ng

Description

@Echo-ng
Echo-ng
opened on Feb 9, 2026

What happened?

This error will occur after running for a period of time。

log

2026-02-09T10:15:11.993Z error recoveryhandler/zap.go:22 runtime error: invalid memory address or nil pointer dereference {"resource": {"service.instance.id": "cd72dc0c-7115-4f72-940c-d08b18251f2c", "service.name": "jaeger", "service.version": "v2.15.1"}, "otelcol.component.id": "jaeger_query", "otelcol.component.kind": "extension"}
github.com/jaegertracing/jaeger/internal/recoveryhandler.zapRecoveryWrapper.Println
github.com/jaegertracing/jaeger/internal/recoveryhandler/zap.go:22
github.com/gorilla/handlers.recoveryHandler.log
github.com/gorilla/handlers@v1.5.2/recovery.go:85
github.com/gorilla/handlers.recoveryHandler.ServeHTTP.func1
github.com/gorilla/handlers@v1.5.2/recovery.go:76
runtime.gopanic
runtime/panic.go:783
runtime.panicmem
runtime/panic.go:262
runtime.sigpanic
runtime/signal_unix.go:925
go.opentelemetry.io/collector/pdata/internal.(*State).AssertMutable
go.opentelemetry.io/collector/pdata@v1.51.0/internal/state.go:58
go.opentelemetry.io/collector/pdata/pcommon.Slice.AppendEmpty
go.opentelemetry.io/collector/pdata@v1.51.0/pcommon/generated_slice.go:96
github.com/jaegertracing/jaeger/internal/jptrace.AddWarnings
github.com/jaegertracing/jaeger/internal/jptrace/warning.go:19
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustTimestamps
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:188
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:148
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:150
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:150
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.NewQueryService.StandardAdjusters.CorrectClockSkew.func2
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:49
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.Func.Adjust
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/adjuster.go:23
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.sequence.Adjust
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/adjuster.go:40
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.func1
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:223
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.func2
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:238
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.QueryService.receiveTraces.AggregateTraces.func3.func4
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:37
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics.(*ReadMetricsDecorator).FindTraces.func1-range1
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics/reader_metrics.go:80
github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore.(*TraceReader).FindTraces.func1
github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore/reader.go:88
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics.(*ReadMetricsDecorator).FindTraces.func1
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics/reader_metrics.go:77
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.AggregateTraces.func3
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:22
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:237
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).search.QueryService.FindTraces.func1
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:153
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2.AggregateTraces.func2
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:22
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:44
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).search
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/http_handler.go:260
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.WithRouteTag.func1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:237
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).handleFunc.spanNameHandler.func1
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/http_handler.go:603
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
github.com/gorilla/mux.(*Router).ServeHTTP
github.com/gorilla/mux@v1.8.1/mux.go:212
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.initRouter.traceResponseHandler.func6
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/trace_response_handler.go:23
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
github.com/gorilla/handlers.recoveryHandler.ServeHTTP
github.com/gorilla/handlers@v1.5.2/recovery.go:80
go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP
go.opentelemetry.io/collector/config/confighttp@v0.145.0/compression.go:277
go.opentelemetry.io/collector/config/confighttp.(*ServerConfig).ToServer.maxRequestBodySizeInterceptor.func2
go.opentelemetry.io/collector/config/confighttp@v0.145.0/server.go:378
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:178
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:66
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
go.opentelemetry.io/collector/config/confighttp@v0.145.0/clientinfohandler.go:26
net/http.serverHandler.ServeHTTP
net/http/server.go:3340
net/http.(*conn).serve
net/http/server.go:2109
2026-02-09T10:15:11.993Z error recoveryhandler/zap.go:22 goroutine 580 [running]:
runtime/debug.Stack()
runtime/debug/stack.go:26 +0x5e
github.com/gorilla/handlers.recoveryHandler.log({{0x39d9fa0, 0xc0000107b0}, {0x39df420, 0xc0008c6780}, 0x1}, {0xc001592e60?, 0xc0006942b7?, 0xc000694308?})
github.com/gorilla/handlers@v1.5.2/recovery.go:91 +0xb2
github.com/gorilla/handlers.recoveryHandler.ServeHTTP.func1()
github.com/gorilla/handlers@v1.5.2/recovery.go:76 +0xbe
panic({0x2c960e0?, 0x56b7690?})
runtime/panic.go:783 +0x132
go.opentelemetry.io/collector/pdata/internal.(*State).AssertMutable(...)
go.opentelemetry.io/collector/pdata@v1.51.0/internal/state.go:58
go.opentelemetry.io/collector/pdata/pcommon.Slice.AppendEmpty({0x0?, 0x0?})
go.opentelemetry.io/collector/pdata@v1.51.0/pcommon/generated_slice.go:96 +0x1c
github.com/jaegertracing/jaeger/internal/jptrace.AddWarnings({0xc001247a00?, 0xc0003ffdc8?}, {0xc000694558, 0x1, 0x1?})
github.com/jaegertracing/jaeger/internal/jptrace/warning.go:19 +0xdf
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustTimestamps(0xee11ba1e2?, 0xc00128d0c0, {0xc00128d000?, {0x0?, 0xee11ba1d5?}})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:188 +0x1bf
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode(0xc000694710, 0xc00128d0c0, 0xc00128d000?, {0x0?, {0x0?, 0xc4c70ea1b7d02a13?}})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:148 +0xc7
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode(0xc000694710, 0xc00128d000, 0xc00128ccc0?, {0x0?, {0x0?, 0xffffffffffffffff?}})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:150 +0x105
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode(0xc000694710, 0xc00128ccc0, 0x0?, {0xc001246f70?, {0x0?, 0xfb71b3873f687802?}})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:150 +0x105
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.NewQueryService.StandardAdjusters.CorrectClockSkew.func2({0xc0012886f0?, 0xc0003ffdc8?})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:49 +0xc5
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.Func.Adjust(0xc000b80160?, {0xc0012886f0?, 0xc0003ffdc8?})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/adjuster.go:23 +0x26
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.sequence.Adjust(...)
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/adjuster.go:40
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.func1({0xc001290a00, 0x1, 0x1}, {0x0?, 0x0?})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:223 +0xf0
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.func2({0xc0012886f0?, 0xc0003ffdc8?}, {0x0, 0x0})
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:238 +0x94
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.QueryService.receiveTraces.AggregateTraces.func3.func4({0xc0012909f0?, 0xc001288708?, 0xc0006949d8?}, {0x0?, 0x0?})
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:37 +0x1ae
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics.(*ReadMetricsDecorator).FindTraces.func1-range1({0xc0012909f0?, 0x3?, 0x3?}, {0x0?, 0x0?})
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics/reader_metrics.go:80 +0x83
github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore.(*TraceReader).FindTraces.func1(0xc0007b7350)
github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore/reader.go:88 +0x1cd
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics.(*ReadMetricsDecorator).FindTraces.func1(0xc0007b7320)
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics/reader_metrics.go:77 +0x254
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.AggregateTraces.func3(...)
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:22
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces({{0x3a1c640, 0xc0007ce480}, {0x39e2ea0, 0xc000b80120}, {0x39df380, 0xc000330c78}, {{0x0, 0x0}, {0x0, 0x0}, ...}}, ...)
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:237 +0x2c2
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).search.QueryService.FindTraces.func1(0xc0007b7230)
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:153 +0xc5
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2.AggregateTraces.func2(...)
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:22
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2(0xc0004dd0b0)
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:44 +0x19b
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).search(0xc000ba29c0, {0x3a034d0, 0xc0002b5140}, 0xc0007ae280)
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/http_handler.go:260 +0x2e5
net/http.HandlerFunc.ServeHTTP(0xc000988600?, {0x3a034d0?, 0xc0002b5140?}, 0x1141022?)
net/http/server.go:2322 +0x29
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.WithRouteTag.func1({0x3a034d0, 0xc0002b5140}, 0xc0007ae280)
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:237 +0x1d8
net/http.HandlerFunc.ServeHTTP(0x3a050a0?, {0x3a034d0?, 0xc0002b5140?}, 0x47f445?)
net/http/server.go:2322 +0x29
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).handleFunc.spanNameHandler.func1({0x3a034d0, 0xc0002b5140}, 0xc0007ae280)
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/http_handler.go:603 +0x8a
net/http.HandlerFunc.ServeHTTP(0x3a050a0?, {0x3a034d0?, 0xc0002b5140?}, 0x39b4528?)
net/http/server.go:2322 +0x29
net/http.HandlerFunc.ServeHTTP(0xc0007ae000?, {0x3a034d0?, 0xc0002b5140?}, 0xc00111c6e0?)
net/http/server.go:2322 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00039ac00, {0x3a034d0, 0xc0002b5140}, 0xc00002bcc0)
github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.initRouter.traceResponseHandler.func6({0x3a034d0, 0xc0002b5140}, 0xc00002bcc0)
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/trace_response_handler.go:23 +0xcc
net/http.HandlerFunc.ServeHTTP(0xc0004dd5c8?, {0x3a034d0?, 0xc0002b5140?}, 0x0?)
net/http/server.go:2322 +0x29
github.com/gorilla/handlers.recoveryHandler.ServeHTTP({{0x39d9fa0, 0xc0000107b0}, {0x39df420, 0xc0008c6780}, 0x1}, {0x3a034d0?, 0xc0002b5140?}, 0x0?)
github.com/gorilla/handlers@v1.5.2/recovery.go:80 +0xd9
go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP(0xc000cb92f0, {0x3a034d0, 0xc0002b5140}, 0xc00002bcc0)
go.opentelemetry.io/collector/config/confighttp@v0.145.0/compression.go:277 +0x204
go.opentelemetry.io/collector/config/confighttp.(*ServerConfig).ToServer.maxRequestBodySizeInterceptor.func2({0x3a034d0, 0xc0002b5140}, 0xc00002bcc0)
go.opentelemetry.io/collector/config/confighttp@v0.145.0/server.go:378 +0x8a
net/http.HandlerFunc.ServeHTTP(0x3a050a0?, {0x3a034d0?, 0xc0002b5140?}, 0x39c02a8?)
net/http/server.go:2322 +0x29
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP(0xc00039afc0, {0x39fc740, 0xc00002c3c0}, 0xc00002bb80, {0x39d9fa0, 0xc00080f120})
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:178 +0x12c3
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1({0x39fc740?, 0xc00002c3c0?}, 0x455401?)
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:66 +0x35
net/http.HandlerFunc.ServeHTTP(0xc00002b900?, {0x39fc740?, 0xc00002c3c0?}, 0xc000ca2b80?)
net/http/server.go:2322 +0x29
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP(0xc0000108b8, {0x39fc740, 0xc00002c3c0}, 0xc00002b900)
go.opentelemetry.io/collector/config/confighttp@v0.145.0/clientinfohandler.go:26 +0xee
net/http.serverHandler.ServeHTTP({0xc000800d40?}, {0x39fc740?, 0xc00002c3c0?}, 0x1?)
net/http/server.go:3340 +0x8e
net/http.(*conn).serve(0xc000d7ebd0, {0x3a050a0, 0xc000b85b60})
net/http/server.go:2109 +0x665
created by net/http.(*Server).Serve in goroutine 298
net/http/server.go:3493 +0x485
{"resource": {"service.instance.id": "cd72dc0c-7115-4f72-940c-d08b18251f2c", "service.name": "jaeger", "service.version": "v2.15.1"}, "otelcol.component.id": "jaeger_query", "otelcol.component.kind": "extension"}
github.com/jaegertracing/jaeger/internal/recoveryhandler.zapRecoveryWrapper.Println
github.com/jaegertracing/jaeger/internal/recoveryhandler/zap.go:22
github.com/gorilla/handlers.recoveryHandler.log
github.com/gorilla/handlers@v1.5.2/recovery.go:93
github.com/gorilla/handlers.recoveryHandler.ServeHTTP.func1
github.com/gorilla/handlers@v1.5.2/recovery.go:76
runtime.gopanic
runtime/panic.go:783
runtime.panicmem
runtime/panic.go:262
runtime.sigpanic
runtime/signal_unix.go:925
go.opentelemetry.io/collector/pdata/internal.(*State).AssertMutable
go.opentelemetry.io/collector/pdata@v1.51.0/internal/state.go:58
go.opentelemetry.io/collector/pdata/pcommon.Slice.AppendEmpty
go.opentelemetry.io/collector/pdata@v1.51.0/pcommon/generated_slice.go:96
github.com/jaegertracing/jaeger/internal/jptrace.AddWarnings
github.com/jaegertracing/jaeger/internal/jptrace/warning.go:19
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustTimestamps
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:188
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:148
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:150
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.(*clockSkewAdjuster).adjustNode
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:150
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.NewQueryService.StandardAdjusters.CorrectClockSkew.func2
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/clockskew.go:49
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.Func.Adjust
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/adjuster.go:23
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster.sequence.Adjust
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/adjuster/adjuster.go:40
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.func1
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:223
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.func2
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:238
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.QueryService.receiveTraces.AggregateTraces.func3.func4
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:37
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics.(*ReadMetricsDecorator).FindTraces.func1-range1
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics/reader_metrics.go:80
github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore.(*TraceReader).FindTraces.func1
github.com/jaegertracing/jaeger/internal/storage/v2/elasticsearch/tracestore/reader.go:88
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics.(*ReadMetricsDecorator).FindTraces.func1
github.com/jaegertracing/jaeger/internal/storage/v2/api/tracestore/tracestoremetrics/reader_metrics.go:77
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces.AggregateTraces.func3
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:22
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc.QueryService.receiveTraces
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:237
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).search.QueryService.FindTraces.func1
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go:153
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2.AggregateTraces.func2
github.com/jaegertracing/jaeger/internal/jptrace/aggregator.go:22
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter.V1TracesFromSeq2
github.com/jaegertracing/jaeger/internal/storage/v2/v1adapter/translator.go:44
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).search
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/http_handler.go:260
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.WithRouteTag.func1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:237
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.(*APIHandler).handleFunc.spanNameHandler.func1
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/http_handler.go:603
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
github.com/gorilla/mux.(*Router).ServeHTTP
github.com/gorilla/mux@v1.8.1/mux.go:212
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal.initRouter.traceResponseHandler.func6
github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerquery/internal/trace_response_handler.go:23
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
github.com/gorilla/handlers.recoveryHandler.ServeHTTP
github.com/gorilla/handlers@v1.5.2/recovery.go:80
go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP
go.opentelemetry.io/collector/config/confighttp@v0.145.0/compression.go:277
go.opentelemetry.io/collector/config/confighttp.(*ServerConfig).ToServer.maxRequestBodySizeInterceptor.func2
go.opentelemetry.io/collector/config/confighttp@v0.145.0/server.go:378
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:178
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.64.0/handler.go:66
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2322
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
go.opentelemetry.io/collector/config/confighttp@v0.145.0/clientinfohandler.go:26
net/http.serverHandler.ServeHTTP
net/http/server.go:3340
net/http.(*conn).serve
net/http/server.go:2109

Steps to reproduce

This error will occur after running for a period of time。

Expected behavior

long term stable operation

Relevant log output



Screenshot


No response


Additional context


No response


Jaeger backend version


v2.15.1


SDK


No response


Pipeline


No response


Stogage backend


elasticsearch v9.2.3


Operating system


linux


Deployment model


docker


Deployment configs


## docker-compse.yaml
version: '3.8'
services:
  # 分布式链路追踪
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.2.3@sha256:0768db3277f2e60f57fdefdb225ee0be786b5d682feeb1ff162478b95ae13d4c
    container_name: mingx-elasticsearch
    networks:
      - mingx
    environment:
      - discovery.type=single-node
      - http.host=0.0.0.0
      - transport.host=127.0.0.1
      - xpack.security.enabled=false  # Disable security features
      - xpack.security.http.ssl.enabled=false  # Disable HTTPS
      - action.destructive_requires_name=false
      - xpack.monitoring.collection.enabled=false  # Disable monitoring features
      - ES_LOG_LEVEL=trace  # Set log level to capture all queries
      - logger.org.elasticsearch.index.search.slowlog=TRACE  # Log all slow queries
    volumes:
      - ./trace/data:/usr/share/elasticsearch/data
      - ./trace/plugins:/usr/share/elasticsearch/plugins
    ports:
      - "9200:9200"
      - "9300:9300"
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:9200 || exit 1" ]
      interval: 10s
      timeout: 10s
      retries: 30

  jaeger:
    container_name: mingx-jaeger
    networks:
      - mingx
    image: jaegertracing/jaeger:${JAEGER_VERSION:-latest}
    volumes:
      - "./trace/jaeger-ui.json:/etc/jaeger/jaeger-ui.json" # Do we need this for v2 ? Seems to be running without this.
      - "./trace/config-spm-elasticsearch.yaml:/etc/jaeger/config.yml"
    command: ["--config", "/etc/jaeger/config.yml"]
    environment:
      - SPANMETRICS_FLUSH_INTERVAL=${SPANMETRICS_FLUSH_INTERVAL:-60s}
    ports:
      - "16686:16686" # Jaeger UI http://localhost:16686
      - "18888:8888"
      - "18889:8889"
      - "4317:4317"
      - "4318:4318"
    depends_on:
      elasticsearch:
        condition: service_healthy


## config.yml
service:
  extensions: [basicauth/server, jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]

extensions:
  jaeger_query:
    storage:
      traces: elasticsearch_trace_storage
      metrics: elasticsearch_trace_storage
  jaeger_storage:
    backends:
      elasticsearch_trace_storage: &elasticsearch_config
        elasticsearch:
          server_urls:
            - http://elasticsearch:9200
    metric_backends:
      elasticsearch_trace_storage: *elasticsearch_config
  basicauth/server:
    htpasswd:
      inline: |
        topway@example.com:dG9wd2F5QDEyMzQ=

receivers:
  otlp:
    protocols:
      grpc:
      http:
        endpoint: "0.0.0.0:4318"
        auth:
          authenticator: basicauth/server

processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: elasticsearch_trace_storage
Metadata
Metadata
Assignees
No one assigned
    Labels
    area/storagebugstorage/elasticsearch
    Type
    No type
    Projects
    No projects
    Milestone
    No milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions