Flaky OpenAI Tests

[jaydeluca]

Starting on 2025-07-28, the OpenAI ChatTests have become very flaky. Sometimes the flakyness occurs on both the library and javaagent tests. This seems to have started after this PR was merged.

Example build scan

The stack traces seem to indicate that an okhttp executor is shutting down

[otel.javaagent 2025-08-01 04:13:32:998 +0000] [openai-stream-handler-thread-0] INFO io.opentelemetry.javaagent.testing.exporter.OtlpInMemorySpanExporter - Exporting span SpanData{spanContext=ImmutableSpanContext{traceId=18bfbdfc826388284db75032ab1f38d6, spanId=b4ef8454eb09d12f, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=null, attributes={service.instance.id="9ba1a90d-19ac-4b9d-bb95-177f0258e18b", service.name="unknown_service:java", telemetry.distro.name="opentelemetry-java-instrumentation", telemetry.distro.version="2.19.0-SNAPSHOT", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.52.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.openai-java-1.1, version=2.19.0-alpha-SNAPSHOT, schemaUrl=null, attributes={}}, name=chat gpt-4o-mini, kind=INTERNAL, startEpochNanos=1754021611435470082, endEpochNanos=1754021612994200865, attributes=AttributesMap{data={gen_ai.system=openai, gen_ai.request.model=gpt-4o-mini, gen_ai.operation.name=chat}, capacity=128, totalAddedValues=3}, totalAttributeCount=3, events=[ImmutableExceptionEventData{attributes=AttributesMap{data={exception.stacktrace=com.openai.errors.OpenAIIoException: Request failed |  
-- | --
at com.openai.client.okhttp.OkHttpClient$executeAsync$2.onFailure(OkHttpClient.kt:63) |  
at okhttp3.internal.connection.RealCall$AsyncCall.executeOn(RealCall.kt:504) |  
at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.kt:184) |  
at okhttp3.Dispatcher.enqueue$okhttp(Dispatcher.kt:125) |  
at okhttp3.internal.connection.RealCall.enqueue(RealCall.kt:164) |  
at com.openai.client.okhttp.OkHttpClient.executeAsync(OkHttpClient.kt:56) |  
at com.openai.core.http.RetryingHttpClient.executeAsync$executeWithRetries(RetryingHttpClient.kt:96) |  
at com.openai.core.http.RetryingHttpClient.access$executeAsync$executeWithRetries(RetryingHttpClient.kt:23) |  
at com.openai.core.http.RetryingHttpClient$executeAsync$executeWithRetries$1$1.invoke(RetryingHttpClient.kt:118) |  
at com.openai.core.http.RetryingHttpClient$executeAsync$executeWithRetries$1$1.invoke(RetryingHttpClient.kt:117) |  
at com.openai.core.http.RetryingHttpClient$executeAsync$executeWithRetries$1.invoke$lambda$0(RetryingHttpClient.kt:117) |  
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1194) |  
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554) |  
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223) |  
at com.openai.core.http.RetryingHttpClient$Companion$sleepAsync$1.run(RetryingHttpClient.kt:236) |  
at java.base/java.util.TimerThread.mainLoop(Timer.java:611) |  
at java.base/java.util.TimerThread.run(Timer.java:549) |  
Caused by: java.io.InterruptedIOException: executor rejected |  
at okhttp3.internal.connection.RealCall$AsyncCall.executeOn(RealCall.kt:501) |  
... 15 more |  
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.internal.connection.RealCall$AsyncCall@8bb88c70 rejected from java.util.concurrent.ThreadPoolExecutor@ad046f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] |  
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2082) |  
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841) |  
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1377) |  
at okhttp3.internal.connection.RealCall$AsyncCall.executeOn(RealCall.kt:498) |  
... 15 more |  
, exception.message=Request failed, exception.type=com.openai.errors.OpenAIIoException}, capacity=128, totalAddedValues=3}, epochNanos=1754021612994063149, totalAttributeCount=3, exception=com.openai.errors.OpenAIIoException: Request failed}], totalRecordedEvents=1, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=ERROR, description=}, hasEnded=true} |  

[jaydeluca]

cc @anuraaga

[anuraaga]

Thanks for the notice @jaydeluca. I added memoization of the raw clients in #14353

https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/14353/files#diff-5e7745b15dc7d3e32eef4e28652928f0436259fb2352333448c72b7f5d4244bcR44

Hoping it helps with this

[jaydeluca]

@anuraaga it looks like there are some testLatestDeps and muzzle failures with the 3.0.0 release of the openai-java sdk.

ChatCompletionMessageToolCall became ChatCompletionMessageFunctionToolCall and maybe some other changes

:instrumentation:openai:openai-java-1.1:library:compileTestJava FAILED |  
-- | --
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:321: error: cannot find symbol |  
.filter(call -> call.function().arguments().contains("New York")) |  
^ |  
symbol:   method arguments() |  
location: class Optional<ChatCompletionMessageFunctionToolCall> |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:322: error: invalid method reference |  
.map(ChatCompletionMessageToolCall::id) |  
^ |  
cannot find symbol |  
symbol:   method id() |  
location: class ChatCompletionMessageToolCall |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:327: error: cannot find symbol |  
.filter(call -> call.function().arguments().contains("London")) |  
^ |  
symbol:   method arguments() |  
location: class Optional<ChatCompletionMessageFunctionToolCall> |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:328: error: invalid method reference |  
.map(ChatCompletionMessageToolCall::id) |  
^ |  
cannot find symbol |  
symbol:   method id() |  
location: class ChatCompletionMessageToolCall |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:779: error: cannot find symbol |  
ChatCompletionMessageToolCall.Builder currentToolCall = null; |  
^ |  
symbol:   class Builder |  
location: class ChatCompletionMessageToolCall |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:780: error: cannot find symbol |  
ChatCompletionMessageToolCall.Function.Builder currentFunction = null; |  
^ |  
symbol:   class Function |  
location: class ChatCompletionMessageToolCall |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:798: error: cannot find symbol |  
currentToolCall = ChatCompletionMessageToolCall.builder().id(call.id().get()); |  
^ |  
symbol:   method builder() |  
location: class ChatCompletionMessageToolCall |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:799: error: cannot find symbol |  
currentFunction = ChatCompletionMessageToolCall.Function.builder(); |  
^ |  
symbol:   variable Function |  
location: class ChatCompletionMessageToolCall |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:826: error: cannot find symbol |  
.filter(call -> call.function().arguments().contains("New York")) |  
^ |  
symbol:   method arguments() |  
location: class Optional<ChatCompletionMessageFunctionToolCall> |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:827: error: invalid method reference |  
.map(ChatCompletionMessageToolCall::id) |  
^ |  
cannot find symbol |  
symbol:   method id() |  
location: class ChatCompletionMessageToolCall |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:832: error: cannot find symbol |  
.filter(call -> call.function().arguments().contains("London")) |  
^ |  
symbol:   method arguments() |  
location: class Optional<ChatCompletionMessageFunctionToolCall> |  
/home/runner/work/opentelemetry-java-instrumentation/opentelemetry-java-instrumentation/instrumentation/openai/openai-java-1.1/library/src/test/java/io/opentelemetry/instrumentation/openai/v1_1/ChatTest.java:833: error: invalid method reference |  
.map(ChatCompletionMessageToolCall::id) |  
^ |  
cannot find symbol |  
symbol:   method id() |  
location: class ChatCompletionMessageToolCall |  
12 errors

[anuraaga]

Thanks @laurit for fixing it

[laurit]

@anuraaga I didn't fix the flaky tests I only restricted the latest dep test version, which is a separate issue. Instrumentation needs some work for it to work with 3.0.0.

[jaydeluca]

test flakyness looks better after #14353 was merged, and support for 3.0 is fixed in #14423

Thanks all!