-
Notifications
You must be signed in to change notification settings - Fork 71
Description
Describe the bug
My app is crashing after opentelemetrty server is unavailable for a while. It seems to be very close to #open-telemetry/opentelemetry-java#6946 with same behavior but some different message
Steps to reproduce
I'm using io.opentelemetry.android:android-agent:0.13.0-alpha based on io.opentelemetry:opentelemetry-exporter-otlp:1.52.0 and it is working great until I shutdown the server and wait for about a couple of minutes. In the meanwhile the application log says:
Failed to export logs. The request could not be executed. Error message: timeout
java.io.InterruptedIOException: timeout
at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:430)
at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:392)
at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:351)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:216)
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:530)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:919)
Caused by: java.io.IOException: Canceled
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:70)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:96)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:208)
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:530)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:919)
periodically, which is expected. But after some time the app just crashing with following messages:
FromDiskExporterImpl Attempting to export logs batch from disk. FromDiskExporterImpl *** I Attempting to read data from ReadableFile{file=/data/user/0/*.*.*/cache/opentelemetry/signals/logs/1754561713102} FromDiskExporterImpl *** I Read 10809 logs bytes from storage. FromDiskExporterImpl *** I Now exporting batch of 14 logs
and then:
FATAL EXCEPTION: okhttp-dispatch-2 Process: *.*.*, PID: 23410 java.lang.Error: java.lang.InterruptedException at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1173) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:919) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2042) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2119) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at okhttp3.internal.connection.FastFallbackExchangeFinder.awaitTcpConnect(FastFallbackExchangeFinder.kt:162) at okhttp3.internal.connection.FastFallbackExchangeFinder.find(FastFallbackExchangeFinder.kt:69) at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:280) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:101) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:85) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:74) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:96) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:208) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:530) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:919)
What did you expect to see?
I expecting, the error is handled properly and not crashing the app. Maybe I have missed some point to handle such problems by my code
What did you see instead?
I see my app crashing.
What version and what artifacts are you using?
my dependencies looking like:
implementation("io.opentelemetry.android:android-agent:0.13.0-alpha") implementation 'io.opentelemetry:opentelemetry-exporter-otlp:1.52.0' implementation('io.opentelemetry:opentelemetry-api-incubator')
I'm using OtlpGrpcSpanExporter, OtlpGrpcLogRecordExporter and OtlpGrpcMetricExporter
Environment
Compiler: (Gradle 8.9.0)
OS: (Android 10)
Additional context
Running app on the emulator
Thank you