Skip to content

App crashes after some period of downtime of the otel endpoint #1134

@50c

Description

@50c

Describe the bug
My app is crashing after opentelemetrty server is unavailable for a while. It seems to be very close to #open-telemetry/opentelemetry-java#6946 with same behavior but some different message

Steps to reproduce
I'm using io.opentelemetry.android:android-agent:0.13.0-alpha based on io.opentelemetry:opentelemetry-exporter-otlp:1.52.0 and it is working great until I shutdown the server and wait for about a couple of minutes. In the meanwhile the application log says:
Failed to export logs. The request could not be executed. Error message: timeout
java.io.InterruptedIOException: timeout
at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:430)
at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:392)
at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:351)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:216)
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:530)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:919)
Caused by: java.io.IOException: Canceled
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:70)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:96)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:208)
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:530) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
at java.lang.Thread.run(Thread.java:919) 

periodically, which is expected. But after some time the app just crashing with following messages:

FromDiskExporterImpl Attempting to export logs batch from disk. FromDiskExporterImpl *** I Attempting to read data from ReadableFile{file=/data/user/0/*.*.*/cache/opentelemetry/signals/logs/1754561713102} FromDiskExporterImpl *** I Read 10809 logs bytes from storage. FromDiskExporterImpl *** I Now exporting batch of 14 logs

and then:

FATAL EXCEPTION: okhttp-dispatch-2 Process: *.*.*, PID: 23410 java.lang.Error: java.lang.InterruptedException at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1173) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:919) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2042) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2119) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at okhttp3.internal.connection.FastFallbackExchangeFinder.awaitTcpConnect(FastFallbackExchangeFinder.kt:162) at okhttp3.internal.connection.FastFallbackExchangeFinder.find(FastFallbackExchangeFinder.kt:69) at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:280) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:101) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:85) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:74) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor.intercept(RetryInterceptor.java:96) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:126) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:208) at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:530) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)  at java.lang.Thread.run(Thread.java:919)

What did you expect to see?
I expecting, the error is handled properly and not crashing the app. Maybe I have missed some point to handle such problems by my code

What did you see instead?
I see my app crashing.

What version and what artifacts are you using?
my dependencies looking like:
implementation("io.opentelemetry.android:android-agent:0.13.0-alpha") implementation 'io.opentelemetry:opentelemetry-exporter-otlp:1.52.0' implementation('io.opentelemetry:opentelemetry-api-incubator')
I'm using OtlpGrpcSpanExporter, OtlpGrpcLogRecordExporter and OtlpGrpcMetricExporter

Environment
Compiler: (Gradle 8.9.0)
OS: (Android 10)

Additional context
Running app on the emulator

Thank you

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions