Internal error in Cloud Firestore (26.0.0) on some Xiaomi / Oppo

[mathieuedet]

[REQUIRED] Step 2: Describe your environment

Environment

• Android Studio version: Narwhal Feature Drop | 2025.1.2 Patch 1
• Firebase Component: firestore
• Component version: 26.0.0

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

After upgrading to firebase-bom 34.1.0, our application is experiencing some crashes mainly on Xiaomi and Oppo devices (49% Xiaomi, 45% Oppo, 4% Samsung and some others like unknown phone brands) on production. The crashes are related to Cloud Firestore and appear to be caused by a corruption in the preferences protocol buffer. For information, we don't use datastore in our application.

The application crashes with the following exception stack trace (retrieved from crashlytics) :

Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.0.0).
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
       at com.google.firebase.firestore.util.AsyncQueue$$ExternalSyntheticLambda3.run(D8$$SyntheticClass)
       at android.os.Handler.handleCallback(Handler.java:959)
       at android.os.Handler.dispatchMessage(Handler.java:100)
       at android.os.Looper.loopOnce(Looper.java:249)
       at android.os.Looper.loop(Looper.java:337)
       at android.app.ActivityThread.main(ActivityThread.java:9500)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:636)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1005)

Caused by androidx.datastore.core.CorruptionException: Unable to parse preferences proto.
       at androidx.datastore.preferences.PreferencesMapCompat$Companion.readFrom(PreferencesMapCompat.kt:34)
       at androidx.datastore.preferences.core.PreferencesSerializer.readFrom(PreferencesSerializer.jvm.kt:46)
       at androidx.datastore.core.okio.OkioReadScope.readData$suspendImpl(OkioStorage.kt:180)
       at androidx.datastore.core.okio.OkioReadScope.readData()
       at androidx.datastore.core.StorageConnectionKt$readData$2.invokeSuspend(StorageConnection.kt:74)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:9)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:10)
       at androidx.datastore.core.okio.OkioStorageConnection.readScope(OkioStorage.kt:113)
       at androidx.datastore.core.StorageConnectionKt.readData(StorageConnection.kt:74)
       at androidx.datastore.core.DataStoreImpl.readDataFromFileOrDefault(DataStoreImpl.kt:323)
       at androidx.datastore.core.DataStoreImpl.readDataOrHandleCorruption(DataStoreImpl.kt:365)
       at androidx.datastore.core.DataStoreImpl.access$readDataOrHandleCorruption(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invokeSuspend(DataStoreImpl.kt:437)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:8)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:2)
       at androidx.datastore.core.SingleProcessCoordinator.lock(SingleProcessCoordinator.kt:41)
       at androidx.datastore.core.DataStoreImpl$InitDataStore.doRun(DataStoreImpl.kt:434)
       at androidx.datastore.core.RunOnce.runIfNeeded(DataStoreImpl.kt:497)
       at androidx.datastore.core.DataStoreImpl.readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:266)
       at androidx.datastore.core.DataStoreImpl.access$readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$readState$2.invokeSuspend(DataStoreImpl.kt:218)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

Caused by androidx.datastore.preferences.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
       at androidx.datastore.preferences.protobuf.CodedInputStream$StreamDecoder.readTag(CodedInputStream.java:2074)
       at androidx.datastore.preferences.protobuf.CodedInputStreamReader.getFieldNumber(CodedInputStreamReader.java:59)
       at androidx.datastore.preferences.protobuf.MessageSchema.mergeFromHelper(MessageSchema.java:2973)
       at androidx.datastore.preferences.protobuf.MessageSchema.mergeFrom(MessageSchema.java:2955)
       at androidx.datastore.preferences.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1599)
       at androidx.datastore.preferences.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1742)
       at androidx.datastore.preferences.PreferencesProto$PreferenceMap.parseFrom(PreferencesProto.java:198)
       at androidx.datastore.preferences.PreferencesMapCompat$Companion.readFrom(PreferencesMapCompat.kt:32)
       at androidx.datastore.preferences.core.PreferencesSerializer.readFrom(PreferencesSerializer.jvm.kt:46)
       at androidx.datastore.core.okio.OkioReadScope.readData$suspendImpl(OkioStorage.kt:180)
       at androidx.datastore.core.okio.OkioReadScope.readData()
       at androidx.datastore.core.StorageConnectionKt$readData$2.invokeSuspend(StorageConnection.kt:74)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:9)
       at androidx.datastore.core.StorageConnectionKt$readData$2.invoke(:10)
       at androidx.datastore.core.okio.OkioStorageConnection.readScope(OkioStorage.kt:113)
       at androidx.datastore.core.StorageConnectionKt.readData(StorageConnection.kt:74)
       at androidx.datastore.core.DataStoreImpl.readDataFromFileOrDefault(DataStoreImpl.kt:323)
       at androidx.datastore.core.DataStoreImpl.readDataOrHandleCorruption(DataStoreImpl.kt:365)
       at androidx.datastore.core.DataStoreImpl.access$readDataOrHandleCorruption(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invokeSuspend(DataStoreImpl.kt:437)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:8)
       at androidx.datastore.core.DataStoreImpl$InitDataStore$doRun$initData$1.invoke(:2)
       at androidx.datastore.core.SingleProcessCoordinator.lock(SingleProcessCoordinator.kt:41)
       at androidx.datastore.core.DataStoreImpl$InitDataStore.doRun(DataStoreImpl.kt:434)
       at androidx.datastore.core.RunOnce.runIfNeeded(DataStoreImpl.kt:497)
       at androidx.datastore.core.DataStoreImpl.readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:266)
       at androidx.datastore.core.DataStoreImpl.access$readAndInitOrPropagateAndThrowFailure(DataStoreImpl.kt:48)
       at androidx.datastore.core.DataStoreImpl$readState$2.invokeSuspend(DataStoreImpl.kt:218)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
       at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
       at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
       at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
       at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[georgeselkhoury]

Happening with me too

[Phillip-Schackier]

Same problem here, we're trying to downgrade to 33.16.0.

[dconeybe]

@mathieuedet You mentioned that "After upgrading to firebase-bom 34.1.0" you started seeing this problem. Which BoM version were you using before that, where this issue did not occur? It will help to be able to determine the diff between those two versions to see if a culprit can be determined.

[mathieuedet]

@dconeybe Before the upgrade, we were on firebase-bom version 33.12.0. Upon checking, we found that internal errors were already occurring with cloud-firestore 25.1.3 before the update although the errors were not identical (potentially related but uncertain). If it is related, this information might be helpful to you.

Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (25.1.3).
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
       at com.google.firebase.firestore.util.AsyncQueue$$ExternalSyntheticLambda3.run(D8$$SyntheticClass)
       at android.os.Handler.handleCallback(Handler.java:959)
       at android.os.Handler.dispatchMessage(Handler.java:100)
       at android.os.Looper.loopOnce(Looper.java:257)
       at android.os.Looper.loop(Looper.java:342)
       at android.app.ActivityThread.main(ActivityThread.java:9638)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:619)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:929)

Caused by java.lang.AssertionError: INTERNAL ASSERTION FAILED: Document leak -- detected dangling mutation references when queue is empty. Dangling keys: [**********/**********]
       at com.google.firebase.firestore.util.Assert.fail(Assert.java:63)
       at com.google.firebase.firestore.util.Assert.hardAssert(Assert.java:31)
       at com.google.firebase.firestore.local.SQLiteMutationQueue.performConsistencyCheck(SQLiteMutationQueue.java:433)
       at com.google.firebase.firestore.local.LocalStore.lambda$acknowledgeBatch$3$com-google-firebase-firestore-local-LocalStore(LocalStore.java:318)
       at com.google.firebase.firestore.local.LocalStore$$ExternalSyntheticLambda5.get(D8$$SyntheticClass)
       at com.google.firebase.firestore.local.SQLitePersistence.runTransaction(SQLitePersistence.java:233)
       at com.google.firebase.firestore.local.LocalStore.acknowledgeBatch(LocalStore.java:312)
       at com.google.firebase.firestore.core.SyncEngine.handleSuccessfulWrite(SyncEngine.java:494)
       at com.google.firebase.firestore.core.MemoryComponentProvider$RemoteStoreCallback.handleSuccessfulWrite(MemoryComponentProvider.java:130)
       at com.google.firebase.firestore.remote.RemoteStore.handleWriteStreamMutationResults(RemoteStore.java:685)
       at com.google.firebase.firestore.remote.RemoteStore.access$500(RemoteStore.java:60)
       at com.google.firebase.firestore.remote.RemoteStore$2.onWriteResponse(RemoteStore.java:213)
       at com.google.firebase.firestore.remote.WriteStream.onNext(WriteStream.java:189)
       at com.google.firebase.firestore.remote.WriteStream.onNext(WriteStream.java:49)
       at com.google.firebase.firestore.remote.AbstractStream$StreamObserver.lambda$onNext$1$com-google-firebase-firestore-remote-AbstractStream$StreamObserver(AbstractStream.java:126)
       at com.google.firebase.firestore.remote.AbstractStream$StreamObserver$$ExternalSyntheticLambda0.run(D8$$SyntheticClass)
       at com.google.firebase.firestore.remote.AbstractStream$CloseGuardedRunner.run(AbstractStream.java:67)
       at com.google.firebase.firestore.remote.AbstractStream$StreamObserver.onNext(AbstractStream.java:113)
       at com.google.firebase.firestore.remote.FirestoreChannel$1.onMessage(FirestoreChannel.java:162)
       at io.grpc.internal.DelayedClientCall$DelayedListener.onMessage(DelayedClientCall.java:473)
       at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:667)
       at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:654)
       at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
       at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:524)
       at java.util.concurrent.FutureTask.run(FutureTask.java:317)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:348)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:651)
       at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
       at java.lang.Thread.run(Thread.java:1119)

@Phillip-Schackier Don't hesitate to share the result of downgrade, I'm not really confident about a downgrading the library.

[dconeybe]

Hmm both of those stack traces are definitely different, but both are plausibly symptoms of local data corruption. The stack trace from the OP looks like some data saved by androidx.datastore.preferences got corrupted, and the stack trace from #7286 (comment) looks like some data in the local sqlite database got corrupted.

Do you have any other stack traces you can share? Do you see this error on other devices, or just Xiaomi/Oppo devices? If you are able to reproduce, could you try doing so with Firestore debug logging enabled and provide those logs as well?

At the moment I have no idea what's going on so will need to take some time to think about this.

[dconeybe]

Also, do you have any evidence of this bug with firebase-firestore versions less than 25.1.3 (e.g. 25.1.2)?

I ask because a change went into version 25.1.3 that updated the protolite-well-known-types dependency which could theoretically be related to the proto parsing error from the op. This seems like an unlikely root cause, but seems worthwhile to at least consider at this point.

https://firebase.google.com/support/release-notes/android#firestore_v25-1-3

[dconeybe]

Update: The CorruptionException appears to be a bug in the androidx.datastore.preferences dependency that was fixed in its version 1.1.5. (see #7191)

@mathieuedet Could you check which version of this dependency you have?

For example, I ran the following command in the https://github.com/firebase/quickstart-android repository in a bash shell to find that it is using v1.1.3:

$ ./gradlew :firestore:app:dependencies 2>&1 | grep datastore-preferences: | sed -e 's/|\|\s\+\|+\|\\//g' | sort | uniq
---androidx.datastore:datastore-preferences:1.1.3
---androidx.datastore:datastore-preferences:1.1.3(c)
---androidx.datastore:datastore-preferences:{strictly1.1.3}->1.1.3(c)

If you do find that it's using an older version, you may be able to force-upgrade it to a new version by adding this line into your build.gradle.kts (firestore/app/build.gradle.kts in the case of quickstart-android) in the dependencies section:

implementation("androidx.datastore:datastore-preferences:1.1.7")

After making that change, re-running the ./gradlew :firestore:app:dependencies command above showed that the dependency had been upgraded:

$ ./gradlew :firestore:app:dependencies 2>&1 | grep datastore-preferences: | sed -e 's/|\|\s\+\|+\|\\//g' | sort | uniq
---androidx.datastore:datastore-preferences:1.1.3->1.1.7
---androidx.datastore:datastore-preferences:1.1.3->1.1.7(*)
---androidx.datastore:datastore-preferences:1.1.7
---androidx.datastore:datastore-preferences:1.1.7(c)
---androidx.datastore:datastore-preferences:1.1.7(n)
---androidx.datastore:datastore-preferences:{strictly1.1.7}->1.1.7(c)

[mathieuedet]

Thank you for your analysis @dconeybe. Indeed, the version of the dependency is 1.1.3 and it is version 1.1.7 which is used when I manually add this dependency in my build.gradle.kts.

Do you think Firestore will work properly if I manually install this version, or would you advise me to wait for a fix from you with the updated dependency? Do you know release date of a fix from your side with the right datastore version ?