[Firestore Performance Issue] snapshots() are stalled

[svenjacobs]

@dconeybe @ehsannas This is related to issue #7198.

I was finally able to reproduce the issue. It seems that occasionally on a fresh app start, all invocations of snapshots() don't deliver any data. Even if the cache has been populated before, neither cached nor updated data are provided. Breakpoints after snapshots() are never caught.

I've uploaded the SQLite database here. Note, since the database contains sensitive information, I will only allow access requests coming from official Google addresses. The archive contains three folders: 01 before, 02 after and 03 after restart. before contains the database before the app was started. after contains the database after the app was started and is stuck in the frozen state. Lastly 03 after restart contains the database after the app has been restarted and Firestore queries work. One exemplary query that we use is:

firestore.collection("offers_weeks")
    .whereGreaterThanOrEqualTo(FieldPath.of("visible", "until"), Timestamp.now())
    .orderBy(FieldPath.of("visible", "until"))
    .limit(2)
    .snapshots()

We're using Firebase BOM 34.10.0.

I hope this helps in identifying the problem 🤞🏻

[dconeybe]

Thanks for reporting @svenjacobs. I tried to reproduce any sort of "stall" with the sqlite databases you provided but was unable to: every query ran in less than 100 milliseconds. Also, the sqlite databases you showed me a pretty small (only about 1 megabyte) so I wouldn't expect caching to be a performance bottleneck.

The test app that I used I pushed up to a branch: https://github.com/firebase/firebase-android-sdk/tree/dconeybe/firestore/testapp/firebase-firestore/testapp

I was wondering if you'd be able to tinker around with the app and see if you can reproduce the problematic behavior reliably. If you can, do the following:

1. git clone https://github.com/firebase/firebase-android-sdk.git
2. cd firebase-android-sdk
3. git checkout -b testapp remotes/origin/dconeybe/firestore/testapp
4. Open the firebase-android-sdk directory in Android Studio (do NOT open the testapp directory itself, as it is a submodule of the overall Firebase Android project)
5. Follow the instructions at the bottom of README.md
6. When you (eventually) see the "Run Configurations" in Android Studio populate, run firebase-firestore.testapp
7. Check logcat for messages.

If you are able to reproduce the stall reliably with this app then we can do some profiling and step right into the Firestore source code to see if we can figure out what's going on.

Let me know what you think.

[svenjacobs]

@dconeybe Okay, I will try to reproduce the stall with the test app. One more thing though: when this stall happens, it affects all Firestore queries at once. So when opening different screens of the app, they are all stuck in a loading state. Then suddenly the stall disappears and all screens behave normally. Maybe we are looking at the wrong place? Maybe it's not the database but a network issue, for example? Does this ring a bell?

[dconeybe]

Hmm well under the hood the Firestore SDK does all of its "work" on a sequential queue (known internally as the "AsyncQueue"). So if one task on this queue stalls then it blocks every task enqueued behind it. So likely the investigation will involved finding the problematic task in the queue and then going from there.

[svenjacobs]

In case this happens again in our app, can you give me a hint where in the Firebase SDK I can debug to find out which task is blocking?

[dconeybe]

Sure! Although, it's tough to narrow it down too much given that the actual source of the problem isn't known. But back a few months ago when I made a bunch of performance improvements to query execution against the local cache I was focussing my efforts around https://github.com/firebase/firebase-android-sdk/blob/main/firebase-firestore/src/main/java/com/google/firebase/firestore/local/SQLiteRemoteDocumentCache.java.

That actually reminds me, if you are able to reproduce the issue with Firestore debug logging enabled then that could also point to a root cause. That's another avenue you could try.

[guillermoST]

We've had the same issue.

We are using react-native-firebase 23.8.4, which uses firebase-android-sdk 34.7.0. When we upgraded to 23.8.6, which uses android-sdk 34.10.0, for some reason, our threads would sporadically start to stall, basically grinding our Android version to a halt.

iOS has always continued to work.

It would stay stalled through app restarts; the only way was to log out of Firebase, delete the app, and reinstall, wiping the SQLite DB. When we were in this situation, we downgraded to 34.7.0 and then the app worked like a charm again, no reinstall was done, so probably there was not SQLite corruption.

Not quite sure what the changes were from 34.7.0 to 34.10.0, but it looks like while trying to improve performance the whole android snapshot listeners took a hit.