SimpleMongoRepository.java's public Page<T> findAll(Pageable pageable) runs 1000 times slower than before

[herbertwu]

The findAll(Pageable pageable) method on a given collection calls count() first. The count() call uses MongoDB driver's collection.countDocuments(emptyQuery) to get the result.

However MongoDB driver collection.countDocuments(emptyQuery) changed the implementation lately and uses the much slower "aggregate" query than the original "count" query.
The "count" query is very fast. And MongoDB driver added a new method collection.estimatedDocumentCount() to use "count" query.

Can we enhance findAll(Pageable pageable) to add option to use the estimatedDocumentCount()?

In my application, findAll(Pageable pageable) method(finding oldest 100 records by timestamp) runs about ~1000 times slower after driver upgrade because of the count() using MongoDB driver's exact count.

[mp911de]

estimatedDocumentCount() is an estimate (derived from collection statistics) and doesn't necessarily reflect the actual count, see also MongoDB's warnings about estimatedDocumentCount

[herbertwu]

In my case, I ONLY need to find oldest 100 records in a large collection and no need for actual total count. So estimatedDocumentCount() option is perfect. I am not asking to replace collection.countDocuments().

The main problem with collection.countDocuments() is it is so slow with an "aggregate" query even on a small collection with perfect index, and renders the paging process useless.

Of course if there is way for Spring to automatically generate some kind of hint for collection.countDocuments(query) to run fast, I am all for it.

[herbertwu]

My use case is sort of like google search: it needs to be fast and result is paged, but users rarely go to second page, almost never go to third page...

[herbertwu]

BTW, we have been running MongoDBv3.x for 7 years in which collection.countDocuments(query) is using the estimated query, and we never had the issues mentioned in the estimatedDocumentCount() API doc.
We only noticed the slowness of collection.countDocuments(query) recently after upgrading to MongoDB v6.x. It looks like estimatedDocumentCount() was introduced in MongoDB v4.x

[christophstrobl]

Have you tried to set MongoTemplate#useEstimatedCount(boolean) to true as mentioned in the documentation?

[herbertwu]

Hmm, no. I am using org.springframework.data.mongodb.repository.MongoRepository.findAll(pageable) interface. I do not use MongoTemplate at all in the app.

[christophstrobl]

The repository operates on top of the template.

[herbertwu]

Awesome. it worked. Thanks so much folks! No way I could figure the flag out myself.

[christophstrobl]

Good to hear it works for you now.