Ensure last page of documents is included in metadata cache #13141
+7
−6
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
When the metadata cache for documents is being built, we build it page by page in a while loop. We were deciding when to break by using the number of pages we'd fetched so far multiplied by the number of documents per page and comparing to the total to understand how many documents were remaining. The problem was we were incrementing the page count before this calculation, ensuring that we missed the last page of documents.
For example.
I have 288 documents. If I use a page size of 30 then there are 10 pages of documents:
288 / 30 = 9.6So once we've fetched 9 pages of documents we'd have 270 documents cached:
9 * 30 = 270which leaves 18 documents remaining on the last page. But since we increment the page count first the calculation to determine how many documents remain is:288 - (10*30) = -12instead of288 - (9*30) = 18. The result is the last 18 documents are missed.We could just move the page increment after this calculation but I feel like this solution is over-complicated. The cleaner thing to do is just break on the first page with no documents.
Type of change