[MongoDB Storage] Increase timeouts for checksums from 40s -> 80s #338
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#300 introduced proper error messages for checksum queries timing out. A side effect of that was also reducing the 60s connection-based timeout to a 40s query-specific timeout.
While the results are cached in memory and incrementally updated, in some cases the initial calculation takes too long, causing the service to repeatedly run the same query. We've seen this hit the 40s timeout when calculating checksums over around 5-10 million operations at a time.
This is a quick-fix that just doubles the timeout, which should make it safe to sync up to around 10 million operations per user. A proper future fix will focus on significantly reducing the time needed to calculate these initial checksums, after which we can reduce the timeout values again.