Ingest records immediately when next payload will exceed buffer limit#336
Merged
timacdonald merged 7 commits into1.xfrom Mar 3, 2026
Merged
Ingest records immediately when next payload will exceed buffer limit#336timacdonald merged 7 commits into1.xfrom
timacdonald merged 7 commits into1.xfrom
Conversation
timacdonald
force-pushed
the
digest-before-when-exceeding-limit
branch
from
9d247fa to
29492da
Compare
…48E8AD843750845E011E19E53D7CAC9C92382E1242D4A83933DF9A0686B5A9A6E0330D69100C8A047
timacdonald
commented
Mar 2, 2026
timacdonald
commented
Mar 2, 2026
avosalmon
approved these changes
Mar 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We currently ingest records once the buffer reaches the threshold. The problem is we don't account for the incoming payload when deciding when to ingest.
Imagine we have ingested 7mb of data. The threshold is 8mb.
When a 2mb payload comes in, we append it and ingest. This means the payload was 9mb.
With these changes, we check if the incoming payload will put the buffer over the threshold before appending. If it does, we ingest the current buffer and then use the payload as the start of the new buffer.
This will likely cause more ingesting to occur, but it should mean ingest sizes are smaller and we don't hit issues with larger than expected payloads.