Skip to content

Comments

feat(etl): add stream length checks to prevent overflow#92

Merged
armyhaylenko merged 1 commit intomainfrom
feat/MTG-1320-check-len
Feb 10, 2025
Merged

feat(etl): add stream length checks to prevent overflow#92
armyhaylenko merged 1 commit intomainfrom
feat/MTG-1320-check-len

Conversation

@armyhaylenko
Copy link
Contributor

This pr:

  1. adds a stream counter to the snapshot binary, which counts how many consecutive messages were sent
  2. when the stream counter reaches some constant value (20M in our case), check accounts stream length to be below the desired value
  3. if it's above the desired value, sleep in 1-second intervals until the stream length drops below the desired value
  4. reset the stream counter and send another batch of accounts to the redis instance.

n00m4d
n00m4d previously approved these changes Feb 10, 2025
@armyhaylenko armyhaylenko merged commit ce6a79d into main Feb 10, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants