Improve batch logic#145

Open

jfrancoa wants to merge 8 commits intomainfrom

improve-batch-logic

Commits on Jan 5, 2026

Attempt streaming the data generation.
jfrancoa
committed
Changed code to producer -> queue -> consumer pattern

Byron Voorbach
authored and
jfrancoa
committed
Support choosing dynamic batching or not
jfrancoa
committed
Make dynamic_batch way more performant.
jfrancoa
committed
Default batch size 1000

Byron Voorbach
authored and
jfrancoa
committed
Improve printing at end of indexing run (errors are overloading terminal, showing last 10 now)

Byron Voorbach
authored and
jfrancoa
committed
Cleanup and improved error logging

Byron Voorbach
authored and
jfrancoa
committed
Retry increasing timeout if connection is slow.
jfrancoa
committed