Conversation
|
Might want to tease that guaranteed_order's current single-threaded constraint is not a philosophical hard stop - we may invest in making guaranteed_order capable of batching and threading by partition key, it's just that the current implementation does not do any of those optimizations. |
| failed = stream_events.map do |event| | ||
| Journaled::KinesisFailedEvent.new( | ||
| event:, | ||
| error_code: error.class.to_s, |
There was a problem hiding this comment.
Why does this produce "ValidationException" and not "Aws::Kinesis::Errors::ValidationException"?
There was a problem hiding this comment.
it's an error code on the event returned by kinesis in the batch response, and not an actual raised exception by the ruby SDK
There was a problem hiding this comment.
Ah, okay. So the library just uses the same name for its Errors:: classes as the possible error codes the batch API might return.
|
@jmileham good call, updated the readme to reflect that future optimizations could make the guaranteed order mode parallelizable |
README.md
Outdated
| bundle exec rake journaled_worker:work | ||
| ``` | ||
|
|
||
| **Note:** In `:batch` mode (the default), you can run multiple worker processes concurrently for horizontal scaling. In `:guaranteed_order` mode, only run a single worker to maintain ordering guarantees. |
There was a problem hiding this comment.
Just to make sure I'm not missing something, if you run more than one worker, the locking strategy means only one will ever be able to act at a time. So it won't break anything, it's just inefficient and entirely unnecessary to run more than one in the guaranteed order mode.
There was a problem hiding this comment.
yeah, it's worded a bit confusingly. i guess, it's designed to be processed by a single worker at a time, but would be resilient to multiple workers running concurrently
There was a problem hiding this comment.
Oh just saw @jmileham's top-level comment -- yeah ditto.
smudge
left a comment
smudge
left a comment
There was a problem hiding this comment.
The slight differences in error-handling threw me off a little because they feel unnecessarily duplicative, but I get how in one mode Kinesis is handing you a string, and in the other the SDK is bubbling up a Ruby exception.
domain LGTM && platform LGTM on the rest -- thanks for breaking this apart into two different strategies -- I'll be eager to see how much better the batch PUTs API performs without any additional worker concurrency on our end.
…ive database polling
|
After some stage testing, i changed the behavior to always sleep between cycles. We were easily hitting kinesis rate limits, and i didn't want spiky periods of job processing to unnecessarily hit the database too often |
3 participants
Summary
Add a config option to Journaled that allows the batch outbox to be processed using
:guaranteed_order(current approach) or:batch(new approach). The:batchoption makes use ofSKIP LOCKEDin the query and the kinesisput_eventsbatch api to increase throughput at the expense of guaranteeing events are delivered in the order they are enqueued.