[BUG] AWS DynamoDB Stream Event not emitting all events #16109
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ |
WalkthroughThis pull request updates version numbers and refactors shard iterator management logic. In the AWS component’s package.json, the @pipedream/aws package version is incremented from 0.7.7 to 0.7.8. In the New DynamoDB Stream Event component, the shard iterator management within the deploy and run hooks is overhauled, including method renaming and removal of outdated error handling. Additionally, the component’s version is updated from 0.0.4 to 0.0.5 and its summary string in the generateMeta method is refined. Changes
Sequence Diagram(s)sequenceDiagram
participant Component as NewDynamoDBStreamEvent Component
participant Stream as DynamoDB Stream
Note over Component,Stream: Deploy Phase - Retrieve Shard Iterators
Component->>Stream: Request shard details
Stream-->>Component: Return list of shards
Component->>Component: Identify active shards and retrieve iterators
Component-->>Component: Store shard iterators
sequenceDiagram
participant Component as NewDynamoDBStreamEvent Component
participant Stream as DynamoDB Stream
Note over Component,Stream: Run Phase - Process Records
loop For each shard iterator
Component->>Stream: Fetch records using current iterator
Stream-->>Component: Return records
Component->>Component: Process records and update iterator
end
Assessment against linked issues
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs ✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs (4)
39-78: Consider adding shard pagination support.Currently,
listShardsis limited to 100 shards. In high-throughput scenarios with more than 100 shards, additional shards won't be processed. A follow-up improvement might paginate through all shards to ensure full coverage.
82-87: Persistence of shard iterators is straightforward.Storing iterators in
this.dbis simple and effective. If concurrency or re-entrancy is possible, consider implementing locking or conflict resolution to handle multiple processes updating shard iterators at once.
99-106: Potential partial retrieval of records.The limit of 100 records in each run means larger batches are split across multiple runs. If you prefer processing all new records within the same execution, consider looping until
NextShardIteratorindicates no more records or a threshold has been met.
108-139: Consider removing completed shards.If
NextShardIteratorbecomes null because a shard is closed, you could remove it fromshardIteratorsto avoid pointless checks in subsequent runs.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (2)
components/aws/package.json(1 hunks)components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: Verify TypeScript components
- GitHub Check: pnpm publish
- GitHub Check: Publish TypeScript components
🔇 Additional comments (3)
components/aws/package.json (1)
3-3: Version bump noticed.This minor version increment helps track and distribute the recent improvements. Ensure you provide release notes or changelog entries if required for proper documentation.
components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs (2)
9-9: Component version update looks good.Incrementing from
0.0.4to0.0.5aligns with the recent refactoring changes and ensures clarity in tracking new features.
93-93: Clearer event summary.Changing the summary to
New Event: ${eventName}improves readability. No issues here.
jcortes
deleted the
fix-aws-dynamodb-stream-event-not-emitting-all-events
branch
WHY
Resolves #16056
Summary by CodeRabbit