[Components] Amazon Redshift - new components #17457
Conversation
|
""" WalkthroughThis update introduces comprehensive support for Amazon Redshift within the AWS integration, including new actions for creating, updating, deleting, and querying rows, as well as new polling sources for detecting new and updated rows. The AWS app module is extended with Redshift Serverless and Data API capabilities, and new dependencies are added. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Action (Create/Update/Delete/Query)
participant AWS App
participant Redshift
User->>Action (Create/Update/Delete/Query): Triggers action with parameters
Action (Create/Update/Delete/Query)->>AWS App: Calls executeStatement with SQL and parameters
AWS App->>Redshift: Executes SQL statement via Data API
Redshift-->>AWS App: Returns execution result
AWS App-->>Action (Create/Update/Delete/Query): Returns result
Action (Create/Update/Delete/Query)-->>User: Outputs summary and data
sequenceDiagram
participant Source (New/Updated Row)
participant AWS App
participant Redshift
Source (New/Updated Row)->>AWS App: Calls executeStatement with polling SQL
AWS App->>Redshift: Executes SQL statement via Data API
Redshift-->>AWS App: Returns query results
AWS App-->>Source (New/Updated Row): Returns records
Source (New/Updated Row)-->>Source (New/Updated Row): Emits events for new/updated rows
Assessment against linked issues
Assessment against linked issues: Out-of-scope changesNo out-of-scope changes found. Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 3 Skipped Deployments
|
jcortes
force-pushed
the
amazon-redshift-new-components
branch
from
100750b to
c0cabea
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (37)
components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs(1 hunks)components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs(1 hunks)components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs(1 hunks)components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs(1 hunks)components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs(1 hunks)components/aws/actions/dynamodb-query/dynamodb-query.mjs(1 hunks)components/aws/actions/dynamodb-scan/dynamodb-scan.mjs(1 hunks)components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs(1 hunks)components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs(1 hunks)components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs(1 hunks)components/aws/actions/lambda-create-function/lambda-create-function.mjs(1 hunks)components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs(1 hunks)components/aws/actions/redshift-create-rows/redshift-create-rows.mjs(1 hunks)components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs(1 hunks)components/aws/actions/redshift-query-database/redshift-query-database.mjs(1 hunks)components/aws/actions/redshift-update-rows/redshift-update-rows.mjs(1 hunks)components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs(1 hunks)components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs(1 hunks)components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs(1 hunks)components/aws/actions/s3-upload-files/s3-upload-files.mjs(1 hunks)components/aws/actions/sns-send-message/sns-send-message.mjs(1 hunks)components/aws/actions/sqs-send-message/sqs-send-message.mjs(1 hunks)components/aws/aws.app.mjs(3 hunks)components/aws/package.json(2 hunks)components/aws/sources/common/redshift-polling.mjs(1 hunks)components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs(1 hunks)components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs(1 hunks)components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs(1 hunks)components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs(1 hunks)components/aws/sources/new-sns-messages/new-sns-messages.mjs(1 hunks)components/aws/sources/rds-new-event/rds-new-event.mjs(1 hunks)components/aws/sources/redshift-new-row/redshift-new-row.mjs(1 hunks)components/aws/sources/redshift-updated-row/redshift-updated-row.mjs(1 hunks)components/aws/sources/s3-deleted-file/s3-deleted-file.mjs(1 hunks)components/aws/sources/s3-new-event/s3-new-event.mjs(1 hunks)components/aws/sources/s3-new-file/s3-new-file.mjs(1 hunks)components/aws/sources/s3-restored-file/s3-restored-file.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (28)
- components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs
- components/aws/sources/s3-deleted-file/s3-deleted-file.mjs
- components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs
- components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs
- components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs
- components/aws/sources/new-sns-messages/new-sns-messages.mjs
- components/aws/actions/sns-send-message/sns-send-message.mjs
- components/aws/sources/s3-new-event/s3-new-event.mjs
- components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs
- components/aws/actions/lambda-create-function/lambda-create-function.mjs
- components/aws/sources/s3-restored-file/s3-restored-file.mjs
- components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs
- components/aws/actions/sqs-send-message/sqs-send-message.mjs
- components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs
- components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs
- components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs
- components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs
- components/aws/actions/dynamodb-scan/dynamodb-scan.mjs
- components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs
- components/aws/sources/rds-new-event/rds-new-event.mjs
- components/aws/sources/s3-new-file/s3-new-file.mjs
- components/aws/actions/s3-upload-files/s3-upload-files.mjs
- components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs
- components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs
- components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs
- components/aws/actions/dynamodb-query/dynamodb-query.mjs
- components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs
- components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs
🚧 Files skipped from review as they are similar to previous changes (7)
- components/aws/package.json
- components/aws/actions/redshift-create-rows/redshift-create-rows.mjs
- components/aws/sources/common/redshift-polling.mjs
- components/aws/actions/redshift-query-database/redshift-query-database.mjs
- components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs
- components/aws/actions/redshift-update-rows/redshift-update-rows.mjs
- components/aws/aws.app.mjs
🧰 Additional context used
🧠 Learnings (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (2)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (2)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
🪛 Biome (1.9.4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs
[error] 75-75: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs
[error] 77-77: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: Lint Code Base
- GitHub Check: Publish TypeScript components
- GitHub Check: Verify TypeScript components
🔇 Additional comments (8)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (4)
1-32: LGTM! Good component structure and configuration.The component properly extends the common redshift-polling module and correctly configures the uniqueColumn prop with dynamic options. The metadata and prop definitions follow established patterns.
46-61: Good SQL parameterization for security.The SQL query construction properly uses parameterized queries to prevent SQL injection attacks. The conditional WHERE clause logic for incremental polling is implemented correctly.
80-88: Correct implementation of event emission and ordering.The logic correctly reverses the records to emit events in ascending order (oldest first) and properly constructs the event metadata with ID, timestamp, and summary.
90-96: Proper state management for incremental polling.The lastId tracking is implemented correctly to ensure only new rows are processed in subsequent polls. The logic handles empty result sets appropriately.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (4)
1-35: LGTM! Proper configuration for timestamp-based polling.The component correctly extends the common module and properly configures the timestampColumn prop with a filter for timestamp types only. The component name appropriately omits "New" prefix since it tracks updated items, consistent with component guidelines.
49-62: Good SQL parameterization and timestamp filtering.The SQL query construction properly uses parameterized queries and correctly implements timestamp-based filtering for incremental polling.
81-89: Correct timestamp handling and event emission.The logic properly converts the timestamp to milliseconds for the event timestamp and correctly constructs the event metadata. The ordering logic ensures events are emitted in chronological order.
91-97: Proper state management for timestamp-based polling.The lastTimestamp tracking is implemented correctly to ensure only updated rows since the last poll are processed. The logic handles empty result sets appropriately.
luancazarine
left a comment
luancazarine
left a comment
There was a problem hiding this comment.
Hi @jcortes, LGTM! Ready for QA!
jcortes
force-pushed
the
amazon-redshift-new-components
branch
from
c0cabea to
f58e1fd
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
73-78: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => { - return record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {}); - }); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (1)
75-79: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => - record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {})); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });
🧹 Nitpick comments (2)
components/aws/common/utils.mjs (2)
58-95: Consider performance optimization and edge case handling.The
parseJsonfunction has a few areas for improvement:
- Performance: The use of
Object.assignin the reduce accumulator on line 87 creates unnecessary object copies- Edge case: The function doesn't handle arrays consistently - it only processes objects but not arrays at the top level
- return Object.entries(value) - .reduce((acc, [ - key, - val, - ]) => Object.assign(acc, { - [key]: parse(val), - }), {}); + if (Array.isArray(value)) { + return value.map(parse); + } + const result = {}; + for (const [key, val] of Object.entries(value)) { + result[key] = parse(val); + } + return result;
97-103: Add input validation for better error handling.The
parseArrayfunction should handle edge cases where input might be null or undefined to prevent potential runtime errors.function parseArray (input, maxDepth = 100) { - if (!Array.isArray(input)) { + if (!input || !Array.isArray(input)) { return input; } return input.map((item) => parseJson(item, maxDepth)); }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (38)
components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs(1 hunks)components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs(1 hunks)components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs(1 hunks)components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs(1 hunks)components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs(1 hunks)components/aws/actions/dynamodb-query/dynamodb-query.mjs(1 hunks)components/aws/actions/dynamodb-scan/dynamodb-scan.mjs(1 hunks)components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs(1 hunks)components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs(1 hunks)components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs(1 hunks)components/aws/actions/lambda-create-function/lambda-create-function.mjs(1 hunks)components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs(1 hunks)components/aws/actions/redshift-create-rows/redshift-create-rows.mjs(1 hunks)components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs(1 hunks)components/aws/actions/redshift-query-database/redshift-query-database.mjs(1 hunks)components/aws/actions/redshift-update-rows/redshift-update-rows.mjs(1 hunks)components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs(1 hunks)components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs(1 hunks)components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs(1 hunks)components/aws/actions/s3-upload-files/s3-upload-files.mjs(1 hunks)components/aws/actions/sns-send-message/sns-send-message.mjs(1 hunks)components/aws/actions/sqs-send-message/sqs-send-message.mjs(1 hunks)components/aws/aws.app.mjs(4 hunks)components/aws/common/utils.mjs(1 hunks)components/aws/package.json(2 hunks)components/aws/sources/common/redshift-polling.mjs(1 hunks)components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs(1 hunks)components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs(1 hunks)components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs(1 hunks)components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs(1 hunks)components/aws/sources/new-sns-messages/new-sns-messages.mjs(1 hunks)components/aws/sources/rds-new-event/rds-new-event.mjs(1 hunks)components/aws/sources/redshift-new-row/redshift-new-row.mjs(1 hunks)components/aws/sources/redshift-updated-row/redshift-updated-row.mjs(1 hunks)components/aws/sources/s3-deleted-file/s3-deleted-file.mjs(1 hunks)components/aws/sources/s3-new-event/s3-new-event.mjs(1 hunks)components/aws/sources/s3-new-file/s3-new-file.mjs(1 hunks)components/aws/sources/s3-restored-file/s3-restored-file.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (6)
- components/aws/actions/lambda-create-function/lambda-create-function.mjs
- components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs
- components/aws/sources/rds-new-event/rds-new-event.mjs
- components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs
- components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs
- components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs
🚧 Files skipped from review as they are similar to previous changes (29)
- components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs
- components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs
- components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs
- components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs
- components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs
- components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs
- components/aws/sources/s3-new-file/s3-new-file.mjs
- components/aws/sources/common/redshift-polling.mjs
- components/aws/actions/sns-send-message/sns-send-message.mjs
- components/aws/actions/redshift-create-rows/redshift-create-rows.mjs
- components/aws/actions/redshift-update-rows/redshift-update-rows.mjs
- components/aws/package.json
- components/aws/actions/sqs-send-message/sqs-send-message.mjs
- components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs
- components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs
- components/aws/sources/s3-deleted-file/s3-deleted-file.mjs
- components/aws/actions/s3-upload-files/s3-upload-files.mjs
- components/aws/sources/new-sns-messages/new-sns-messages.mjs
- components/aws/actions/dynamodb-query/dynamodb-query.mjs
- components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs
- components/aws/sources/s3-new-event/s3-new-event.mjs
- components/aws/sources/s3-restored-file/s3-restored-file.mjs
- components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs
- components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs
- components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs
- components/aws/actions/dynamodb-scan/dynamodb-scan.mjs
- components/aws/actions/redshift-query-database/redshift-query-database.mjs
- components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs
- components/aws/aws.app.mjs
🧰 Additional context used
🧠 Learnings (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
🪛 Biome (1.9.4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs
[error] 75-75: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs
[error] 77-77: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: Publish TypeScript components
- GitHub Check: Verify TypeScript components
- GitHub Check: pnpm publish
- GitHub Check: Lint Code Base
🔇 Additional comments (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
46-51: Good use of parameterized queries for SQL injection protection.The implementation correctly uses parameterized queries to prevent SQL injection vulnerabilities when filtering by
lastId.components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (1)
31-31: Excellent column filtering for timestamp-based tracking.The filter restricting to only timestamp columns ensures the component works correctly with appropriate data types for tracking updates.
jcortes
force-pushed
the
amazon-redshift-new-components
branch
from
f58e1fd to
973994d
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
73-78: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => { - return record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {}); - }); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (1)
81-85: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => - record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {})); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (38)
components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs(1 hunks)components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs(1 hunks)components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs(1 hunks)components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs(1 hunks)components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs(1 hunks)components/aws/actions/dynamodb-query/dynamodb-query.mjs(1 hunks)components/aws/actions/dynamodb-scan/dynamodb-scan.mjs(1 hunks)components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs(1 hunks)components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs(1 hunks)components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs(1 hunks)components/aws/actions/lambda-create-function/lambda-create-function.mjs(1 hunks)components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs(1 hunks)components/aws/actions/redshift-create-rows/redshift-create-rows.mjs(1 hunks)components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs(1 hunks)components/aws/actions/redshift-query-database/redshift-query-database.mjs(1 hunks)components/aws/actions/redshift-update-rows/redshift-update-rows.mjs(1 hunks)components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs(1 hunks)components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs(1 hunks)components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs(1 hunks)components/aws/actions/s3-upload-files/s3-upload-files.mjs(1 hunks)components/aws/actions/sns-send-message/sns-send-message.mjs(1 hunks)components/aws/actions/sqs-send-message/sqs-send-message.mjs(1 hunks)components/aws/aws.app.mjs(4 hunks)components/aws/common/utils.mjs(1 hunks)components/aws/package.json(2 hunks)components/aws/sources/common/redshift-polling.mjs(1 hunks)components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs(1 hunks)components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs(1 hunks)components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs(1 hunks)components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs(1 hunks)components/aws/sources/new-sns-messages/new-sns-messages.mjs(1 hunks)components/aws/sources/rds-new-event/rds-new-event.mjs(1 hunks)components/aws/sources/redshift-new-row/redshift-new-row.mjs(1 hunks)components/aws/sources/redshift-updated-row/redshift-updated-row.mjs(1 hunks)components/aws/sources/s3-deleted-file/s3-deleted-file.mjs(1 hunks)components/aws/sources/s3-new-event/s3-new-event.mjs(1 hunks)components/aws/sources/s3-new-file/s3-new-file.mjs(1 hunks)components/aws/sources/s3-restored-file/s3-restored-file.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (4)
- components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs
- components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs
- components/aws/sources/rds-new-event/rds-new-event.mjs
- components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs
🚧 Files skipped from review as they are similar to previous changes (32)
- components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs
- components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs
- components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs
- components/aws/actions/sns-send-message/sns-send-message.mjs
- components/aws/sources/new-sns-messages/new-sns-messages.mjs
- components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs
- components/aws/actions/lambda-create-function/lambda-create-function.mjs
- components/aws/sources/s3-restored-file/s3-restored-file.mjs
- components/aws/sources/s3-deleted-file/s3-deleted-file.mjs
- components/aws/sources/s3-new-event/s3-new-event.mjs
- components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs
- components/aws/actions/dynamodb-query/dynamodb-query.mjs
- components/aws/actions/sqs-send-message/sqs-send-message.mjs
- components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs
- components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs
- components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs
- components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs
- components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs
- components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs
- components/aws/actions/dynamodb-scan/dynamodb-scan.mjs
- components/aws/sources/s3-new-file/s3-new-file.mjs
- components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs
- components/aws/actions/s3-upload-files/s3-upload-files.mjs
- components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs
- components/aws/package.json
- components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs
- components/aws/actions/redshift-query-database/redshift-query-database.mjs
- components/aws/sources/common/redshift-polling.mjs
- components/aws/actions/redshift-update-rows/redshift-update-rows.mjs
- components/aws/actions/redshift-create-rows/redshift-create-rows.mjs
- components/aws/common/utils.mjs
- components/aws/aws.app.mjs
🧰 Additional context used
🧠 Learnings (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
🪛 Biome (1.9.4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs
[error] 75-75: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs
[error] 83-83: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: pnpm publish
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
- GitHub Check: Lint Code Base
🔇 Additional comments (5)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (3)
3-32: Component structure and props definition look good.The component properly extends the common module and has well-defined props with appropriate parameter passing for dynamic column loading.
46-51: Verify SQL injection protection for identifier names.The SQL query construction uses string concatenation for schema, table, and uniqueColumn names. Ensure these values are properly validated or escaped to prevent SQL injection attacks.
Consider adding identifier escaping or validation:
- let sql = `SELECT * FROM ${schema}.${table}`; + let sql = `SELECT * FROM "${schema}"."${table}"`; if (lastId) { - sql += ` WHERE ${uniqueColumn} > :lastId`; + sql += ` WHERE "${uniqueColumn}" > :lastId`; } - sql += ` ORDER BY ${uniqueColumn} DESC LIMIT 100`; + sql += ` ORDER BY "${uniqueColumn}" DESC LIMIT 100`;
53-60: Verify data type handling for lastId parameter.The conversion of
lastIdto string (line 57) might cause comparison issues if the unique column is a numeric type. Ensure the parameter conversion matches the column's data type.Consider preserving the original data type:
- value: String(lastId), + value: lastId,components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (2)
3-41: Component structure and timestamp column filtering look good.The component properly extends the common module and the timestampColumn prop correctly filters for timestamp types, which is appropriate for tracking row updates.
55-59: Verify SQL injection protection for identifier names.The SQL query construction uses string concatenation for schema, table, and timestampColumn names. Ensure these values are properly validated or escaped to prevent SQL injection attacks.
Consider adding identifier escaping or validation:
- let sql = `SELECT * FROM ${schema}.${table}`; + let sql = `SELECT * FROM "${schema}"."${table}"`; if (lastTimestamp) { - sql += ` WHERE ${timestampColumn} > :lastTimestamp`; + sql += ` WHERE "${timestampColumn}" > :lastTimestamp`; } - sql += ` ORDER BY ${timestampColumn} DESC LIMIT 100`; + sql += ` ORDER BY "${timestampColumn}" DESC LIMIT 100`;
jcortes
force-pushed
the
amazon-redshift-new-components
branch
2 times, most recently
from
f5c9e8f to
c1664ec
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
73-78: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => { - return record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {}); - }); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (2)
81-85: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => - record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {})); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });
88-94: Add error handling for timestamp parsing.The
new Date(record[timestampColumn]).getTime()could fail if the timestamp format is not parseable by JavaScript's Date constructor, potentially causing runtime errors.Consider adding error handling:
- const ts = new Date(record[timestampColumn]).getTime(); + const ts = Date.parse(record[timestampColumn]) || Date.now();Or verify the timestamp format first:
- const ts = new Date(record[timestampColumn]).getTime(); + const tsValue = record[timestampColumn]; + const ts = tsValue ? new Date(tsValue).getTime() : Date.now();
🧹 Nitpick comments (5)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
90-95: Improve efficiency of last ID extraction.The current approach maps all records and then reverses just to get the first element, which is inefficient.
- const [ - lastId, - ] = records.map((record) => record[uniqueColumn]).reverse(); + const lastId = records[records.length - 1][uniqueColumn];components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (1)
97-102: Improve efficiency of last timestamp extraction.Similar to the new-row component, the current approach is inefficient.
- const [ - newLastTimestamp, - ] = records.map((record) => record[timestampColumn]).reverse(); + const newLastTimestamp = records[records.length - 1][timestampColumn];components/aws/aws.app.mjs (3)
68-70: Add validation for missing dependencies.The early return when workgroupName is missing is good, but consider adding user-friendly messaging.
if (!workgroupName) { - return []; + return { + options: [], + error: "Please select a workgroup first" + }; }
277-295: Consider making the polling interval configurable.The hardcoded 3-second delay might not be optimal for all use cases. Some queries might complete faster, while others might need longer intervals.
- await new Promise((resolve) => setTimeout(resolve, 3000)); + await new Promise((resolve) => setTimeout(resolve, pollInterval || 3000));And add a parameter to the executeStatement method:
async executeStatement({ - region, workgroupName, database, sql, parameters, + region, workgroupName, database, sql, parameters, pollInterval = 3000, } = {}) {
285-290: Improve error handling with more context.The error handling correctly throws on failed/aborted states, but could provide more context about the failure.
if ( describeStatementResponse.Status === "FAILED" || describeStatementResponse.Status === "ABORTED" ) { - throw new Error(`Query failed: ${describeStatementResponse.Error}`); + throw new Error(`Query ${describeStatementResponse.Status.toLowerCase()}: ${describeStatementResponse.Error || 'Unknown error'}`); }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (38)
components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs(1 hunks)components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs(1 hunks)components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs(1 hunks)components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs(1 hunks)components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs(1 hunks)components/aws/actions/dynamodb-query/dynamodb-query.mjs(1 hunks)components/aws/actions/dynamodb-scan/dynamodb-scan.mjs(1 hunks)components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs(1 hunks)components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs(1 hunks)components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs(1 hunks)components/aws/actions/lambda-create-function/lambda-create-function.mjs(1 hunks)components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs(1 hunks)components/aws/actions/redshift-create-rows/redshift-create-rows.mjs(1 hunks)components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs(1 hunks)components/aws/actions/redshift-query-database/redshift-query-database.mjs(1 hunks)components/aws/actions/redshift-update-rows/redshift-update-rows.mjs(1 hunks)components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs(1 hunks)components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs(1 hunks)components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs(1 hunks)components/aws/actions/s3-upload-files/s3-upload-files.mjs(1 hunks)components/aws/actions/sns-send-message/sns-send-message.mjs(1 hunks)components/aws/actions/sqs-send-message/sqs-send-message.mjs(1 hunks)components/aws/aws.app.mjs(4 hunks)components/aws/common/utils.mjs(1 hunks)components/aws/package.json(2 hunks)components/aws/sources/common/redshift-polling.mjs(1 hunks)components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs(1 hunks)components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs(1 hunks)components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs(1 hunks)components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs(1 hunks)components/aws/sources/new-sns-messages/new-sns-messages.mjs(1 hunks)components/aws/sources/rds-new-event/rds-new-event.mjs(1 hunks)components/aws/sources/redshift-new-row/redshift-new-row.mjs(1 hunks)components/aws/sources/redshift-updated-row/redshift-updated-row.mjs(1 hunks)components/aws/sources/s3-deleted-file/s3-deleted-file.mjs(1 hunks)components/aws/sources/s3-new-event/s3-new-event.mjs(1 hunks)components/aws/sources/s3-new-file/s3-new-file.mjs(1 hunks)components/aws/sources/s3-restored-file/s3-restored-file.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (3)
- components/aws/sources/rds-new-event/rds-new-event.mjs
- components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs
- components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs
🚧 Files skipped from review as they are similar to previous changes (32)
- components/aws/actions/sns-send-message/sns-send-message.mjs
- components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs
- components/aws/sources/s3-restored-file/s3-restored-file.mjs
- components/aws/package.json
- components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs
- components/aws/actions/sqs-send-message/sqs-send-message.mjs
- components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs
- components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs
- components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs
- components/aws/sources/s3-deleted-file/s3-deleted-file.mjs
- components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs
- components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs
- components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs
- components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs
- components/aws/actions/dynamodb-scan/dynamodb-scan.mjs
- components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs
- components/aws/actions/dynamodb-query/dynamodb-query.mjs
- components/aws/sources/new-sns-messages/new-sns-messages.mjs
- components/aws/sources/s3-new-event/s3-new-event.mjs
- components/aws/actions/s3-upload-files/s3-upload-files.mjs
- components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs
- components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs
- components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs
- components/aws/sources/s3-new-file/s3-new-file.mjs
- components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs
- components/aws/actions/redshift-create-rows/redshift-create-rows.mjs
- components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs
- components/aws/sources/common/redshift-polling.mjs
- components/aws/actions/redshift-update-rows/redshift-update-rows.mjs
- components/aws/actions/redshift-query-database/redshift-query-database.mjs
- components/aws/actions/lambda-create-function/lambda-create-function.mjs
- components/aws/common/utils.mjs
🧰 Additional context used
🧠 Learnings (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
🪛 Biome (1.9.4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs
[error] 75-75: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs
[error] 83-83: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
- GitHub Check: Lint Code Base
🔇 Additional comments (4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
46-60: Validate schema and table names before interpolationI didn’t find evidence of upstream validation for the
schemaandtablevariables—please verify that these values are sanitized or constrained (e.g. via a whitelist or strict regex) before being used in the query string to prevent SQL injection.
- Confirm in your common module or calling code that both identifiers are properly validated
- Consider enforcing a naming policy (allowed characters, maximum length) or pulling values from a known-safe list
components/aws/aws.app.mjs (3)
9-22: LGTM: Proper AWS SDK imports for Redshift functionality.The imports are well-structured and include all necessary commands for Redshift Serverless and Data API operations.
50-58: LGTM: Well-structured workgroup selection prop.The prop definition correctly uses async options to load workgroups dynamically based on the selected region.
133-133: Excellent use of configurable filter function.The filter parameter in the uniqueColumn prop allows for flexible column filtering (e.g., timestamp columns only), which is used effectively in the updated-row component.
|
/approve |
jcortes
force-pushed
the
amazon-redshift-new-components
branch
from
c1664ec to
a806be7
Compare
|
/approve |
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
73-78: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => { - return record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {}); - }); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (2)
81-85: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => - record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {})); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });
88-94: Add error handling for timestamp parsing.The
new Date(record[timestampColumn]).getTime()could fail if the timestamp format is not parseable by JavaScript's Date constructor, potentially causing runtime errors.Consider adding error handling:
- const ts = new Date(record[timestampColumn]).getTime(); + const ts = Date.parse(record[timestampColumn]) || Date.now();Or verify the timestamp format first:
- const ts = new Date(record[timestampColumn]).getTime(); + const tsValue = record[timestampColumn]; + const ts = tsValue ? new Date(tsValue).getTime() : Date.now();components/aws/aws.app.mjs (1)
277-295: Add timeout protection for long-running queries.The polling loop could run indefinitely if a query gets stuck in a running state. Consider adding a timeout mechanism.
+ const maxAttempts = 100; // ~5 minutes with 3-second intervals + let attempts = 0; + do { await new Promise((resolve) => setTimeout(resolve, 3000)); + attempts++; + if (attempts > maxAttempts) { + throw new Error(`Query timed out after ${maxAttempts * 3} seconds`); + } describeStatementResponse = await client.send( new DescribeStatementCommand({ Id: executeStatementResponse.Id, }), );
🧹 Nitpick comments (1)
components/aws/aws.app.mjs (1)
285-290: Improve error handling for failed queries.The error handling correctly catches failed and aborted queries, but the error message could be more informative.
- throw new Error(`Query failed: ${describeStatementResponse.Error}`); + throw new Error(`Query ${describeStatementResponse.Status.toLowerCase()}: ${describeStatementResponse.Error || 'Unknown error'}`);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (38)
components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs(1 hunks)components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs(1 hunks)components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs(1 hunks)components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs(1 hunks)components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs(1 hunks)components/aws/actions/dynamodb-query/dynamodb-query.mjs(1 hunks)components/aws/actions/dynamodb-scan/dynamodb-scan.mjs(1 hunks)components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs(1 hunks)components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs(1 hunks)components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs(1 hunks)components/aws/actions/lambda-create-function/lambda-create-function.mjs(1 hunks)components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs(1 hunks)components/aws/actions/redshift-create-rows/redshift-create-rows.mjs(1 hunks)components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs(1 hunks)components/aws/actions/redshift-query-database/redshift-query-database.mjs(1 hunks)components/aws/actions/redshift-update-rows/redshift-update-rows.mjs(1 hunks)components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs(1 hunks)components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs(1 hunks)components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs(1 hunks)components/aws/actions/s3-upload-files/s3-upload-files.mjs(1 hunks)components/aws/actions/sns-send-message/sns-send-message.mjs(1 hunks)components/aws/actions/sqs-send-message/sqs-send-message.mjs(1 hunks)components/aws/aws.app.mjs(4 hunks)components/aws/common/utils.mjs(1 hunks)components/aws/package.json(2 hunks)components/aws/sources/common/redshift-polling.mjs(1 hunks)components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs(1 hunks)components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs(1 hunks)components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs(1 hunks)components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs(1 hunks)components/aws/sources/new-sns-messages/new-sns-messages.mjs(1 hunks)components/aws/sources/rds-new-event/rds-new-event.mjs(1 hunks)components/aws/sources/redshift-new-row/redshift-new-row.mjs(1 hunks)components/aws/sources/redshift-updated-row/redshift-updated-row.mjs(1 hunks)components/aws/sources/s3-deleted-file/s3-deleted-file.mjs(1 hunks)components/aws/sources/s3-new-event/s3-new-event.mjs(1 hunks)components/aws/sources/s3-new-file/s3-new-file.mjs(1 hunks)components/aws/sources/s3-restored-file/s3-restored-file.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (2)
- components/aws/sources/rds-new-event/rds-new-event.mjs
- components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs
🚧 Files skipped from review as they are similar to previous changes (33)
- components/aws/sources/new-sns-messages/new-sns-messages.mjs
- components/aws/sources/s3-deleted-file/s3-deleted-file.mjs
- components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs
- components/aws/sources/s3-restored-file/s3-restored-file.mjs
- components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs
- components/aws/actions/lambda-create-function/lambda-create-function.mjs
- components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs
- components/aws/package.json
- components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs
- components/aws/actions/sns-send-message/sns-send-message.mjs
- components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs
- components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs
- components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs
- components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs
- components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs
- components/aws/actions/dynamodb-query/dynamodb-query.mjs
- components/aws/actions/sqs-send-message/sqs-send-message.mjs
- components/aws/sources/s3-new-event/s3-new-event.mjs
- components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs
- components/aws/sources/s3-new-file/s3-new-file.mjs
- components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs
- components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs
- components/aws/actions/dynamodb-scan/dynamodb-scan.mjs
- components/aws/actions/s3-upload-files/s3-upload-files.mjs
- components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs
- components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs
- components/aws/actions/redshift-query-database/redshift-query-database.mjs
- components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs
- components/aws/actions/redshift-update-rows/redshift-update-rows.mjs
- components/aws/sources/common/redshift-polling.mjs
- components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs
- components/aws/common/utils.mjs
- components/aws/actions/redshift-create-rows/redshift-create-rows.mjs
🧰 Additional context used
🧠 Learnings (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (4)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#17538
File: components/aircall/sources/new-sms/new-sms.mjs:19-25
Timestamp: 2025-07-09T18:07:12.388Z
Learning: In Aircall API webhook payloads, the `created_at` field is returned as an ISO 8601 string format (e.g., "2020-02-18T20:52:22.000Z"), not as milliseconds since epoch. For Pipedream components, this needs to be converted to milliseconds using `Date.parse()` before assigning to the `ts` field in `generateMeta()`.
🪛 Biome (1.9.4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs
[error] 75-75: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs
[error] 83-83: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: pnpm publish
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
- GitHub Check: Lint Code Base
🔇 Additional comments (5)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
1-32: LGTM! Clean component structure and proper parameterized SQL usage.The component extends the common polling module appropriately and uses parameterized SQL queries to prevent injection attacks. The prop definitions correctly reference the app's dynamic option loaders.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (1)
19-40: LGTM! Good timestamp column filtering.The timestamp column prop correctly filters columns by
typeName === "timestamp", ensuring only appropriate columns are available for tracking row updates.components/aws/aws.app.mjs (3)
9-22: LGTM! Comprehensive Redshift SDK imports.The imports include all necessary Redshift Serverless and Data API clients and commands required for the new functionality.
50-149: LGTM! Well-structured prop definitions with dynamic options.The prop definitions provide comprehensive coverage for Redshift resources with proper async option loaders that dynamically fetch workgroups, databases, schemas, tables, and columns. The filtering logic for the uniqueColumn prop is particularly well-designed.
223-260: LGTM! Clean and consistent method implementations.The new methods follow a consistent pattern using the AWS SDK clients with proper parameter handling. The describeTable method correctly structures the command parameters.
jcortes
force-pushed
the
amazon-redshift-new-components
branch
from
a806be7 to
a4a775e
Compare
|
/approve |
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (5)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (2)
73-78: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => { - return record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {}); - }); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });
80-95: Simplify record processing and lastId extraction logic.The current logic reverses the records array twice and has unnecessary complexity. Since we're ordering by
uniqueColumn ASC, we can directly process records and extract thelastIdfrom the last record.- records.reverse().forEach((record) => { + records.forEach((record) => { const id = record[uniqueColumn]; const ts = Date.now(); this.$emit(record, { id, ts, summary: `New Row Added ${id}`, }); }); if (records?.length > 0) { - const [ - lastId, - ] = records.map((record) => record[uniqueColumn]).reverse(); + const lastId = records[records.length - 1][uniqueColumn]; this.db.set("lastId", lastId); }components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (2)
81-85: Fix performance issue with spread operator in reduce accumulator.The spread operator in the reduce accumulator causes O(n²) time complexity, which can be problematic for large datasets.
Replace the spread operator with a more efficient approach:
- const records = response.Records.map((record) => - record.reduce((row, field, index) => ({ - ...row, - [columnNames[index]]: getValue(field), - }), {})); + const records = response.Records.map((record) => { + const row = {}; + record.forEach((field, index) => { + row[columnNames[index]] = getValue(field); + }); + return row; + });
88-94: Add error handling for timestamp parsing.The
new Date(record[timestampColumn]).getTime()could fail if the timestamp format is not parseable by JavaScript's Date constructor, potentially causing runtime errors.Consider adding error handling:
- const ts = new Date(record[timestampColumn]).getTime(); + const ts = Date.parse(record[timestampColumn]) || Date.now();Or verify the timestamp format first:
- const ts = new Date(record[timestampColumn]).getTime(); + const tsValue = record[timestampColumn]; + const ts = tsValue ? new Date(tsValue).getTime() : Date.now();components/aws/aws.app.mjs (1)
277-295: Add timeout protection for long-running queries.The polling loop could run indefinitely if a query gets stuck in a running state. Consider adding a timeout mechanism.
+ const maxAttempts = 100; // ~5 minutes with 3-second intervals + let attempts = 0; + do { await new Promise((resolve) => setTimeout(resolve, 3000)); + attempts++; + if (attempts > maxAttempts) { + throw new Error(`Query timed out after ${maxAttempts * 3} seconds`); + } describeStatementResponse = await client.send( new DescribeStatementCommand({ Id: executeStatementResponse.Id, }), );
🧹 Nitpick comments (2)
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (2)
55-59: Consider changing ORDER BY to ASC for consistency.Similar to the new-row component, ordering by
timestampColumn ASCwould be more intuitive and consistent with the logic of processing records from oldest to newest.- sql += ` ORDER BY ${timestampColumn} DESC LIMIT 100`; + sql += ` ORDER BY ${timestampColumn} ASC LIMIT 100`;
97-102: Simplify lastTimestamp extraction logic.The current logic reverses the records array twice. If we order by
timestampColumn ASC, we can directly extract the last timestamp from the last record.if (records?.length > 0) { - const [ - newLastTimestamp, - ] = records.map((record) => record[timestampColumn]).reverse(); + const newLastTimestamp = records[records.length - 1][timestampColumn]; this.db.set("lastTimestamp", newLastTimestamp); }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (38)
components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs(1 hunks)components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs(1 hunks)components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs(1 hunks)components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs(1 hunks)components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs(1 hunks)components/aws/actions/dynamodb-query/dynamodb-query.mjs(1 hunks)components/aws/actions/dynamodb-scan/dynamodb-scan.mjs(1 hunks)components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs(1 hunks)components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs(1 hunks)components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs(1 hunks)components/aws/actions/lambda-create-function/lambda-create-function.mjs(1 hunks)components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs(1 hunks)components/aws/actions/redshift-create-rows/redshift-create-rows.mjs(1 hunks)components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs(1 hunks)components/aws/actions/redshift-query-database/redshift-query-database.mjs(1 hunks)components/aws/actions/redshift-update-rows/redshift-update-rows.mjs(1 hunks)components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs(1 hunks)components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs(1 hunks)components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs(1 hunks)components/aws/actions/s3-upload-files/s3-upload-files.mjs(1 hunks)components/aws/actions/sns-send-message/sns-send-message.mjs(1 hunks)components/aws/actions/sqs-send-message/sqs-send-message.mjs(1 hunks)components/aws/aws.app.mjs(4 hunks)components/aws/common/utils.mjs(1 hunks)components/aws/package.json(2 hunks)components/aws/sources/common/redshift-polling.mjs(1 hunks)components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs(1 hunks)components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs(1 hunks)components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs(1 hunks)components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs(1 hunks)components/aws/sources/new-sns-messages/new-sns-messages.mjs(1 hunks)components/aws/sources/rds-new-event/rds-new-event.mjs(1 hunks)components/aws/sources/redshift-new-row/redshift-new-row.mjs(1 hunks)components/aws/sources/redshift-updated-row/redshift-updated-row.mjs(1 hunks)components/aws/sources/s3-deleted-file/s3-deleted-file.mjs(1 hunks)components/aws/sources/s3-new-event/s3-new-event.mjs(1 hunks)components/aws/sources/s3-new-file/s3-new-file.mjs(1 hunks)components/aws/sources/s3-restored-file/s3-restored-file.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (3)
- components/aws/sources/new-records-returned-by-cloudwatch-logs-insights-query/new-records-returned-by-cloudwatch-logs-insights-query.mjs
- components/aws/actions/eventbridge-send-event/eventbridge-send-event.mjs
- components/aws/sources/rds-new-event/rds-new-event.mjs
🚧 Files skipped from review as they are similar to previous changes (32)
- components/aws/sources/s3-deleted-file/s3-deleted-file.mjs
- components/aws/sources/new-sns-messages/new-sns-messages.mjs
- components/aws/sources/s3-new-event/s3-new-event.mjs
- components/aws/actions/s3-generate-presigned-url/s3-generate-presigned-url.mjs
- components/aws/actions/lambda-create-function/lambda-create-function.mjs
- components/aws/actions/dynamodb-create-table/dynamodb-create-table.mjs
- components/aws/actions/dynamodb-update-item/dynamodb-update-item.mjs
- components/aws/sources/new-scheduled-tasks/new-scheduled-tasks.mjs
- components/aws/actions/dynamodb-scan/dynamodb-scan.mjs
- components/aws/actions/sqs-send-message/sqs-send-message.mjs
- components/aws/actions/dynamodb-execute-statement/dynamodb-execute-statement.mjs
- components/aws/actions/lambda-invoke-function/lambda-invoke-function.mjs
- components/aws/actions/sns-send-message/sns-send-message.mjs
- components/aws/sources/s3-new-file/s3-new-file.mjs
- components/aws/sources/s3-restored-file/s3-restored-file.mjs
- components/aws/actions/dynamodb-get-item/dynamodb-get-item.mjs
- components/aws/package.json
- components/aws/actions/cloudwatch-logs-put-log-event/cloudwatch-logs-put-log-event.mjs
- components/aws/actions/s3-upload-base64-as-file/s3-upload-base64-as-file.mjs
- components/aws/actions/s3-download-file-to-tmp/s3-download-file-to-tmp.mjs
- components/aws/sources/new-emails-sent-to-ses-catch-all-domain/new-emails-sent-to-ses-catch-all-domain.mjs
- components/aws/actions/dynamodb-update-table/dynamodb-update-table.mjs
- components/aws/actions/dynamodb-query/dynamodb-query.mjs
- components/aws/actions/dynamodb-put-item/dynamodb-put-item.mjs
- components/aws/actions/s3-upload-files/s3-upload-files.mjs
- components/aws/sources/new-dynamodb-stream-event/new-dynamodb-stream-event.mjs
- components/aws/actions/redshift-update-rows/redshift-update-rows.mjs
- components/aws/actions/redshift-query-database/redshift-query-database.mjs
- components/aws/sources/common/redshift-polling.mjs
- components/aws/actions/redshift-delete-rows/redshift-delete-rows.mjs
- components/aws/actions/redshift-create-rows/redshift-create-rows.mjs
- components/aws/common/utils.mjs
🧰 Additional context used
🧠 Learnings (2)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (3)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs (4)
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#15376
File: components/monday/sources/name-updated/name-updated.mjs:6-6
Timestamp: 2025-01-23T03:55:15.166Z
Learning: Source names in Monday.com components don't need to start with "New" if they emit events for updated items (e.g., "Name Updated", "Column Value Updated") rather than new items. This follows the component guidelines exception where the "New" prefix is only required when emits are limited to new items.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#14265
File: components/the_magic_drip/sources/common.mjs:35-43
Timestamp: 2024-10-10T19:18:27.998Z
Learning: In `components/the_magic_drip/sources/common.mjs`, when processing items in `getAndProcessData`, `savedIds` is intentionally updated with IDs of both emitted and non-emitted items to avoid emitting retroactive events upon first deployment and ensure only new events are emitted as they occur.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#16954
File: components/salesloft/salesloft.app.mjs:14-23
Timestamp: 2025-06-04T17:52:05.780Z
Learning: In the Salesloft API integration (components/salesloft/salesloft.app.mjs), the _makeRequest method returns response.data which directly contains arrays for list endpoints like listPeople, listCadences, listUsers, and listAccounts. The propDefinitions correctly call .map() directly on these responses without needing to destructure a nested data property.
Learnt from: GTFalcao
PR: PipedreamHQ/pipedream#17538
File: components/aircall/sources/new-sms/new-sms.mjs:19-25
Timestamp: 2025-07-09T18:07:12.426Z
Learning: In Aircall API webhook payloads, the `created_at` field is returned as an ISO 8601 string format (e.g., "2020-02-18T20:52:22.000Z"), not as milliseconds since epoch. For Pipedream components, this needs to be converted to milliseconds using `Date.parse()` before assigning to the `ts` field in `generateMeta()`.
🪛 Biome (1.9.4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs
[error] 75-75: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
components/aws/sources/redshift-updated-row/redshift-updated-row.mjs
[error] 83-83: Avoid the use of spread (...) syntax on accumulators.
Spread syntax should be avoided on accumulators (like those in .reduce) because it causes a time complexity of O(n^2).
Consider methods such as .splice or .push instead.
(lint/performance/noAccumulatingSpread)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: Lint Code Base
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
- GitHub Check: pnpm publish
🔇 Additional comments (4)
components/aws/sources/redshift-new-row/redshift-new-row.mjs (1)
46-51: No changes needed: ORDER BY clause already present
The SQL query in components/aws/sources/redshift-new-row/redshift-new-row.mjs already includesORDER BY ${uniqueColumn} DESCensuring a predictable sort on the unique column. The suggestion to add an ORDER BY is unnecessary.
Likely an incorrect or invalid review comment.
components/aws/aws.app.mjs (3)
9-22: LGTM! Clean import structure for Redshift SDK clients.The imports are well-organized and include all necessary Redshift Serverless and Data API commands for the new functionality.
50-148: LGTM! Well-structured prop definitions with proper dependency handling.The new prop definitions for Redshift resources are well-implemented with:
- Proper async option loaders
- Correct dependency chaining (workgroup → database → schema → table)
- Appropriate filtering for unique columns
- Good error handling with empty array returns when dependencies are missing
223-260: LGTM! Clean and consistent method implementations.The new methods for listing Redshift resources and describing tables follow a consistent pattern and properly use the AWS SDK clients.
WHY
Resolves #17058
Summary by CodeRabbit
New Features
Chores