Skip to content

Fix #14074 - Paginate not working #14080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 27, 2024
Merged

Fix #14074 - Paginate not working #14080

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7910d70
Fix #14074 - Paginate not working
Sep 25, 2024
e31def9
Increment version
Sep 25, 2024
92c44e4
Add missing comma
LucBerge Sep 25, 2024
d163ec7
Rename `lastPage` variable to `hasMore`
Sep 26, 2024
a357346
Merge branch 'master' into master
vunguyenhung Sep 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion components/nocodb/actions/add-record/add-record.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ export default {
key: "nocodb-add-record",
name: "Add Record",
description: "This action adds a record in a table. [See the documentation](https://data-apis-v2.nocodb.com/#tag/Table-Records/operation/db-data-table-row-create)",
version: "0.0.3",
version: "0.0.4",
type: "action",
props: {
...common.props,
Expand Down
2 changes: 1 addition & 1 deletion components/nocodb/actions/delete-record/delete-record.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ export default {
key: "nocodb-delete-record",
name: "Delete Record",
description: "This action deletes a row in a table. [See the documentation](https://data-apis-v2.nocodb.com/#tag/Table-Records/operation/db-data-table-row-delete)",
version: "0.0.3",
version: "0.0.4",
type: "action",
props: {
...common.props,
Expand Down
2 changes: 1 addition & 1 deletion components/nocodb/actions/get-record/get-record.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ export default {
key: "nocodb-get-record",
name: "Get Record (from row number)",
description: "This action gets a row by row Id. [See the documentation](https://data-apis-v2.nocodb.com/#tag/Table-Records/operation/db-data-table-row-read)",
version: "0.0.3",
version: "0.0.4",
type: "action",
props: {
...common.props,
Expand Down
2 changes: 1 addition & 1 deletion components/nocodb/actions/list-records-matching-criteria/list-records-matching-criteria.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ export default {
key: "nocodb-list-records-matching-criteria",
name: "List Records in Table Matching Criteria",
description: "This action lists all rows in a table. [See the documentation](https://data-apis-v2.nocodb.com/#tag/Table-Records/operation/db-data-table-row-list)",
version: "0.0.4",
version: "0.0.5",
type: "action",
props: {
...common.props,
Expand Down
2 changes: 1 addition & 1 deletion components/nocodb/actions/update-record/update-record.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ export default {
key: "nocodb-update-record",
name: "Update Record",
description: "This action updates a record in a table. [See the documentation](https://data-apis-v2.nocodb.com/#tag/Table-Records/operation/db-data-table-row-update)",
version: "0.0.3",
version: "0.0.4",
type: "action",
props: {
...common.props,
Expand Down
12 changes: 7 additions & 5 deletions components/nocodb/nocodb.app.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,24 +119,26 @@ export default {
async *paginate({
fn, args = {}, max,
}) {
let lastPage, count = 0;
let hasMore = false;
let count = 0;
args.params = {
...args.params,
page: 1,
offset: 0,
limit: 1000,
};
do {
const {
list, pageInfo,
} = await fn(args);
for (const item of list) {
args.params.offset++;
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix the offset increment logic.

The current implementation increments the offset for each item yielded. This is incorrect for pagination purposes, as it will cause the next API call to skip items.

Replace the line:

args.params.offset++;

with:

// This line should be moved outside the for loop
args.params.offset += list.length;

This change ensures that the offset is incremented by the number of items retrieved in each API call, maintaining correct pagination.

Copy link
Contributor Author

@LucBerge LucBerge Sep 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai Performing args.params.offset++ in the for loop or args.params.offset += list.length outside the loop is the exact same thing.

It is even better to do args.params.offset++ because in case the if (max && ++count === max) condition is met, offset is incremented by the correct number of rows returned.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai Performing args.params.offset++ in the for loop or args.params.offset += list.length outside the loop is the exact same thing.

It is even better to do args.params.offset++ because in case the if (max && ++count === max) condition is met, offset is incremented by the correct number of rows returned.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LucBerge, you're right. Incrementing args.params.offset within the loop ensures the offset accurately reflects the number of items processed, especially when a max limit is specified. Thank you for the clarification. I'll note this for future reviews.

✏️ Learnings added 
Learnt from: LucBerge
PR: PipedreamHQ/pipedream#14080
File: components/nocodb/nocodb.app.mjs:133-133
Timestamp: 2024-09-25T16:13:11.336Z
Learning: When implementing pagination with an offset, incrementing `args.params.offset` within the loop ensures correct tracking of the offset, particularly when a maximum count limit (`max`) is used.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

yield item;
if (max && ++count === max) {
return;
}
}
args.params.page++;
lastPage = !pageInfo.isLastPage;
} while (lastPage);
hasMore = !pageInfo.isLastPage;
} while (hasMore);
},
listWorkspaces(opts = {}) {
return this._makeRequest({
Expand Down
2 changes: 1 addition & 1 deletion components/nocodb/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@pipedream/nocodb",
"version": "0.0.6",
"version": "0.0.7",
"description": "Pipedream Nocodb Components",
"main": "nocodb.app.mjs",
"keywords": [
Expand Down
2 changes: 1 addition & 1 deletion components/nocodb/sources/new-record/new-record.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ export default {
name: "New Record in Table",
key: "nocodb-new-record",
description: "Emit new event for each new record in table. [See the documentation](https://data-apis-v2.nocodb.com/#tag/Table-Records/operation/db-data-table-row-list)",
version: "0.0.4",
version: "0.0.5",
dedupe: "unique",
props: {
...common.props,
Expand Down
2 changes: 1 addition & 1 deletion components/nocodb/sources/updated-record/updated-record.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ export default {
name: "New Update in Table",
key: "nocodb-updated-record",
description: "Emit new event for each update in table. [See the documentation](https://data-apis-v2.nocodb.com/#tag/Table-Records/operation/db-data-table-row-list)",
version: "0.0.4",
version: "0.0.5",
dedupe: "unique",
props: {
...common.props,
Expand Down
Loading