Skip to content

Gmail - increase fetch limit #17167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 17, 2025
Merged

Gmail - increase fetch limit #17167

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fe33f3c
remove get messages function that uses promise all
andrewjschuang Jun 17, 2025
43cedd8
increase max limit
andrewjschuang Jun 16, 2025
3cfec1d
bump versions
andrewjschuang Jun 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions components/gmail/actions/find-email/find-email.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,10 @@ export default {
maxResults: this.maxResults,
});
const messageIds = messages.map(({ id }) => id);
let messagesToEmit = await this.gmail.getMessages(messageIds);
const messagesToEmit = [];
for await (const message of this.gmail.getAllMessages(messageIds)) {
messagesToEmit.push(message);

for await (const message of messagesToEmit) {
let newPayload = "";

const messageIdHeader = message.payload?.headers?.find(
Expand Down
2 changes: 2 additions & 0 deletions components/gmail/common/constants.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,14 @@ const BODY_TYPES = {
};
const HISTORICAL_EVENTS = 10;
const DEFAULT_LIMIT = 100;
const MAX_LIMIT = 500;
const INBOX_LABEL_ID = "INBOX";

export default {
USER_ID,
BODY_TYPES,
HISTORICAL_EVENTS,
DEFAULT_LIMIT,
MAX_LIMIT,
INBOX_LABEL_ID,
};
6 changes: 0 additions & 6 deletions components/gmail/gmail.app.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,12 +410,6 @@ export default {
const { value: subject } = message.payload.headers.find(({ name }) => name === "Subject");
return subject;
},
async getMessages(ids = []) {
const promises = ids.map((id) => this.getMessage({
id,
}));
return Promise.all(promises);
},
async *getAllMessages(ids = []) {
for (const id of ids) {
const message = await this.getMessage({
Expand Down
9 changes: 7 additions & 2 deletions components/gmail/sources/common/polling-history.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,13 @@ export default {
index === self.findIndex((m) => m.id === message.id));

const messageIds = uniqueMessages.map(({ id }) => id);
const messagesWithHistoryId = await this.gmail.getMessages(messageIds);
const messages = [];
for await (const message of this.gmail.getAllMessages(messageIds)) {
messages.push(message);
}

const sortedMessages =
Array.from(messagesWithHistoryId)
Array.from(messages)
.sort((a, b) => (Number(b.historyId) - Number(a.historyId)));

const { historyId } = await this.gmail.getMessage({
Expand All @@ -74,6 +78,7 @@ export default {
const opts = {
startHistoryId: String(startHistoryId),
historyTypes: this.getHistoryTypes(),
maxResults: constants.MAX_LIMIT,
};
Comment on lines +81 to 82
Copy link
Contributor

@coderabbitai coderabbitai bot Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Pagination still ignored when raising the cap

Raising maxResults to constants.MAX_LIMIT (500) is helpful, but the loop doesn’t handle nextPageToken, so histories beyond the first page are silently skipped. Consider iterating until nextPageToken is empty to ensure completeness.

🤖 Prompt for AI Agents 
In components/gmail/sources/common/polling-history.mjs around lines 81 to 82,
the code sets maxResults to constants.MAX_LIMIT but does not handle pagination
with nextPageToken, causing only the first page of histories to be processed.
Modify the code to implement a loop that continues fetching and processing
history pages until nextPageToken is empty, ensuring all history entries are
retrieved completely.


const length = this.labels?.length > 0
Expand Down
5 changes: 4 additions & 1 deletion components/gmail/sources/common/polling-messages.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,10 @@ export default {
}
},
async processHistoricalEvents(messageIds) {
let messages = await this.gmail.getMessages(messageIds);
let messages = [];
for await (const message of this.gmail.getAllMessages(messageIds)) {
messages.push(message);
}
messages = messages.sort((a, b) => (a.internalDate - b.internalDate));
this.setLastDate(messages[messages.length - 1].internalDate);
messages.forEach((message) => this.emitEvent(message));
Expand Down
Loading