Some messages are not emitted to consumer for processing #2069
Description
Please make sure you have searched for information in the following guides.
- Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
- Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
- Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
- Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
- Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
- Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
- Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples
A screenshot that you have tested with "Try this API".
Not available
Link to the code that reproduces this issue. A link to a public Github Repository or gist with a minimal reproduction.
https://gist.github.com/nguyen-vo/200e8800423f55ddb4baab22010c4ba8
A step-by-step description of how to reproduce the issue, based on the linked reproduction.
Initiate 1 PubSub client for one Subscription instance
const client = new PubSub({
projectId: <PROJECT_ID>
apiEndpoint:
'https://europe-west1-pubsub.googleapis.com',
})
const subscription = client.subscription(<subscription_name>)
subscription.on('message', (message) => {
message.modAck(2*60*1000) // extends ack deadline by 2 minutes
// process message
message.ack()
})A clear and concise description of what the bug is, and what you expected to happen.
Environment details
OS:
Node.js version: v18.14.1
npm version: 9.8.1
@google-cloud/pubsub version: ^3.6.0
Bug
- Messages are pulled but some of them are not dispatch to be processed
Expected:
- Identify the issue. Is this known?
- Remedy for the issue e,g: update to specific version, to ensure all messages are dispatched to downstream functions
A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **
Recently we noticed that our Cloud Run service are not processing some messages in ordered. The service is set with CPU always on, CPU is stable at 75% or less, and memory usage is around 40% or less.
The service process heavy traffic in a short period of 2 hours. This is when we noticed some message are not processed in order for one subscription. The issue is intermittent. The publisher is confirmed to published messages to a single PubSub API endpoint europe-west3-pubsub.googleapis.com to be exact.
We have opened a case with GCP support and they confirmed that the message are received in order by PubSub server. However, based on the log the support engineer shared, the client pull messages but not dispatched them causing the message to be redeliver 10 minutes after.
After reviewing the log we identified that the issue is with the PubSub client as every event we receive we first extend the ack deadline by 2 minutes. The missing messages were first pull then and redeliver after 10 minutes without us extending the ack deadline. See the example below
| message_id | ack_fp | EST | operation | modAckDeadline_s |
|---|---|---|---|---|
| 15493210815242071 | null | 2025-07-16 04:10:28 28.296694 | Publish | null |
| 15493210815242071 | 8752572947079043233 | 2025-07-16 04:10:28 28.306213 | StreamingPullMessages | null |
| 15493210815242071 | 8752572947079043233 | 2025-07-16 04:10:28 28.341966 | ModifyAckDeadline | 10 |
| 15493210815242071 | 8752572947079043233 | 2025-07-16 04:10:28 28.344372 | ModifyAckDeadline | 120 |
| 15014881668136367 | null | 2025-07-16 04:10:28 28.410226 | Publish | null |
| 15014881668136367 | 16380187916779358794 | 2025-07-16 04:10:28 28.416644 | StreamingPullMessages | null |
| 15014881668136367 | 15978873186848138515 | 2025-07-16 04:20:30 30.808335 | StreamingPullMessages | null (Pull again after 10 minutes instead of modifying the ack deadline) |
| 15014881668136367 | 15978873186848138515 | 2025-07-16 04:20:30 30.912471 | ModifyAckDeadline | 10 |
| 15014881668136367 | 15978873186848138515 | 2025-07-16 04:20:30 30.913473 | ModifyAckDeadline | 120 |