Pub/Sub pull limit of 5000 messages #2097
Unanswered
WilliamWood01
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hello, I have a question related to pub/sub, and I am honestly not sure if the client library is causing the issue I am facing or not.
Currently I am working on a project that publishes any received request contents to a pub sub topic and subsequent subscriber, in order to pull and batch process all of the messages at once every single day. This is as a gap-filling process is carried out over the requests received for the entire day. I am aware that perhaps this is not the most well designed solution, and there are probably better ways to achieve this, if so I am open to suggestions, but that is unrelated to the issue that I am facing.
To get all of these messages into my application, I am using a loop with repeated runs of v1.SubscriberClient.pull({subscription: subscriptionName, maxMessages: 1000}) to try and pull and flush every single message in the subscriber, repeating until 0 messages are returned three pulls in a row, or until a limit of pulls is reached.
While this seems to work with small amounts of messages, when testing with large amounts of unacked messages it seems to cap out at 5000 messages returned:


Case 1:
Case 2:
I was hoping someone here could help identify/resolve this issue, ideally it would be able to pull ALL messages in the subscriber for the batch processing method I mentioned earlier. CoPilot/GPT seem to think this is a limit placed within the client library preventing the delivery of more than 5000 messages, but I could not find any resources to prove this, so the issue may also be on the infrastructure itself?
Any ideas or solutions for this would be greatly appreciated, if more details are required please let me know
Beta Was this translation helpful? Give feedback.
All reactions