Question in regards to Slow Consumer: max pending issue and Dropping messages due to excessive stream ingest rate...IPQ Len limit reached errors

[Kardee12]

Hello!

I'm currently looking into using NATS Jetstream. Could someone clarify if there's an internal buffer used when calling Fetch(), or if you have any suggestions on how to solve our problem?

We have a setup with 1 stream and 1 durable consumer, and have noticed critical issues when we start dealing with ~60k messages/sec. Primarily, we face Ingest rate issues and slow consumer where we see MaxPending of 67108864 exceeded--at this point, we no longer consume messages properly. We rarely get the dropping messages error otherwise. Is this the expected behavior with this high a throughput? We're using nats-go 1.46 and nats-server 2.11! In regards to a config, the consumer is listening to all subjects in the stream, and we only set the max_batch and max_ack_pending to be 1.2x max_batch. On the stream, however, we subscribe to a bunch of wildcards, have a 24 hour retention period and 100 GB for max_bytes. We've also set the max message size to be 1 MB. I usually set max_batch to be 2x the throughput/sec. It takes ~2.2s to get through the entire fetch loop + our processing with a batch of 100k messages. We also have an AckAll policy,. Any help would be greatly appreciated.

We're ideally trying to hit ~150k messages/sec from a total of 7500 subjects