1000 nats cli watchers leads to unrecoverable state of servers.

[vineethm13]

Hi, I'm having 1000 clients watching for a key in a bucket using nats cli. My nats cluster is of 3 nodes deployed in k8s. I'm observing a unrecoverable state in nats cluster around this scale. I can see that all the pods are throttling at configured limit of 4 cores cpu. Same issue can be observed after a spike in connection requests to the server. What could I be possibly doing wrong?

16:24:30.698166 [INF] JetStream cluster new consumer leader for '$G > KV_wincloud-sdc-delta > h0mGFs2q'                                                                                │
│ [60] 2024/03/25 16:24:30.698325 [WRN] Healthcheck failed: "JetStream consumer '$G > KV_wincloud-sdc-delta > h0mGFs2q' is not current"                                                                  │
│ [60] 2024/03/25 16:24:30.700038 [WRN] 40.0.156.123:50022 - cid:30807 - Readloop processing time: 2m11.969109716s                                                                                         │
│ [60] 2024/03/25 16:24:30.818906 [WRN] Consumer assignment for '$G > KV_wincloud-sdc-delta > Vfwfb1rz' not cleaned up, retrying                                                                         │
│ [60] 2024/03/25 16:24:30.854757 [WRN] 40.0.31.23:57036 - cid:33047 - Readloop processing time: 1m55.007560264s                                                                                           │
│ [60] 2024/03/25 16:24:31.061018 [WRN] 40.0.31.23:57082 - cid:33051 - Readloop processing time: 1m55.132755622s                                                                                           │
│ [60] 2024/03/25 16:24:31.102976 [WRN] 40.0.156.123:49276 - cid:29229 - Readloop processing time: 2m15.400043668s                                                                                         │
│ [60] 2024/03/25 16:24:31.363616 [WRN] 40.0.31.23:34432 - cid:32285 - Readloop processing time: 2m0.286123064s                                                                                            │
│ [60] 2024/03/25 16:24:31.368848 [WRN] 40.0.156.123:50172 - cid:30847 - Readloop processing time: 2m11.758467552s                                                                                         │
│ [60] 2024/03/25 16:24:31.588666 [ERR] Error trying to connect to route (attempt 2): dial tcp 40.0.22.98:6222: i/o timeout                                                                                │
│ [60] 2024/03/25 16:24:31.699269 [WRN] 40.0.156.123:48678 - cid:32223 - Readloop processing time: 2m1.879502946s                                                                                          │
│ [60] 2024/03/25 16:24:34.563968 [WRN] Consumer assignment for '$G > KV_wincloud-sdc-delta > ez6ULYpd' not cleaned up, retrying                                                                         │
│ [60] 2024/03/25 16:24:35.298900 [WRN] Consumer assignment for '$G > KV_wincloud-sdc-delta > K4R9DWlX' not cleaned up, retrying                                                                         │
│ [60] 2024/03/25 16:24:32.024105 [WRN] Consumer assignment for '$G > KV_wincloud-sdc-delta > S6RhKKTX' not cleaned up, retrying                                                                         │
│ [60] 2024/03/25 16:24:32.095089 [WRN] Consumer assignment for '$G > KV_wincloud-sdc-delta > pQJ3gNio' not cleaned up, retrying                                                                         │
│ [60] 2024/03/25 16:24:32.135600 [WRN] Consumer assignment for '$G > KV_wincloud-sdc-delta > QTVmKMJL' not cleaned up, retrying                                                                         │
│ [60] 2024/03/25 16:24:32.329053 [WRN] Consumer assignment for '$G > KV_wincloud-sdc-delta > OCSC0ANf' not cleaned up, retrying

[derekcollison]

What server version? What does stream info look for the KV?

[vineethm13]

server version nats:2.10.12-alpine

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                              Stream Report                                               │
├─────────────────────────┬─────────┬───────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream                  │ Storage │ Placement │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├─────────────────────────┼─────────┼───────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ KV_wincloud-sdc-delta   │ File    │           │ 0         │ 1        │ 118 B │ 0    │ 0       │ nats-1*  │
╰─────────────────────────┴─────────┴───────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

[derekcollison]

Looping in @wallyqs and @caleblloyd as they probably want to know more about how the cluster is deployed and if its healthy..

[vineethm13]

Any update here

[derekcollison]

Could you try against latest release candidate for our next release?

https://github.com/nats-io/nats-server/releases/tag/v2.10.17-RC.5

[vineethm13]

Sure, Can you explain what is exactly causing this issue? I have tried with the python sdk watch as well, issue is seen there too.