Pipeline is not requesting items at some point

[siewp]

Hello there,

We are currently investigating an issue in our application where a Mutiny pipeline is suddenly not requesting items anymore. For context, we are using the Mutiny Vert.x Bindings RabbitMQ client. We debugged the app and found out that the RabbitMQ client consumes the message, but the message is not going further. I won't post all the code because of the complexity, but it looks somewhat like this:

    // ... other upstream stuff including setting up the connection
    .onItem()
    .transformToMulti(RabbitMQConsumer::toMulti) // there are messages in the ReadStream of the consumer
    // no messages are arriving in the downstream from here
    .call(this::callOtherMethod)
   // other stuff...
   .subscribe().with(item -> ...);

The app (and other apps as well which are basically using the same code) eventually gets in the state where the downstream after the transformToMulti is not executed. Internally, inside the onItem-method of the MultiConcatMapOp (created by the call-method downstream), the state is READY instead of PUBLISHER_REQUESTED. This happened multiple times in different apps, but a thing that they have in common is that, they received a lot of messages in a short time window before the pipeline hangs.

We don't know why this happens. It is very difficult to debug since it happened in Kubernetes environments and we have not been able to reproduce this state locally yet. Do you have any tips to find the cause of this or any idea why this happens? Any help would be greatly appreciated!

Thank you!

[jponge]

Hi,

Without a reproducer this is indeed hard to investigate.

Since you subscribe with .subscribe().with(...) the subscriber makes an unbounded initial request (Long.MAX_VALUE), but what happens next is likely to be impacted by the Rabbit driver and by the methods you have to make operations / transformations.

If this was to be a race-condition in Mutiny itself, then the ideal reproducer is agnostic of client libraries. Easier said that done, I once tracked a one in a million race condition for 3 months 🤣

[ivu-mawi]

That sounds very suspiciously similar indeed. We have the same pipeline in several apps (it's a library) and the "stalling" only occurs when the pipeline is "under load" (I'd guess 10s to 100s of items per second, maybe significantly more in certain cases).

First time I've seen this was 1 year ago in case that's relevant (Haven't checked when we've added the Uni.combine() code, can't have been much longer ago than that)

I will try Uni.join(), but we have in the past gone many weeks at a time without seeing the issue and currently it happens every few days. Which I'm happy about, otherwise I wouldn't have tracked it down now. But it means I can't be sure after switching to Uni.join() whether the absence of the stall means it's fixed or just randomly doesn't occur. ;)

[jponge]

No worries, but it looks like we indeed have some fun issue in combine() 😄

[ivu-mawi]

Struggling to understand what the functional difference between Join and Combine is atm 😅
Edit: Aha, #1426

[ivu-mawi]

Been testing for a week with Uni.join() instead and the issue has not occurred since, even though just before deploying that change, it had happened fairly regularly. I'm decently convinced that Uni.combine() is race-y and was the cause. Thanks!

[jponge]

Thanks a lot @ivu-mawi

Catching the bug is definitely on the radar.