Activities are inactive for a couple of minutes (up to 5 minutes)

[well0549]

Ok I have 12 seperate durable functions. (Consumption plan)
Each durable function listens to a servicebus queue and has approximately 4 to 5 activities. The activities are quite fast and will take at most 1 to 2 seconds.

So a batch of 20 messages is picked up from the queue and an orchestration is started for each message.
Then the activities will do their thing and eventually put the message to another servicebus queue. (or endpoint).
If it is send to a servicebus queue another durable function will process the message.

Everything works great. But I have a problem that activities get stuck for about 5 minutes.

If i put 1 message in everything works fine and usually all the functions execute quite fast and I am not left with stuck activities.

But when i put 1000 messages in, everything works graeat and I can see the instances scaling up. So far so good.
But when I am almost at the end of 1000 messages (say 10 left or so) I can see that activities become stuck for several minutes.
Eventually everything will get processed but those last 10 or 20 message take a long time to process. (up to 5 minutes)

If I have a small partition count (4) the behaviour is less obvious and I sometimes get 2 or three message stuck for 5 minutes.
But when I set the partition count to 16, the troughput is a lot better but I do get more of those stuck messages.

Is there anything i can do to optimize this behaviour ?