How is edgehub_queue_length calculated?

[mmoles-Growlink]

I can't make sense of this value. I know its tied to MessageCleanupInterval , Message TTL , Message Rate, and some form of failed\undelivered count.

I know about this documentation, but that just showed me that MessageCleanupInterval is part of its calculation. https://learn.microsoft.com/en-us/azure/iot-edge/troubleshoot-common-errors#solution-10

For my data here, I have a module that sends Messages back and forth with another module on the same device with multiple rates.
This data is from the azureiotedge-metrics-collector , and im viewing it from the device details workbook, so p90

Message TTL : 30 seconds , MessageCleanupIntervalSecs = default (1800)
30 Messages a second, queue size ~880
5 Messages a second, queue size 150
1 Message a second , queue size 10
1 Message per 30s , queue size 0
anything slower is 0.

Message TTL 30s , MessageCleanupIntervalSecs = 25
5 Messages a second, queue size 3 (bounces between 2 - 4 avg being 3)
1 Message a second, queue size is 0-1 , id guess 0.1 avg or less.

With no drops, the graph for edgehub_queue_length is basically a straight line. With Failed to deliver messages - it grows until cleanup hits, then drops to its usual queue length. As soon as it can deliver again, the line goes straight. So, if it fails for 1 minute it will look like a plateau.

Probably not necessary, here is some logs i pulled from my test module verifying message counts of sending and receiving.

It logs every 5 minutes.

Loop is an env var currently set to 30 messages a second in this instance, 1s = 1 a second, 30s = 1 every 30 seconds, 60s = 1 every minute, 5m = 1 every 5 minutes.

There is _Rec , _Send, _Diff. Rec represents total it has received, Send represents total it has sent. _Diff is simply Send-Receive. _Diff can be off due to the different times the 2 modules came online, but should stay consistent after that.

When sending at 30 messages a second, it kinda struggles to maintain the 9000 messages per 5min, but it hovers around it.

<2> 2025-04-11T07:33:06.3369093+00:00 [INF] - {"Loop_Rec":1011447,"  1s_Rec":33885," 30s_Rec":1130," 60s_Rec":565,"  5m_Rec":113,"Loop_Send":1011656,"  1s_Send":33889," 30s_Send":1130," 60s_Send":565,"  5m_Send":113,"Loop_Diff":209,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T07:38:06.3114408+00:00 [INF] - {"Loop_Rec":1020269,"  1s_Rec":34185," 30s_Rec":1140," 60s_Rec":570,"  5m_Rec":114,"Loop_Send":1020439,"  1s_Send":34189," 30s_Send":1140," 60s_Send":570,"  5m_Send":114,"Loop_Diff":170,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T07:43:06.3034823+00:00 [INF] - {"Loop_Rec":1029093,"  1s_Rec":34485," 30s_Rec":1150," 60s_Rec":575,"  5m_Rec":115,"Loop_Send":1029251,"  1s_Send":34489," 30s_Send":1150," 60s_Send":575,"  5m_Send":115,"Loop_Diff":158,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T07:48:06.3009612+00:00 [INF] - {"Loop_Rec":1038144,"  1s_Rec":34785," 30s_Rec":1160," 60s_Rec":580,"  5m_Rec":116,"Loop_Send":1038295,"  1s_Send":34789," 30s_Send":1160," 60s_Send":580,"  5m_Send":116,"Loop_Diff":151,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T07:53:06.3022863+00:00 [INF] - {"Loop_Rec":1047192,"  1s_Rec":35085," 30s_Rec":1170," 60s_Rec":585,"  5m_Rec":117,"Loop_Send":1047343,"  1s_Send":35089," 30s_Send":1170," 60s_Send":585,"  5m_Send":117,"Loop_Diff":151,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T07:58:06.3043911+00:00 [INF] - {"Loop_Rec":1056219,"  1s_Rec":35385," 30s_Rec":1180," 60s_Rec":590,"  5m_Rec":118,"Loop_Send":1056370,"  1s_Send":35389," 30s_Send":1180," 60s_Send":590,"  5m_Send":118,"Loop_Diff":151,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:03:06.2941528+00:00 [INF] - {"Loop_Rec":1065263,"  1s_Rec":35685," 30s_Rec":1190," 60s_Rec":595,"  5m_Rec":119,"Loop_Send":1065407,"  1s_Send":35689," 30s_Send":1190," 60s_Send":595,"  5m_Send":119,"Loop_Diff":144,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:08:06.3228129+00:00 [INF] - {"Loop_Rec":1074301,"  1s_Rec":35985," 30s_Rec":1200," 60s_Rec":600,"  5m_Rec":120,"Loop_Send":1074446,"  1s_Send":35989," 30s_Send":1200," 60s_Send":600,"  5m_Send":120,"Loop_Diff":145,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:13:06.3033699+00:00 [INF] - {"Loop_Rec":1083329,"  1s_Rec":36285," 30s_Rec":1210," 60s_Rec":605,"  5m_Rec":121,"Loop_Send":1083475,"  1s_Send":36289," 30s_Send":1210," 60s_Send":605,"  5m_Send":121,"Loop_Diff":146,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:18:06.3150303+00:00 [INF] - {"Loop_Rec":1092162,"  1s_Rec":36585," 30s_Rec":1220," 60s_Rec":610,"  5m_Rec":122,"Loop_Send":1092255,"  1s_Send":36589," 30s_Send":1220," 60s_Send":610,"  5m_Send":122,"Loop_Diff":93,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:23:06.3214776+00:00 [INF] - {"Loop_Rec":1100944,"  1s_Rec":36885," 30s_Rec":1230," 60s_Rec":615,"  5m_Rec":123,"Loop_Send":1101071,"  1s_Send":36889," 30s_Send":1230," 60s_Send":615,"  5m_Send":123,"Loop_Diff":127,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:28:06.3024978+00:00 [INF] - {"Loop_Rec":1109962,"  1s_Rec":37185," 30s_Rec":1240," 60s_Rec":620,"  5m_Rec":124,"Loop_Send":1110087,"  1s_Send":37189," 30s_Send":1240," 60s_Send":620,"  5m_Send":124,"Loop_Diff":125,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:33:06.3398763+00:00 [INF] - {"Loop_Rec":1118959,"  1s_Rec":37485," 30s_Rec":1250," 60s_Rec":625,"  5m_Rec":125,"Loop_Send":1119088,"  1s_Send":37489," 30s_Send":1250," 60s_Send":625,"  5m_Send":125,"Loop_Diff":129,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:38:06.2962871+00:00 [INF] - {"Loop_Rec":1127998,"  1s_Rec":37785," 30s_Rec":1260," 60s_Rec":630,"  5m_Rec":126,"Loop_Send":1128131,"  1s_Send":37789," 30s_Send":1260," 60s_Send":630,"  5m_Send":126,"Loop_Diff":133,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:43:06.3027652+00:00 [INF] - {"Loop_Rec":1137031,"  1s_Rec":38085," 30s_Rec":1270," 60s_Rec":635,"  5m_Rec":127,"Loop_Send":1137154,"  1s_Send":38089," 30s_Send":1270," 60s_Send":635,"  5m_Send":127,"Loop_Diff":123,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:48:06.3024747+00:00 [INF] - {"Loop_Rec":1146067,"  1s_Rec":38385," 30s_Rec":1280," 60s_Rec":640,"  5m_Rec":128,"Loop_Send":1146185,"  1s_Send":38389," 30s_Send":1280," 60s_Send":640,"  5m_Send":128,"Loop_Diff":118,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:53:06.3329195+00:00 [INF] - {"Loop_Rec":1155118,"  1s_Rec":38685," 30s_Rec":1290," 60s_Rec":645,"  5m_Rec":129,"Loop_Send":1155227,"  1s_Send":38689," 30s_Send":1290," 60s_Send":645,"  5m_Send":129,"Loop_Diff":109,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T08:58:06.3165276+00:00 [INF] - {"Loop_Rec":1163984,"  1s_Rec":38985," 30s_Rec":1300," 60s_Rec":650,"  5m_Rec":130,"Loop_Send":1164084,"  1s_Send":38989," 30s_Send":1300," 60s_Send":650,"  5m_Send":130,"Loop_Diff":100,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:03:06.3598239+00:00 [INF] - {"Loop_Rec":1172789,"  1s_Rec":39285," 30s_Rec":1310," 60s_Rec":655,"  5m_Rec":131,"Loop_Send":1172888,"  1s_Send":39289," 30s_Send":1310," 60s_Send":655,"  5m_Send":131,"Loop_Diff":99,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:08:06.3003862+00:00 [INF] - {"Loop_Rec":1181783,"  1s_Rec":39585," 30s_Rec":1320," 60s_Rec":660,"  5m_Rec":132,"Loop_Send":1181874,"  1s_Send":39589," 30s_Send":1320," 60s_Send":660,"  5m_Send":132,"Loop_Diff":91,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:13:06.3114708+00:00 [INF] - {"Loop_Rec":1190808,"  1s_Rec":39885," 30s_Rec":1330," 60s_Rec":665,"  5m_Rec":133,"Loop_Send":1190907,"  1s_Send":39889," 30s_Send":1330," 60s_Send":665,"  5m_Send":133,"Loop_Diff":99,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:18:06.3366935+00:00 [INF] - {"Loop_Rec":1199847,"  1s_Rec":40185," 30s_Rec":1340," 60s_Rec":670,"  5m_Rec":134,"Loop_Send":1199953,"  1s_Send":40189," 30s_Send":1340," 60s_Send":670,"  5m_Send":134,"Loop_Diff":106,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:23:06.3002332+00:00 [INF] - {"Loop_Rec":1208876,"  1s_Rec":40485," 30s_Rec":1350," 60s_Rec":675,"  5m_Rec":135,"Loop_Send":1208987,"  1s_Send":40489," 30s_Send":1350," 60s_Send":675,"  5m_Send":135,"Loop_Diff":111,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:28:06.3049431+00:00 [INF] - {"Loop_Rec":1217906,"  1s_Rec":40785," 30s_Rec":1360," 60s_Rec":680,"  5m_Rec":136,"Loop_Send":1218031,"  1s_Send":40789," 30s_Send":1360," 60s_Send":680,"  5m_Send":136,"Loop_Diff":125,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:33:06.3200418+00:00 [INF] - {"Loop_Rec":1226946,"  1s_Rec":41085," 30s_Rec":1370," 60s_Rec":685,"  5m_Rec":137,"Loop_Send":1227068,"  1s_Send":41089," 30s_Send":1370," 60s_Send":685,"  5m_Send":137,"Loop_Diff":122,"  1s_Diff":4," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:38:06.3098966+00:00 [INF] - {"Loop_Rec":1235772,"  1s_Rec":41384," 30s_Rec":1380," 60s_Rec":690,"  5m_Rec":138,"Loop_Send":1235839,"  1s_Send":41389," 30s_Send":1380," 60s_Send":690,"  5m_Send":138,"Loop_Diff":67,"  1s_Diff":5," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:43:06.3160146+00:00 [INF] - {"Loop_Rec":1244545,"  1s_Rec":41684," 30s_Rec":1390," 60s_Rec":695,"  5m_Rec":139,"Loop_Send":1244624,"  1s_Send":41689," 30s_Send":1390," 60s_Send":695,"  5m_Send":139,"Loop_Diff":79,"  1s_Diff":5," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:48:06.3220244+00:00 [INF] - {"Loop_Rec":1253507,"  1s_Rec":41984," 30s_Rec":1400," 60s_Rec":700,"  5m_Rec":140,"Loop_Send":1253577,"  1s_Send":41989," 30s_Send":1400," 60s_Send":700,"  5m_Send":140,"Loop_Diff":70,"  1s_Diff":5," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:53:06.3032416+00:00 [INF] - {"Loop_Rec":1262548,"  1s_Rec":42284," 30s_Rec":1410," 60s_Rec":705,"  5m_Rec":141,"Loop_Send":1262622,"  1s_Send":42289," 30s_Send":1410," 60s_Send":705,"  5m_Send":141,"Loop_Diff":74,"  1s_Diff":5," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T09:58:06.3212994+00:00 [INF] - {"Loop_Rec":1271590,"  1s_Rec":42584," 30s_Rec":1420," 60s_Rec":710,"  5m_Rec":142,"Loop_Send":1271673,"  1s_Send":42589," 30s_Send":1420," 60s_Send":710,"  5m_Send":142,"Loop_Diff":83,"  1s_Diff":5," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}
<2> 2025-04-11T10:03:06.3316186+00:00 [INF] - {"Loop_Rec":1280625,"  1s_Rec":42884," 30s_Rec":1430," 60s_Rec":715,"  5m_Rec":143,"Loop_Send":1280717,"  1s_Send":42889," 30s_Send":1430," 60s_Send":715,"  5m_Send":143,"Loop_Diff":92,"  1s_Diff":5," 30s_Diff":0," 60s_Diff":0,"  5m_Diff":0}

[mmoles-Growlink]

Also, what are dropped messages? ~285K each. Those modules are sending at ~30 messages a second back and forth to each other.

When I compare the logs the test modules are writing, they send and receive ~2,573,795 messages a day, and when i compare how many messages Mod1 sent to how many messages Mod2 received and vice versa, it showed a 9 and 13 message difference. So, it is not dropping 285k.

The code is simple. here are the relevant parts. It writes out every 5 minutes.

await moduleClient.SetInputMessageHandlerAsync( "onEveryLoopRec", onEveryLoopRec, moduleClient );
 _everyLoopTimer = new Timer( onEveryLoop, null, 0, throttleRefreshPeriod );


        private static Task<MessageResponse> onEveryLoopRec( Message message, object usercontext )
        {
            onEveryLoopRecCount++;
            return Task.FromResult(MessageResponse.Completed);
        }


private static async void onEveryLoop( object state )
{
    if ( _everyLoopTimerExecuting )
    {
       // I should have logged this, but it wont matter for this issue.
        return;
    }

    _everyLoopTimerExecuting = true;

    try
    {
        onEveryLoopSendCount++;
        await _moduleClient.SendEventAsync( "onEveryLoopSend", createMessage(onEveryLoopSendCount.ToString() ) );

    }
    catch ( Exception ex )
    {
        Console.WriteLine( ex.Message );
    }
    finally
    {
        _everyLoopTimerExecuting = false;
    }
}

[mmoles-Growlink]

I was doing some offline testing on another device and unplugged the switch my device was on. This is module to module communication, a device getting disconnected from its gateway shouldnt be a problem. I looked at the logs on my modules, and everything was working fine. No messages were lost. So, once again, i have no idea what EdgeHub is reporting for this metric.

[nyanzebra]

Hi @mmoles-Growlink the edgehub_queue_len metric is representative of how many messages are stored in the storage provider. For example, UsePersistentStorage is default to true and will use a RocksDB storage provider for holding onto messages until delivery is guaranteed. Upon storing a message the queue metric is incremented and on removal the metric is decremented. Because constantly removing values is expensive the cleanup happens periodically, specified with MessageCleanupIntervalSecs.

As for messages dropped the TTL for message expired before it was acknowledged. There should also be a reason label for the dropped messages that should indicate where and why those were dropped. Let us know if that does not appear in the portal :)

[mmoles-Growlink]

Hi @mmoles-Growlink the edgehub_queue_len metric is representative of how many messages are stored in the storage provider. For example, UsePersistentStorage is default to true and will use a RocksDB storage provider for holding onto messages until delivery is guaranteed. Upon storing a message the queue metric is incremented and on removal the metric is decremented. Because constantly removing values is expensive the cleanup happens periodically, specified with MessageCleanupIntervalSecs.

As for messages dropped the TTL for message expired before it was acknowledged. There should also be a reason label for the dropped messages that should indicate where and why those were dropped. Let us know if that does not appear in the portal :)

When is it decremented? Its NOT when its completed, and its not when the Message Cleanup fires. When message rate is lower, the queue length is dramatically lower.
5 Messages a second, queue size 150
1 Message a second , queue size 10... why is it not 30?

I know my stuff is all successful and sends \ completes instantly, so the queue must represent a combination of successful and unsuccessful messages. Otherwise, how would i have 900 queued messages ? Also, its a flat line, so the MessageCleanup is not lowering that count. If the queue was growing, then i know stuff is not delivering and getting queued, when it drops, thats when the MessageCleanup actually clears the failed messages. The Queue including completed messages would also explain why would the queue grows or shrinks depending on the setting of TTL \ MessageCleanupIntervalSecs.

But, that wouldnt explain why the queue shrinks in a non-linear fashion. If you saw a queue of 900, what does that tell you? I can send more messages a second, but lower the cleanup interval and achieve the same 900, or lower the messages a second and increase the cleanup interval. My 900 means everything is working perfectly and messages are completed which is confusing

What's the formula for it? Its not messages sent / cleanup interval or anything like that because of the non linear growth of messages sent.

[nyanzebra]

Hi @mmoles-Growlink the edgehub_queue_len metric is representative of how many messages are stored in the storage provider. For example, UsePersistentStorage is default to true and will use a RocksDB storage provider for holding onto messages until delivery is guaranteed. Upon storing a message the queue metric is incremented and on removal the metric is decremented. Because constantly removing values is expensive the cleanup happens periodically, specified with MessageCleanupIntervalSecs.
As for messages dropped the TTL for message expired before it was acknowledged. There should also be a reason label for the dropped messages that should indicate where and why those were dropped. Let us know if that does not appear in the portal :)

When is it decremented? Its NOT when its completed, and its not when the Message Cleanup fires. When message rate is lower, the queue length is dramatically lower. 5 Messages a second, queue size 150 1 Message a second , queue size 10... why is it not 30?

I know my stuff is all successful and sends \ completes instantly, so the queue must represent a combination of successful and unsuccessful messages. Otherwise, how would i have 900 queued messages ? Also, its a flat line, so the MessageCleanup is not lowering that count. If the queue was growing, then i know stuff is not delivering and getting queued, when it drops, thats when the MessageCleanup actually clears the failed messages. The Queue including completed messages would also explain why would the queue grows or shrinks depending on the setting of TTL \ MessageCleanupIntervalSecs.

But, that wouldnt explain why the queue shrinks in a non-linear fashion. If you saw a queue of 900, what does that tell you? I can send more messages a second, but lower the cleanup interval and achieve the same 900, or lower the messages a second and increase the cleanup interval. My 900 means everything is working perfectly and messages are completed which is confusing

What's the formula for it? Its not messages sent / cleanup interval or anything like that because of the non linear growth of messages sent.

Just to be clear, are you saying that the expectation is that if ingress is 900msg/s and egress is 900msg/s then the queue should be 0? Or are you asking what does 900 mean in this metric?

[mmoles-Growlink]

Just to be clear, are you saying that the expectation is that if ingress is 900msg/s and egress is 900msg/s then the queue should be 0? Or are you asking what does 900 mean in this metric?

What does a queue size of 900 mean.
30 Messages a second, queue size ~880
5 Messages a second, queue size 150
1 Message a second , queue size 10
1 Message per 30s , queue size 0

The metrics update interval in my charts are happening every 5 minutes, but I dont know how often edgeHub updates its own metrics.

With latency of 10-15ms and 100% delivery, they are only sitting in the queue as undelivered for .01 seconds. I have no idea how long they are sitting in the queue as delivered waiting to be cleaned up. I can force clear them by setting a shorter cleanup timer and it does lower the queue size. But when using the default cleanup interval of 1 hour, why are my graphs that use a 5 minute interval to poll the data a consistent flat line? In addition, why is it a flat line even though latency ranges from 10ms to 15ms? I would assume a 50% increase in latency would increase the queue size.

Also, if 30 messages a second = a queue size of 880 and 5 messages = 150... why does 1 message a second have a queue size of 10 and not 30, and 0.5 messages a second have 0 and not 15?

** These messages are being sent individually at a consistent rate and not as a batch or at the same time.

[nyanzebra]

Just to be clear, are you saying that the expectation is that if ingress is 900msg/s and egress is 900msg/s then the queue should be 0? Or are you asking what does 900 mean in this metric?

What does a queue size of 900 mean. 30 Messages a second, queue size ~880 5 Messages a second, queue size 150 1 Message a second , queue size 10 1 Message per 30s , queue size 0

The metrics update interval in my charts are happening every 5 minutes, but I dont know how often edgeHub updates its own metrics.

With latency of 10-15ms and 100% delivery, they are only sitting in the queue as undelivered for .01 seconds. I have no idea how long they are sitting in the queue as delivered waiting to be cleaned up. I can force clear them by setting a shorter cleanup timer and it does lower the queue size. But when using the default cleanup interval of 1 hour, why are my graphs that use a 5 minute interval to poll the data a consistent flat line? In addition, why is it a flat line even though latency ranges from 10ms to 15ms? I would assume a 50% increase in latency would increase the queue size.

Also, if 30 messages a second = a queue size of 880 and 5 messages = 150... why does 1 message a second have a queue size of 10 and not 30, and 0.5 messages a second have 0 and not 15?

** These messages are being sent individually at a consistent rate and not as a batch or at the same time.

Sorry this got lost for a bit... Anyway, there are a few variables at play. Firstly is the frequency in which the metrics are collected and pushed upstream. If you have a storage folder it looks like we will try to process whatever messages we got as a batch and then commit, and without a storage folder we will do thing one by one. These are still async tasks and there could be many happening and trying to progress. When processing of the message is complete, and if successful, we will decrement the queue count.

So to go through the example of consistent traffic (single msgs) and let's imagine that the metric is sent every minute. So with 30 msgs / s and a queue size of 880 means that at about 60s we sent out the messages but only roughly half of the responses were handled and so about half the queue was drained. With 5 msgs / s and a queue size of 150, it is the same story, at 60s we would have sent close to 300 msgs but it would seem about half were fully processed. Then finally with 1 msgs / s the pattern is a bit different, which might be due to MaxUpstreamBatchSize which defaults to 10. Basically, we cap the amount of messages to batch and send upstream and the previous examples might be sending faster to EdgeHub faster than EdgeHub can send upstream due to lower batch sizes. But with 1 msg /s and less, it is able to keep up and then the results differ.

We can confirm this if we change the batch size and see what the ingress and egress rates with EdgeHub are in your tests.