Ack latency per message

[kirpichenko]

I was exploring Google Cloud PubSub metrics and still not quite sure how to interpret Ack latency for subscription. From the metric description it's a Distribution of ack latencies in milliseconds. The ack latency is the time between when Cloud Pub/Sub sends a message to a subscriber client and when Cloud Pub/Sub receives an Acknowledge request for that message.

I did a simple node.js service which publishes message with a constant rate 2 messages per second.
Subscriber was configured to consume one message at the time (flowControl's maxMessages is set to 1) and waits 5 seconds before acknowledge received message. Metrics were pretty expected. Acknowledge operations rate is ~0.2/sec and Ack latency is around 5.2 sec.

Then I increased the amount of consumed messages to 5. The Acknowledge operations rate became 1/sec but Ack latency increased to 27sec even though messages were still resolving in 5 seconds

Increased the amount of consumed messages to 10 changed the Acknowledge operations rate to expected 2/sec but Ack latency stayed on the level of 27sec.

I assume GC builds that metrics on it's own but does it really show the latency per message and is there anything to do with the nodejs-pubsub library itself?

Environment:
Node.js 10.14.2
@google-cloud/pubsub: 2.7.0