Clarification on Vector's Reaction to Loki Ingestion Rate Limit Exceeded

[LinTechSo]

Hi,
When i using the Loki sink in Vector, it appears that if the ingestion_rate_mb or ingestion_burst_size_mb rate limit is exceeded for a tenant, Loki drops the logs with a 429 Too Many Requests HTTP status code. This raises a question about how Vector, specifically through vector, reacts in such a scenario.

Retry Mechanism in Vector.dev:
In the event of a Loki ingestion rate limit being exceeded, does Vector have a built-in retry mechanism? If so, how does it operate, and are there any configurable parameters related to retry behavior?

Effect of Acknowledgment Setting:
When the ack setting is set to true, and if there's a Kafka as a vector source, my understanding is that Vector will continue to retry until it receives a 2xx HTTP status code and continues consume for example. Is this correct? Could you provide additional details on how the retry mechanism works in this specific context?

thanks

[jszwedko]

Hi @LinTechSo !

Retry Mechanism in Vector.dev: In the event of a Loki ingestion rate limit being exceeded, does Vector have a built-in retry mechanism? If so, how does it operate, and are there any configurable parameters related to retry behavior?

It should retry 429 errors. You can find retry configuration parameters here: https://vector.dev/docs/reference/configuration/sinks/loki/#request

Effect of Acknowledgment Setting: When the ack setting is set to true, and if there's a Kafka as a vector source, my understanding is that Vector will continue to retry until it receives a 2xx HTTP status code and continues consume for example. Is this correct? Could you provide additional details on how the retry mechanism works in this specific context?

It depends on the source. For request/response oriented sources, like the http source, a rejection in the sink will cause the source to return a 400 to the client. For stream sources like kafka or file , as a contrasting example, the data is "ack'd" if it is discarded avoid holding up the stream of processing. #14708 proposes some improvements in this area.

The data won't be ack'd or rejected while Vector is retrying a request. The data is considered "pending" in that case.

Hope this helps clear some thing sup!