Ensuring inflight messages are not dropped when HPA kills a Vector pod

[atibdialpad]

Hi Team,
We are using Vector's gcp_pubsub source to pull pubsub messages. We want to add k8s hpa to auto-scale based on CPU. To that my question: How can we ensure all inflight events (in memory or buffer) are gracefully handled before the HPA kills and removes a vector pod ? a.k.a is there a concept of "drain" ? If not, a hacky way to achieve the same could be to enable end to end acknowledgements, right ?

[jszwedko]

Hey!

If you send a SIGTERM to Vector sources will stop consuming data and Vector will attempt to flush all in-flight data through the topology and out the sinks. This includes disk buffers (this is a known issue: #19600).

There is a timeout of 60 seconds, for the shutdown, but this is configurable via https://vector.dev/docs/reference/cli/#vector_graceful_shutdown_limit_secs

Hopefully this helps.