Control metrics creation from selected pods based on state #2748
Description
What would you like to be added:
Would like to add, if not already present, a feature that allows to stop generating metrics from certain pods based on state. For example, failed/cancelled pods after a certain period of time.
Why is this needed:
To reduce the number of metrics being created and exported, thus reducing the load on an external collector, incase metrics from failed/cancelled pods aren't required.
Describe the solution you'd like:
Allow a configuration that sets the states that can be ignored i.e if a pod is in a certain state or set of states, avoid those when creating metrics after a certain amount of time.
Additional context:
When using an external collector like alloy in an env where there are 1000s of pods in 100s of clusters, not all pods have a ttl and hence continue to exist in the kubernetes environment, we can avoid generating metrics for such pods after a particular duration of time such that alloy won't have to scrape it, reducing the load on such components as well as the underlying network responsible for sending/transferring all these metrics.