How to set annotations on linkerd-viz prometheus pod?

[AlexGoris-KasparSolutions]

I would like to use annotations on the linkerd-viz prometheus pod, in order to use Elastic's metricbeat hints-based autodiscover feature to have one our metricbeat daemonset pods scrape metrics off of the prometheus instance, using the metricbeat prometheus metrics module.
I can not hardcode the metricbeat configuration in the daemonset resource, since that would trigger all pods of the daemonset to start scraping metrics, and generating duplicate data. With the hints-based autodiscover feature, one of the pods will pick up the annotations and start scraping.

Allthough the linkerd-viz helm chart docs do not mention any way to set annotations on pods, I've found some evidence (#5027, #5056 and #6151) that it should be possible to specify pod annotations in the values.yml file like so:

podAnnotations:
  my.custom.label.io/some-variable: some-value

But this will end up setting these annotations to all pods deployed by the linkerd-viz chart (grafana, metrics-api, ...) instead of just prometheus.

Is there any way (or workaround) to set annotations on the prometheus deployment/pod that does not get reset and/or overwritten after an updated helm deployment ?

[kleimkuhler]

If you add the annotation to the Pod spec for the the prometheus deployment, have you observed the annotation being overwritten? Going through a basic helm install for linkerd-viz, adding the annotation to the Pod spec, and helm upgrade, I don't see the annotation get removed from the prometheus Pod.

[adleong]

Personally, I'd recommend using Kustomize to adjust the linkerd-viz helm chart to your needs. https://linkerd.io/2.11/tasks/customize-install/