Help on exposing netbox plugin prometheus metrics

[lordofire]

Recently I'm developing a plugin and utilize rq-worker to perform periodic jobs. I would like to use prometheus exporter to expose the rq-worker metrics out (job latency, failure count, etc), but I'm thinking about what the best design approach it should work:

1. Let the rq-worker container write the prometheus metrics to /tmp/metrics directly, which is shared to netbox server container (nfs shared volume), so I could scrape netbox prometheus exporter in netbox server container to get the plugin metrics.
2. Write the job status to the plugin db directly per run, and then scrape from the netbox server directly to expose the metrics.

I felt the 1st approach is more natural with less db intrusion, and actually, I managed to get it to work for 1st approach with 1 netbox server container and 1 netbox rq-worker container and let them share the /tmp/metrics, and I think this is how this plugin works out as well.

However, to achieve better HA, I originally have two replica of netbox-server container with their individual /tmp/metrics volume, but 1st approach means I need to share the rq-worker volume to two netbox-server containers, and the conflicts start here, since I could not mix the volume between those two netbox-server containers, which would introduce issues for prometheus metrics functionality (return 500).

[lordofire]

After some further investigation, I think using the https://github.com/mdawar/rq-exporter will be perfect to solve my problem, since it will be a separate prometheus exporter which works without the dependency of netbox prometheus exporter, and it could provide a full picture for all the rq jobs status, queue status, etc.