Export metrics once per day for tracking over time

[pbol-samedi]

Hi there,

We would like to use the gitlab-ci-pipelines-exporter as a means to track pipelines and jobs over time. We've seen some issues and comments that the exporter isn't necessarily designed for tracking purposes, but we're hoping to make it work. In our company, we have a fairly large gitlab instance, with a considerable amount of projects (> 100) and about 100 (daily) active users, so jobs and pipelines get created all the time.

However, since we use the exporter for tracking purposes over longer periods of time, we don't need these metrics to be continuously scraped, just once a day is fine. Is there a configuration that would help us get the metrics just once per day? We deploy the exporter with helm in our kubernetes cluster. The configurations we played around with were a combination of:

• Increasing number of replicas
• Enabling redis and increasing maximum_jobs_queue_size
• Setting interval_seconds for pull.projects_from_wildcards, pull.refs_from_projects and pull.metrics to 86400 (1 day).

But either the number of queued tasks in redis don't decrease or not all metrics are exported. Sometimes metrics are also scraped multiple times (from multiple replicas), but we thought that with redis a metric would only get exported once.

Thanks for any help in advance