Requests per second spike every hour, on the hour

[tyronesluck]

I've noticed that every hour on the hour there is a huge spike on the graphs in grafana.
The DNS queries its 3-4k/s, the Blocked / Replaced its 150-200/s, and the client queries its 100-500/s but on only a few hosts.

I then checked prometheus and every hour on the hour, the number of blocked queries for example drops.

But it didn't do this for the first 5 hours or so that I was running the exporter.

I also actually noticed that in the screen shot example of the grafana dashboard in the repo there is a spike at midnight too.

https://github.com/henrywhitaker3/adguard-exporter/blob/main/grafana/dashboard.png

While writing this I believe I may have found the reason, but not sure if it can be fixed. The first drop in the prometheus graph was pretty much 24 hours after I started adguard-home, within 1 min of it starting, and I have adguard keeping stats for 24 hours. So it looks like every hour since then its been dropping off the oldest stats so the number of queries drops, but this shows as a spike in grafana?

[agravgaard]

Thanks for doing the investigation! I was seeing the same and was getting confused what was going on.

It seems setting the interval to 1h (the garbage collection interval you identified), or more, smoothes out the graph properly:

Default:

1h interval:

[swills]

I think it's because of AdGuardHome's querylog rotation. Currently testing setting querylog interval from 1 hour (default) to 8760h (1 year, maximum allowed).

[swills]

FWIW, setting querylog interval high didn't work. but perhaps setting statistics interval will work...

[swills]

Ultimately what worked for me whats setting both querylog interval and statistics interval to 8760h and also setting up a batch/v1 CronJob that would restart the AdGuard Home deployment daily.

[dchaib]

I believe the issue comes from the fact that adguard_queries (and similar metrics) represent "Total queries processed in the last 24 hours".

This makes it a rolling window metric, not a proper Prometheus counter. Since it represents a sliding 24h total, the value can decrease when older queries fall out of the window.

From observing the behavior, it appears that new queries are added continuously, while removal of old data happens in hourly batches. This causes periodic drops in the metric value.

Here is mine over 3 hours:

Because of this, using Prometheus' rate() function on adguard_queries (as done in the dashboard) produces incorrect results. rate() assumes a strictly monotonic counter, and when the value decreases due to window cleanup, it interprets this as a counter reset or large delta — resulting in artificial spikes.

The spikes visible in the dashboard align with the hourly cleanup rather than actual DNS activity.

Ideally, we would need a proper counter... or increasing the window, as @swills did.

[dchaib]

There is also the same issue with things like adguard_top_blocked_domains.

For example, I have a domain that was blocked and I stopped making queries to it. The metric looks like this:

The value never increases and decreases every hour.

Still, in the dashboard, as the query is done with increase (topk(15, sum by (domain) (increase(adguard_top_blocked_domains[$__range]))), it sill shows as being blocked 12k times overt he last hour:

The values are properly represented as gauges in Prometheus, but the way they are used in the dashboard are consistent with a counter which gives false or misleading information.