How to manage > 100 domains?

[deubert-it]

Hi @mconfalonieri ,

thank you for your great work with this project.

Do you know anyone using this to manage > 100 domains?
This is of course meant as open question, so if anyone uses it like this, I'd be very happy about feedback.

We jumped from 6 to 142 domains and since then everything was constantly in rate limit (3600 per hour for our api key as well as 5000 per hour for our NAT ip).

We have externaldns set to "sync" policy, as we saw this in the documentation/readme of this project.

a) Would sync policy vs upsert-only translate to less api requests?

ExternalDNS was configured with a domain filter for 3 domains, however the webhook was not using a filter. After adding the filter and limiting to the same 3 test domains, we seem to fall out of the rate limit again so everything works.

b) Are there any settings that would help us, as we would want to manage ~130 of those domains via externaldns.
I was trying to understand what exactly happens and it seems to consist of pulling all available domain information, then filtering them internally, then sending updates for whatever externaldns says should be updated.
I'd hope to find a configuration set for both externaldns as well as this webhook to allow us to constantly manage > 100 domains. Our goal would be higher than 142, more like 280 actually.
Of course we don't need externaldns to check everything every minute, we already decreased the frequency though.

Maybe some form of tracking or metric related to hetzner api rate limit would be great, as per hetzner api documentation there should always be ratelimit specific headers set? https://docs.hetzner.cloud/reference/cloud#rate-limiting

[mconfalonieri]

Hi @deubert-it,

thank you!

Unfortunately I am not aware of anyone using such a high number of domains, but I will try to test it and do some optimizations for this use case.

Are you using the old DNS API or the new Cloud one? From the rate-limiting comment I am assuming the Cloud one, but could you confirm this?

a) the main difference is that upsert-only will add or update records, but won't delete them, while sync will manage deletions too. Using upsert-only would reduce the API calls for the old API because the deletions won't be handled, but if they occur the records will always be reported as out-of-sync.

b) unfortunately I don't think there is at the moment, but I think I can quickly implement a change that should reduce the API usage.

Metrics regarding rate/quota information sound like a great improvement. Thanks for the suggestion and for reaching out, I will come back with more information ASAP.

[deubert-it]

Hi @mconfalonieri ,

we are using the new Cloud API only.

Until the rate limiting everything worked great. However since there are undocumented rate limits by IP besides the documented rate limits per token, it also affected HCCM, certmanager (and less importantly hcloud-lb-exporter).

It took me a while to figure out what actually caused the rate limiting, but ultimately running external-dns as well as the webhook in debug mode + cilium hubble helped to find the issue.

I don't have "perfect" screenshots as I was scrambling to get out of the rate limiting, but I think I saw 50+ requests per Minute in some areas (the ip beginning with 213 is the hetzner api):

Since applying the domain filter settings to the webhook, it is down to around 4 requests per minute with external-dns interval 2m and policy sync.

a)
I understand the webhook logs always had the message "is running without any domain filters", however I'm wondering if there is any reasoning why the webhook would need settings that differ from external-dns. Maybe we could offer some "reuse filter settings from external-dns" if possible?

b)
Information about rate limiting from headers in the logs, optimally as exported metrics, would be great. However, in some scripts I used to import domains and zones, I did not see the ratelimit headers in every response from dns calls, so we might need to talk to hetzner api support about this.

c)
Allowing a small configurable/optional cache for zones responses probably makes sense, even if you have a "high" churn of domains/zones, I think waiting for something like 15 or 30 minutes until a new zone appears is usually fine. It would be fine for us at least. (if the cache is per process, we could just restart the webhook pod if we really need it to appear right now)

Metrics/Dashboard:
While trying to investigate and understand what's happening, I had AI build a dashboard, and I think it is not too bad.

It's probably obvious when the domainfilter was applied to the webhook.

This is the dashboard source if you are interested:

externaldns-hetzner-webhook.json

[mconfalonieri]

Quick update: before reading your second message I started to add a bulk mode that should reduce the amount of calls needed to update a zone to exactly two using the calls to export and import the zone file.

I like the idea of a cache system and I will start working on that after the bulk mode.

I will answer to the other points later, but in the meantime thank you for the dashboard.

[mconfalonieri]

a)
thanks to this note I found a bug: the provider didn't return properly the configured DomainFilter to ExternalDNS during the initial negotiations, when ExternalDNS tries to understand the capabilities of the webhook, but I don't think there is a mechanism to pass the DomainFilter from ExternalDNS to the webhook

b)
From my side I can track if the calls used by the webhook have the rate limit headers or not when I will implement these metrics. Please keep me in the loop in case you get an answer from Hetzner regarding which calls have these headers and which don't

c)
As I wrote, I like the idea of a cache system. I will implement it after the current change is done.

[mconfalonieri]

@deubert-it I implemented the zone cache, since the bulk mode is taking a lot to debug.

There is a new environment variable: ZONE_CACHE_TTL that sets the cache TTL in seconds. Default is 0 which means disabled for compatibility with previous versions.

The cache is implemented since v0.10.0 for both the cloud and legacy APIs.