Build errors using xcaddy

[csawyerYumaed]

1st, I know nothing about go so perhaps I'm doing something wrong? I apologize if that's the case.

I think I followed the instructions properly. I tried downloading from caddy's build service first, but that didn't work. I'm trying to play with ECH/ESNI released in caddy v2.10.0

Go and Xcaddy were just installed today on Ubuntu 24.04.2 LTS.
user@host:~/go/bin$ go version
go version go1.24.2 linux/amd64
user@host:~/go/bin$ ./xcaddy build --with github.com/caddy-dns/vultr
user@host:~/go/bin$ ./xcaddy version
v0.4.4 h1:xstmMhsxrDF/1rnncpi+CLbBfxEP0PjjdtXe3YFwnuo=
...

2025/04/22 07:13:34 [INFO] exec (timeout=0s): /usr/local/go/bin/go build -o /home/user/go/bin/caddy -ldflags -w -s -trimpath -tags nobadger,nomysql,nopgx
# github.com/libdns/vultr
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:47:14: invalid composite literal type libdns.Record
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:74:16: record.Name undefined (type libdns.Record has no field or method Name)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:75:16: record.Type undefined (type libdns.Record has no field or method Type)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:76:30: record.Value undefined (type libdns.Record has no field or method Value)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:77:20: record.TTL undefined (type libdns.Record has no field or method TTL)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:85:9: record.ID undefined (type libdns.Record has no field or method ID)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:96:64: record.ID undefined (type libdns.Record has no field or method ID)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:111:16: record.Name undefined (type libdns.Record has no field or method Name)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:112:16: record.Type undefined (type libdns.Record has no field or method Type)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:113:30: record.Value undefined (type libdns.Record has no field or method Value)
/home/zie/go/pkg/mod/github.com/libdns/vultr@v1.0.0/client.go:113:30: too many errors
2025/04/22 07:14:26 [INFO] Cleaning up temporary folder: /tmp/buildenv_2025-04-22-0712.366305572
2025/04/22 07:14:26 [FATAL] exit status 1

[Xinayder]

This is probably related to the update to libdns 1.0 libdns/vultr#4

I'm working on the update.

[csawyerYumaed]

Thanks for the response. No worries and No hurry on my part. Like I said I'm just wanting to play with ECH/ESNI, but don't have a need for it right now.

[chemnic]

Hi. How is this going?

Are there workarounds for this? a way of using: Caddy 2.10 + DNS challenge + Vultr

[Xinayder]

I've just pushed an update to both libdns and this library. Can you test if it works with caddy 2.10?

[freemvmt]

Hi folks, I've been trying to implement exactly the same as @chemnic by the sounds of it.

I'm working on an open source project called PlanX, and we're coming up against the LetsEncrypt limit of 50 certs/week, so decided to try and make use of the 'single wildcard cert applying to multiple subdomains' thing new in Caddy 2.10. This requires the DNS challenge, HTTP won't cut it. See the PR: theopensystemslab/planx-new#4757

So the conclusion I've been reaching, after a lot of iteration/trial and error, is that the issue is with the vultr dns module that we're building into the caddy image with the command RUN xcaddy build --with github.com/caddy-dns/vultr.

Then of course I found this issue and realised we're playing with something that is very much in development! So I wondered if our experience could be of use in troubleshooting the situation. I could also be wrong and it's some issue with my Caddyfile/docker-compose/overall setup etc...

So in general, the problem has not been that the Caddy container can't put the _acme_challenge TXT DNS records on our Vultr account. It's very successful at that. The issue is that it doesn't seem to recognise that they're there after the fact. So then it keeps adding them, and doesn't manage to cleanup, so they mount up like so:

Re the Caddyfile, I've ended up with a tls directive like so:

tls {
	dns vultr {$VULTR_API_KEY}
	# read like: wait for propagation_delay before starting poll of DNS records (after put), then keep trying for propagation_timeout
	propagation_delay 300s # default 0s (300s is the time Vultr gives in for DNS record TTL)
	propagation_timeout -1 # default 2mins (-1 indicates to not conduct propagation check, just handover to ACME after delay)
}

I set timeout to -1 in the hopes that the propagation check logic was bugging out and that just letting Caddy hand over to ACME without doing the sanity check would yield a positive result.

I also made sure that Caddy had sufficient buffer space in the kernel to do it's work, like so:

# caddy runs up against kernel buffer limits of host OS, so we increase them
# see https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes
sysctl -w net.core.rmem_max=7500000
sysctl -w net.core.wmem_max=7500000

Unfortunately, none of the above solved the issue, so I come cap in hand with a hefty log file emitted by my Caddy container. If anyone can make heads or tails of it, that'd be grand 🤔😁

caddy2.10_logs_mem.txt

Some choice lines include:

{"level":"error","ts":1749200593.4758897,"msg":"cleaning up solver","identifier":"*.4757.planx.pizza","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \\\"_acme-challenge.4757.planx.pizza\\\" (usually OK if presenting also failed)\", ...}

and...

{"level":"error","ts":1749200593.6530418,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"*.4757.planx.pizza", ... ,"error":"[*.4757.planx.pizza] solving challenges: waiting for solver certmagic.solverWrapper to be ready: no memory of presenting a DNS record for \\\"_acme-challenge.4757.planx.pizza\\\" (usually OK if presenting also failed) ..."}

Not going to have time to dig into this much more for the next 6 weeks, but if the problem persists thereafter, I'm happy to dig into the interals / look at the actual Go code. But perhaps one of youse can diagnose this much quicker.

Thanks in advance for any help!

[chemnic]

I've just pushed an update to both libdns and this library. Can you test if it works with caddy 2.10?

The build succeeded, but I'm having the same problems as mentioned by @freemvmt . The DNS records are not cleaned up, and Caddy is not getting a certificate. I get the same kind of errors shown by @freemvmt , so I guess there is no point in duplicate.

This is what I use in my Caddyfile:

tls admin@example.com {
		dns vultr {$VULTR_API_KEY}
		propagation_timeout -1
		propagation_delay 20s
	}

[Xinayder]

I have an idea what's going on. I'm using the ProviderData field for each named record type, which is not ideal since any call to libdns.RR() gets rid of that field to maintain portability and comparability. I guess the problem lies there, but will double-check later today.

Thanks for testing by the way, I'm not sure how I overlooked this on my tests.

[Xinayder]

@chemnic @freemvmt I've just pushed a new update to libdns/vultr (v2.0.2), and updated caddy-dns/vultr to use the latest version. I've rewritten the main logic for libdns to not use ProviderData, and instead wrapped libdns.RR on an internal struct which also saves the record ID. I also noticed there were some issues related to parsing SRV and MX records and their priority, which were fixed in this latest update.

Could you please try again with the new version and check if everything works now?

[chemnic]

Could you please try again with the new version and check if everything works now?

caddy list-modules --skip-standard --versions
dns.providers.vultr v0.0.0-20250607134539-f8ac7c9ae307

In my latest test I can get a new certificate, but the DNS records are still not cleaned up. This is the error I get:

{"level":"error","ts":1749500697.0253985,"msg":"cleaning up solver","identifier":"assets.fmorales.com","challenge_type":"dns-01","error":"deleting temporary record for name \"fmorales.com.\" in zone {\"_acme-challenge.assets\" \"0s\" \"TXT\" \"khQ45d3gpqktB6TzsUd8_NXxvY7izvFbBvJqe1VtQHM\"}: record has no ID: {_acme-challenge.assets 0s TXT khQ45d3gpqktB6TzsUd8_NXxvY7izvFbBvJqe1VtQHM}","stacktrace":"github.com/mholt/acmez/v3.(*Client).pollAuthorization\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:544\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\n\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:136\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\n\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\ngithub.com/caddyserver/certmagic.doWithRetry\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:700\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\n\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:415\ngithub.com/caddyserver/certmagic.(*jobManager).worker\n\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73"}

And in Vultr:

[Xinayder]

The issue now is that libdns has no tracking of the ID of the record being cleaned up. I don't know why, though. I'm gonna debug it in the next days and try to pinpoint the issue. One workaround I may add is to have an additional call to the API method that lists the DNS records for a domain if the custom struct I made has its ID field set to null, so that the library knows the ID of the record it's manipulating.

[freemvmt]

Could you please try again with the new version and check if everything works now?

Hey @Xinayder, thanks for your work on this. Unfortunately this didn't seem to do the trick! it's still managing to make the TXT records, but the DNS challenge is not succeeding, and it doesn't manage to clean them up either 🧹

Could the module accept failing to cleanup and still provision the SSL wildcard cert?

Let me know if I can be of any use debugging this!

[Xinayder]

@freemvmt @chemnic I'm finally checking up on this issue and I can say I faced the same issue using another DNS provider and wildcard certificates. I had the exact same errors as you, sometimes an error saying the challenge TXT record had the wrong value, but eventually, it obtained a certificate and cleaned up the DNS records.

Can you try again, using both subdomain certificates and wildcard, and check if it will eventually succeed in obtaining a certificate? If it does succeed, I'm willing to bet this is a problem with libdns/caddy, and not the dns provider itself.

Nonetheless, I'll setup a local instance with the Vultr plugin and check if I have the same issues.

If you can provide log files with the errors, including the lines before the errors happen (the ones that mention Caddy is trying to obtain a certificate for the domains), and the Caddyfile you're using, it would help a lot.

[chemnic]

Hi. Sorry for the delay on this. After several testing I can confirm that caddy gets the certificate, but won't remove the DNS record. And I think that even with my small setup (~10 sub domains), it would be a mess in the DNS records, so deploying this version is not good right now.

I have attached the log and Caddyfile from a test. I only redacted the mail address and API key. If you need some more info please let me know, maybe increasing the verbosity in the logs?.

caddy.log

Caddyfile.txt

[Xinayder]

Thanks for the logs. I set up a small instance on a VPS and I could reproduce the error, though the debug logs are not that much helpful. I've pushed an update to the library just now, it will try to retrieve the record ID using the Vultr API, and throw an error if it fails.

Some technical detail for the issue: I've wrapped the libdns.RR struct on a custom VultrRecord struct that contains a RR() function to return the libdns object, and a field to store the record ID we get from Vultr's API. There's a helper function that's supposed to retrieve the ID field from the wrapped struct, and it's returning an empty ID, which I think is due to caddy not storing the wrapped struct information properly.

The new patch attempts to fix this by checking if the ID returned from the helper function is empty, and if it is, then it calls Vultr API to list all records, find the one we want to delete, and get its ID.

It's a bit problematic to debug this because the example I have setup in the libdns repo works just fine for adding, deleting and updating DNS records, i.e., the wrapped struct is saved properly. I have no idea what caddy is doing different here.

EDIT: I tested locally and it seems to work. @chemnic can you test if certificates are issued correctly and the records are cleaned up?

You should use github.com/caddy-dns/vultr@v0.0.0-20250723121531-55bf3e9768be