DNSDist slow QPS

[nicolasbeusen]

Hello,

I have a DNSDist 1.8.3 installed on a 4vcpu server (Hetzner). Following the configuration:

addLocal("X.X.X.X:53", {reusePort=true})
addLocal("X.X.X.X:53", {reusePort=true})
addLocal("X.X.X.X:53", {reusePort=true})
addLocal("X.X.X.X:53", {reusePort=true})


setACL('0.0.0.0/0')

-- Console Configuration
controlSocket('Y.Y.Y.Y:5199')
setConsoleACL('Z.Z.Z.Z/32')
setKey("key")

-- Web Server Configuration
webserver("10.0.0.4:8083")
setWebserverConfig({password="password", apiKey="$scrypt$ln=10,p=1,r=apikey", acl="Z.Z.Z.Z/32"})


newServer({address="1.1.1.1:53", pool="default"})

addAction( AllRule(), PoolAction("default"))


-- disable security status polling via DNS
setSecurityPollSuffix("")

If I use dnsperf on cloudflare (1.1.1.1), I can achieve 25k QPS. But as soon as I use DNSDist in between, the QPS drops to ~900

CPU usage is really low (few percent)....

How can I improve the QPS with DNSDist ?

Thanks.

[paddg]

No idea. In any case, it's not nice to send 25 kqps to a public resolver. Keep in mind that it's not free, and there might be alerts there that need to be investigated by humans.

[nicolasbeusen]

The goal is not to reach 25k qps but to understand why the there is such a big difference when i'm testing with dnsperf directly to cloudflare or via dnsdist.

[rgacogne]

There are a lot of factors that might be in play in such a test. Would you mind telling us:

• which OS do you use? Did you do any tuning there, in particular network-level and firewall (no firewall would be best, otherwise please make sure that connection tracking is disabled) ;
• What exact dnsperf command are you using? Where is dnsperf executed from, in the cloudflare case and in the dnsdist -> cloudflare case?
• Did you test by having dnsdist generate the answer itself, for example by adding something like addAction(AllRule(), RCodeAction(DNSRCode.NOERROR)) before your PoolAction rule?

Normally I would want to know which exact hardware you are using but it looks like you are using a virtual server and might not have this information. Even low-end hardware would be capable of doing more than 900 QPS anyway, dnsdist on a raspberry pi 3b+ can do more QPS than that.

Note that you have declared 4 addLocals, which will use 4 CPU cores, plus one newServer directive which will use one core. This is already too many cores if you only have 4. I would recommend using 2 addLocals and 2 newServers in your case. But 900 qps is such a low rate that it probably does not matter at this point, it's much more likely that you are hitting a rate-limiting measure somewhere. I have seen the (D)DoS protection mechanisms of various providers being triggered by even low-level DNS benchmarks.

[nicolasbeusen]

Hello,

Thanks for your feedback.

Following answers to your questions:

• OS : Ubuntu 24.04
• tuning : I tried the following without any improvement
  sysctl -w net.core.rmem_max = 40000000 sysctl -w net.core.rmem_default = 40000000 sysctl -w net.core.wmem_max = 40000000 sysctl -w net.core.wmem_default = 40000000 sysctl -w net.core.netdev_max_backlog = 65536
• dnsperf command : dnsperf -s X.X.X.X -d dnsperf_source_100k.txt -l 60 -Q 15000 -m udp -p 53 -S 2 -q 1000 -T 4 -c 100
• Where is dnsperf executed from : another server on Hetzner (for both cases: in the cloudflare case and in the dnsdist -> cloudflare case)
• Did you test by having dnsdist generate the answer itself: not yet, I will give a try

Regards,

[nicolasbeusen]

Some new info :

• if I set the backend server with tcponly flag, the QPS are really better but not as high as it should with UDP
• if I run dnsperf on the same server as the dnsdist with a backend in UDP, the QPS ~= 900
• if I use RCodeAction(DNSRCode.NOERROR)), I able to reach high QPS (15K = limit of my dnsperf params)

Could it be a limitation on the OS side ? Any to what should I look at ?

Thanks.

[nicolasbeusen]

For info, I also tested with Alma 9 and I have the same issue... Next steps, I'll test on another cloud provider.

[nicolasbeusen]

Oki. So I tested on digitalocean and same results...

Is there any chance that you test on your side and validate my results ?

Thanks.

[rgacogne]

I'm not a big fan of dnsperf, I usually use our own tools, DNS flamethrower or DNS shotgun depending on what I am testing, so I don't really know what to expect from it. Would you mind trying with a higher value than -q 1000? If you have a fair amount of latency, which is expected since you are forwarding outside of the local network, having only 1000 concurrent queries is probably not going to yield great results.