Performance Issues with SpoofCNAMEAction under High Load in dnsdist 2.0.0-alpha1

[Zqsdoo]

Hello,

I've been experimenting with dnsdist 2.0.0-alpha1, trying to use SpoofCNAMEAction under high traffic conditions, but it's not working as expected.
Specifically, with around 5,000 rules like the one below:

addAction("domain", SpoofCNAMEAction("target"))

I observe a significant performance drop.

Without any rules, my server can handle up to 1 million requests per second. However, with the spoofing rules in place, the throughput drops to around 100,000 requests per second.

My question is: What would be the best approach to efficiently spoof CNAME responses under heavy load in dnsdist?

Do you have any suggestions or best practices for handling this kind of scenario?

Thanks in advance!

[rgacogne]

It's impossible to be sure without seeing your actual full configuration, but I expect the performance impact is coming from the high number of rules you are defining. The way it works is that rules are evaluated one by one, so doing 5000 DNS suffixes comparison is going to hurt. If you are using the same target for all domains you can replace these 5000 by a single SuffixMatchNodeRule, like this:

mySMN = newSuffixMatchNode()
-- add a single name, you would need to do this for all the domains you want to match
mySMN:add(newDNSName("powerdns.com."))

addAction(SuffixMatchNodeRule(mySMN), SpoofCNAMEAction("target"))

If you need to have different targets based on the domain that was matched, I would advise looking at the Key-Value stores supported by dnsdist (CDB and LMDB): https://www.dnsdist.org/reference/kvs.html This would also make it a lot easier to handle updates to your list, since KVS can be automatically reloaded.

[Zqsdoo]

n. Have you looked at the metrics reported by the backend themselves?

Thank you for your response. I have already checked them, but maybe not in an overall graph. It’s true that there is a big difference in the average latency, but it looks like the difference appeared when I upgraded the dnsdist instance
According to the following graph, my downgrade from yesterday clearly affected this part. I’ll try upgrading them again and will give you feedback

After some investigation and upgrading all instances to the latest version, it looks like things are returning to a normal state

[Zqsdoo]

If I'm not mistaken your dnsdist setup is in front of recursive resolvers, right? If that's the case the variation might be due to a lot of things, I have seen cases where a small change in network routing toward one big actor (Cloudflare, Route 53, Akamai, etc..) causes such a small variation. Have you looked at the metrics reported by the backend themselves?

Hello,
Everything has returned to normal currently at a reasonable level :

I have a quick question. I've noticed a small number of errors about 10 to 30 per day which isn't a huge amount, but I’d still like to understand how to avoid them if possible or if you have any tips or good practice about that
Do you have any ideas on what might be causing these errors?
I’ve already done some sysctl tuning, but I may be missing some adjustments on the IPv6 side it could also be related to my dnsdist configuration ?
I’ve attached the contents of /proc/net/snmp6 collected over the past 33 days for reference

One more question I’d like to ask:
Based on our current configuration, do you think another load-balancing policy might be more suitable than the default one (which is leastOutstanding)?
Considering that the current traffic is only "test" traffic, and we expect the real traffic to be around 20 times higher but of course with more dnsdist and pdns-recursor instances I’d like to know your opinion on this.

[Zqsdoo]

If I'm not mistaken your dnsdist setup is in front of recursive resolvers, right? If that's the case the variation might be due to a lot of things, I have seen cases where a small change in network routing toward one big actor (Cloudflare, Route 53, Akamai, etc..) causes such a small variation. Have you looked at the metrics reported by the backend themselves?

Hello, Everything has returned to normal currently at a reasonable level : I have a quick question. I've noticed a small number of errors about 10 to 30 per day which isn't a huge amount, but I’d still like to understand how to avoid them if possible or if you have any tips or good practice about that Do you have any ideas on what might be causing these errors? I’ve already done some sysctl tuning, but I may be missing some adjustments on the IPv6 side it could also be related to my dnsdist configuration ? I’ve attached the contents of /proc/net/snmp6 collected over the past 33 days for reference One more question I’d like to ask: Based on our current configuration, do you think another load-balancing policy might be more suitable than the default one (which is leastOutstanding)? Considering that the current traffic is only "test" traffic, and we expect the real traffic to be around 20 times higher but of course with more dnsdist and pdns-recursor instances I’d like to know your opinion on this.

@rgacogne Do you have any idea about it ?

[rgacogne]

Hello! Sorry, I somehow completely forgot about your question, thank you for bringing it back to my attention! The IPv6 errors you are seeing are coming from a system-wide counter, so they might not be related to dnsdist. Honestly at such a low-level I would not spent any time on it unless you are aware of an actual issue.
As for the load-balancing policy, leastOutstanding is usually a very good default, so I would keep it unless you have an actual need that can be better filled by a different policy, but there is nothing in the latest configuration you shared in this ticket that suggests that to me.

[Zqsdoo]

Hello! Sorry, I somehow completely forgot about your question, thank you for bringing it back to my attention! The IPv6 errors you are seeing are coming from a system-wide counter, so they might not be related to dnsdist. Honestly at such a low-level I would not spent any time on it unless you are aware of an actual issue. As for the load-balancing policy, leastOutstanding is usually a very good default, so I would keep it unless you have an actual need that can be better filled by a different policy, but there is nothing in the latest configuration you shared in this ticket that suggests that to me.

Thank's for the response 😄

[Zqsdoo]

Thanks for the fast response
My actual configuration look's like that :

-- Set the local address for the service
setLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

addLocal("Your local ipv4 addr", {reusePort=true})
addLocal ('Your local ipv6 addr', {reusePort=true})

-- Add the local for DNS-over-HTTP/3 do multiple to be better
addDOH3Local('IPV6', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/.key',{internalPipeBufferSize=1048576,reusePort=true})

addDOH3Local('IPV4', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key',{internalPipeBufferSize=1048576,reusePort=true})


-- Add the local for DNS-over-HTTP/<2 (dont forget to add the header to specify that we have DNS-over-HTTP/3 enabled)
addDOHLocal('IPV6','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', "/dns", {customResponseHeaders={["alt-svc"]='h3=\":443\"'},internalPipeBufferSize=1048576,reusePort=true}) -- Internal pipe buffer value is the recommanded for high load

addDOHLocal('IPV4','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', "/dns", {customResponseHeaders={["alt-svc"]='h3=\":443\"'},internalPipeBufferSize=1048576,reusePort=true}) -- Internal pipe buffer value is the recommanded for high load

-- Add the local for DNS-over-QUIC (DoQ) (adding congestion control algo with brr for this on)
addDOQLocal('IPV6', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', {internalPipeBufferSize=1048576,congestionControlAlgo="bbr",reusePort=true})
addDOQLocal('IPV4', '/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key', {internalPipeBufferSize=1048576,congestionControlAlgo="bbr",reusePort=true})

-- Maybe add TLSlocal
addTLSLocal('IPV6','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key',{reusePort=true})
addTLSLocal('IPV4','/etc/ssl/acme.sh/fullchain.pem', '/etc/ssl/acme.sh/private.key',{reusePort=true})




-- Set the Access Control List (ACL) to allow all IPv4 and IPv6 addresses
setACL({"0.0.0.0/0", "::/0"})

-- Set the DNS Resolver (if you have only one backend you can use the same multiple time to run it on multiple thread)
newServer({address="IPV4 backend", name="example-back"})
newServer({address="IPV6 backend", name="example-back"})

newServer({address="IPV4 backend", name="example-back1"})
newServer({address="IPV6 backend", name="example-back1"})

newServer({address="IPV4 backend", name="example-back2"})
newServer({address="IPV6 backend", name="example-back2"})

newServer({address="IPV4 backend", name="example-back3"})
newServer({address="IPV6 backend", name="example-back3."})

newServer({address="IPV4 backend", name="example-back4"})
newServer({address="IPV6 backend", name="example-back4t"})

newServer({address="IPV4 backend", name="example-back5"})
newServer({address="IPV6 backend", name="example-back5"})

newServer({address="IPV4 backend", name="example-back6"})
newServer({address="IPV6 backend", name="example-back6."})

newServer({address="IPV4 backend", name="example-back7"})
newServer({address="IPV6 backend", name="example-back7"})

newServer({address="IPV4 backend", name="example-back8"})
newServer({address="IPV6 backend", name="example-back8"})

newServer({address="IPV4 backend", name="example-back9"})
newServer({address="IPV6 backend", name="example-back9"})

newServer({address="IPV4 backend", name="example-back10"})
newServer({address="IPV6 backend", name="example-back10"})

newServer({address="IPV4 backend", name="example-back11"})
newServer({address="IPV6 backend", name="example-back11"})

newServer({address="IPV4 backend", name="example-back12"})
newServer({address="IPV6 backend", name="example-back12"})

newServer({address="IPV4 backend", name="example-back13"})
newServer({address="IPV6 backend", name="example-back13"})


webserver("Your local ipv4 addr:8083")
setWebserverConfig({
  password="password",
  apiKey="apikey)",
  acl="ipv4"
})

-- UDP Configuration
setMaxUDPOutstanding(65535)
setRingBuffersSize(65535)
-- Set this to 0 0 to use the systemctl configuration
setUDPSocketBufferSizes(0, 0)
-- Increase thread count for TCP client connections (10 is the default value ) 
setMaxTCPClientThreads(10)
-- https://www.dnsdist.org/advanced/tuning.html#outgoing-doh
setOutgoingDoHWorkerThreads(8)
-- https://www.dnsdist.org/reference/tuning.html#setMaxTCPQueuedConnections
setMaxTCPQueuedConnections(0) --0 mean unlimited
setTCPInternalPipeBufferSize(1048576)


-- Set the cache size (20000000 entries) and the cache policy (https://www.dnsdist.org/guides/cache.html) for 16GB ram
pc = newPacketCache(20000000, {maxTTL=86400, minTTL=0, temporaryFailureTTL=60, staleTTL=60, dontAge=false})
getPool(""):setCache(pc)

-- BPF Configuration (Packet Filtering)
-- Define a new BPF filter with specific limits for IPv4, IPv6, and query names
bpf = newBPFFilter({ipv4MaxItems=500000, ipv6MaxItems=500000, qnamesMaxItems=500000})
setDefaultBPFFilter(bpf)
-- Create a dynamic block rules group (used for rate limiting and blocking)
local dbr = dynBlockRulesGroup()

function maintenance()
  dbr:apply()
end

-- Set the control socket
controlSocket("127.0.0.1:5199")
setKey("To set using makeKey()")

I precise that i run under 6.8.0 kernel with some tuning with sysctl value
And to be honest, I mainly have a lot of different domains that redirect to the same CNAME. Based on that, I'll explore your solution.
Don't hesitate to help me with the configuration to optimize it 😄

[rgacogne]

At a quick glance your configuration looks good. You might be able to optimize the dnsdist <-> backend path, especially if your backends support TCP out-of-order processing: https://www.dnsdist.org/advanced/out-of-order.html
Please let us know if my suggestion worked as expected, as we might want to investigate what is happening if it does not, and otherwise it can be useful to others users :)

[Zqsdoo]

I actually use pdns_recursor as the backend, and based on the performance tuning guide, I believe that I could enable out-of-order processing. I'll test that later, but since my main objective is to optimize the UDP part and DOH, these other features will be a bonus. However, they might slow things down, so I’ll need to proceed with caution.

Here’s my current setup:

dnssec:
  trustanchorfile: /usr/share/dns/root.key 
recursor: 
  hint_file: /usr/share/dns/root.hints
  include_dir: /etc/powerdns/recursor.d

  # Tuning
  threads: 8              # More threads for query handling (scales with CPU cores)
  tcp_threads: 8           # Keep this lower unless many TCP queries are expected
  max_mthreads: 32768      # Increase for high concurrency, allow more lightweight threads
  stack_cache_size: 65536  # Larger for deeper stacks if needed

incoming:
  pdns_distributes_queries: true
  distributor_threads: 8
  reuseport: true
  max_tcp_clients: 16384   # High TCP load tolerance

  max_concurrent_requests_per_tcp_connection: 15000

outgoing:
  # source_address: 0.0.0.0 (default)

recordcache: 
  max_entries: 10000000    # Boosted for high traffic (8.5GB RAM est.)
  max_ttl: 86400           # Cache records longer

packetcache:
  max_entries: 4000000     # 500 bytes each, ~2 GB memory

webservice:
  address: ipv4
  port: port
  api_key: 'apikey'
  password: 'password'
  allow_from: ['ipv4']
  webserver: true

I’m also planning to add the maxInFlight parameter to both the backend and the addlocal section. I’ll need to think about a reasonable value for those settings. Once I have that figured out, I'll send an update here. 😊

[rishu007]

@rgacogne can we dynamically return CNAME from dnsdsit?
for example "-
Query:-
test--hostname.domain.com
expected cname
test.hostname
is this possible like get part of query and generate dynamic cname ?