Dnsdist Frontend Performance Tuning

[paddg]

I did a dnsdist frontend performance test, all queries was answered from packet cache. At a certain point with high IPv6 query rate, dnsdist stops receiving the additional queries and udp6-recvbuf-errors and udp6-in-errors increase.

I followed the dnsdist performance tuning manual and increased the addLocal() calls. But that didn't improve things. Then I started increasing the number of dnsdist processes. That helped. The dnsdist frontend seems to scale very well with the number of processes, but it doesn't benefit from more addLocal() calls.

Does anyone have an explanation for this?

[rgacogne]

Hi! I can see two possibilities:

• some lock in dnsdist gets contended, either in the receive/send path (unlikely, we are not using locks there as far as I know) or the packet cache, and adding more threads increases the contention so there is no benefit. You could try increasing the number of shards to see if it helps ;
• if the behaviour does not show reproduce with IPv4 it might be that something in the Linux kernel UDP IPv6 receive/send path gets contended, and having more processes works around the contention. I consider this a lot less likely, but we have seen strange things with regard to UDP performance in the past so it's not impossible.
  I would have to reproduce the behaviour and investigate, likely using perf to see what is happening when dnsdist is dropping packets, to be 100% sure. Would you mind giving us a few more details about how you are testing? How are you sending queries? From where (same host, local network, etc)? Any OS-level tuning? Anything special about your dnsdist configuration?

[paddg]

I found the bottleneck. I had implemented custom metrics where dq.qtype and dq.dh:getRD() were accessed for 5% (ProbaRule(0.05)) of the queries. This was killing the throughput. It seems to blocking the entire process at some point of qps.

[rgacogne]

Right, I'm guessing this was done via a LuaAction? LuaAction has the big drawback that it has to take a global Lua lock, thus effectively serializing threads. This can be avoided by using LuaFFIPerThreadAction instead, which has a per-thread Lua context and therefore does not need a lock, but it means using the FFI interface which is a bit more tedious to write. Let me know if you want some help to do that!

[paddg]

Help would be very appreciated :) I would like to give it a try if it doesn't get too complicated. This is my dnsdist Lua code:

qtype        = {}
qtype[1]     = "A"
qtype[2]     = "NS"
qtype[5]     = "CNAME"
qtype[6]     = "SOA"
qtype[7]     = "MB"
qtype[8]     = "MG"
qtype[9]     = "MR"
qtype[12]    = "PTR"
-- ...

function CustDqMetrics(dq)
   local qt = qtype[dq.qtype] or "OTHER"
   local lcqt = string.lower(qt)

   if declareMetric("cust-qtype-" .. lcqt, "counter", "QType " .. qt .. " counter" ) then
      incMetric("cust-qtype-" .. lcqt, 20)
   end

   if dq.dh:getRD() then
      if declareMetric("cust-qflag-rd1", "counter", "QFlag RD1 counter" ) then
         incMetric("cust-qflag-rd1", 20)
      end
   else
      if declareMetric("cust-qflag-rd0", "counter", "QFlag RD0 counter" ) then
         incMetric("cust-qflag-rd0", 20)
      end
   end

   return DNSAction.None, ""
end
addAction(ProbaRule(0.05),LuaAction(CustDqMetrics)) -- that kills the throughput at some point of qps

[rgacogne]

Here is a per-thread FFI version that should be equivalent:

local custDqMetrics = [[
  local ffi = require("ffi")
  local bit = require("bit")

  qtype        = {}
  qtype[1]     = "A"
  qtype[2]     = "NS"
  qtype[5]     = "CNAME"
  qtype[6]     = "SOA"
  qtype[7]     = "MB"
  qtype[8]     = "MG"
  qtype[9]     = "MR"
  qtype[12]    = "PTR"

  function declareMetric(name, description)
    ffi.C.dnsdist_ffi_metric_declare(name, #name, "counter", description, ffi.NULL)
  end
  function incMetric(name, value)
    ffi.C.dnsdist_ffi_metric_inc_by(name, #name, value)
  end

  declareMetric("cust-qflag-rd0", "QFlag RD0 counter")
  declareMetric("cust-qflag-rd1", "QFlag RD1 counter")

  return function(dq) 
    local qt = qtype[ffi.C.dnsdist_ffi_dnsquestion_get_qtype(dq)] or "OTHER"
    local lcqt = string.lower(qt)

    local metric = "cust-qtype-" .. lcqt
    local description = "QType " .. qt .. " counter"
    declareMetric(metric, description)
    incMetric(metric, 20)

    local header_void = ffi.C.dnsdist_ffi_dnsquestion_get_header(dq)
    local header = ffi.cast("unsigned char *", header_void)
    local rd = bit.band(header[2], bit.lshift(1, 0)) ~= 0
    if rd then
      metric = "cust-qflag-rd1"
    else
      metric = "cust-qflag-rd0"
    end
    
    incMetric(metric, 20)
    
    return DNSAction.None
  end
]]

addAction(ProbaRule(0.05), LuaFFIPerThreadAction(custDqMetrics))

[paddg]

Hello @rgacogne,

Your code looks like magic to me :) And it works! Thank you so much for that!

The throughput has improved, but remains well below the throughput without custom metrics. With my dnsdist setup, I achieve the following with one dnsdist process:

Without custom metrics: 105 kqps
With custom metrics LuaAction: 75 kqps
With custom metrics LuaFFIPerThreadAction: 90 kqps

Nothing is free. But the improvement is clearly visible.

[rgacogne]

Hello Winfried!
I'm wondering if we we cannot reduce the impact of the custom metrics by pre-declaring the metric for all types, as calling declareMetric involves a global lock. I'll try to test that later today.

[paddg]

I tried it with this code now:

local custDqMetrics = [[
  local ffi = require("ffi")
  local bit = require("bit")

  qtype        = {}
  qtype[1]     = "A"
  qtype[2]     = "NS"
  qtype[5]     = "CNAME"
  qtype[6]     = "SOA"
  qtype[7]     = "MB"
  qtype[8]     = "MG"
  qtype[9]     = "MR"
  qtype[12]    = "PTR"

  function declareMetric(name, description)
    ffi.C.dnsdist_ffi_metric_declare(name, #name, "counter", description, ffi.NULL)
  end
  function incMetric(name, value)
    ffi.C.dnsdist_ffi_metric_inc_by(name, #name, value)
  end

  declareMetric("cust-qflag-rd0", "QFlag RD0 counter")
  declareMetric("cust-qflag-rd1", "QFlag RD1 counter")

  for index, qt in pairs(qtype) do
    local lcqt = string.lower(qt)
    local metric = "cust-qtype-" .. lcqt
    local description = "QType " .. qt .. " counter"
    declareMetric(metric, description)
  end
  declareMetric("cust-qtype-other", "QType OTHER counter")

  return function(dq) 
    local qt = qtype[ffi.C.dnsdist_ffi_dnsquestion_get_qtype(dq)] or "OTHER"
    local lcqt = string.lower(qt)

    local metric = "cust-qtype-" .. lcqt
--  local description = "QType " .. qt .. " counter"
--  declareMetric(metric, description)
    incMetric(metric, 20)

    local header_void = ffi.C.dnsdist_ffi_dnsquestion_get_header(dq)
    local header = ffi.cast("unsigned char *", header_void)
    local rd = bit.band(header[2], bit.lshift(1, 0)) ~= 0
    if rd then
      metric = "cust-qflag-rd1"
    else
      metric = "cust-qflag-rd0"
    end
    
    incMetric(metric, 20)
    
    return DNSAction.None
  end
]]
addAction(ProbaRule(0.01), LuaFFIPerThreadAction(custDqMetrics))

And in fact, the throughput is about 5% higher.

[rgacogne]

Oh, that's nice, thank you! So the impact of using custom metrics is still noticeable. Looking at the code I think we can make the impact a bit lower, I'll do some tests.