pdns_recursor segfault crash on Ubuntu 24.04

[CBbIH9I]

Greetings.

I've two machines running Ubuntu 24.04, and switched to pdns recursor 5.2.0 (official powerdns apt repository), which started to produce following issue:

2025-03-12T20:45:56.808678+02:00 Telos kernel: rec/worker[10474]: segfault at 0 ip 0000590bbbce9f0c sp 00007113817dedc0 error 4 in pdns_recursor[590bbbabc000+605000] likely on CPU 5 (core 5, socket 0)
2025-03-12T20:45:57.121200+02:00 Telos systemd[1]: pdns-recursor.service: Main process exited, code=dumped, status=11/SEGV
2025-03-12T20:45:57.121494+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
2025-03-12T20:45:57.122017+02:00 Telos systemd[1]: pdns-recursor.service: Consumed 17min 1.514s CPU time.

this happens pretty often, for example:

2025-03-12T21:30:20.975481+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
2025-03-12T21:39:07.710787+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
2025-03-12T21:48:25.868074+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
2025-03-12T21:51:58.831804+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
2025-03-12T21:56:12.696496+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
2025-03-12T21:58:52.034868+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.
2025-03-12T22:02:44.946367+02:00 Telos systemd[1]: pdns-recursor.service: Failed with result 'core-dump'.

[Wed Mar 12 22:09:43 2025] rec/worker[27375]: segfault at 0 ip 000060be12713f0c sp 00007aba42597dc0 error 4 in pdns_recursor[60be124e6000+605000] likely on CPU 1 (core 1, socket 0)
[Wed Mar 12 22:09:43 2025] Code: 66 83 f8 06 0f 84 4c 0b 00 00 66 83 f8 29 0f 85 5d 0b 00 00 49 8b 7c 24 18 48 8b b5 70 f2 ff ff 4d 89 ef 48 8d 9d d0 f6 ff ff <48> 8b 07 ff 50 18 48 8b bd 00 f6 ff ff 0f b7 b5 78 f2 ff ff 48 8b
[Wed Mar 12 22:14:14 2025] rec/worker[27499]: segfault at 0 ip 0000583ae450af0c sp 000071215a8fddc0 error 4 in pdns_recursor[583ae42dd000+605000] likely on CPU 19 (core 1, socket 1)
[Wed Mar 12 22:14:14 2025] Code: 66 83 f8 06 0f 84 4c 0b 00 00 66 83 f8 29 0f 85 5d 0b 00 00 49 8b 7c 24 18 48 8b b5 70 f2 ff ff 4d 89 ef 48 8d 9d d0 f6 ff ff <48> 8b 07 ff 50 18 48 8b bd 00 f6 ff ff 0f b7 b5 78 f2 ff ff 48 8b

root@Telos:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.2 LTS
Release:        24.04
Codename:       noble

root@Telos:~# free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.5Gi        12Gi       7.2Mi       1.5Gi        14Gi
Swap:             0B          0B          0B

I've attached a config file to this post.

Any help would be welcome, because I failed to find anything nearly resembling my problem.

Thanks in advance.

local.yml.txt

[CBbIH9I]

Adding this to the systemd unit:

[Service]
LimitNOFILE=16384

seem to have fixed it, but I believe it shouldn't be an issue out of the box, am I wrong? Or, at least, it should provide something more intuitive to interpret, other than the segfault :)

[CBbIH9I]

Actually not, it's not fixed. Just took a bit more time to crash again.

[rgacogne]

segfault at 0 is a null-pointer dereference, so it's clearly a bug. Would you be able to share a core dump, by any chance? coredumpctl list then coredumpctl gdb <ID of core dump> then in the resulting gdb process thread apply all bt full.

[CBbIH9I]

segfault at 0 is a null-pointer dereference, so it's clearly a bug. Would you be able to share a core dump, by any chance? coredumpctl list then coredumpctl gdb <ID of core dump> then in the resulting gdb process thread apply all bt full.

Thanks for your attention, unfortunately, I only see this:

root@Telos:~# coredumpctl list
No coredumps found.

Maybe I need to enable something to save coredumps? I'd gladly share all information I could, because it clearly looks like a bug.

[rgacogne]

Looking at the IP pointer minus the base address, it always crashes at the instruction located at 22DF0C, could you try addr2line -e /usr/bin/pdns_recursor 0x22DF0C?

[CBbIH9I]

Looking at the IP pointer minus the base address, it always crashes at the instruction located at 22DF0C, could you try addr2line -e /usr/bin/pdns_recursor 0x22DF0C?

root@Telos:~# addr2line -e /usr/sbin/pdns_recursor 0x22DF0C
??:?

Not very helpful, I assume. However, now I managed to save coredump, will attach it here in a txt file.
coredump.txt

[omoerbeek]

No real useful info in there, because the debug info is lacking. Can you share the contents of /etc/powerdns/blacklist?
It's also possble it's the RPZ handling, so sharing the info to downlod the RPZ would also be helpful.

Additional info: did the crashes start the moment you switched to 5.2.0, or was it running fine before? What version were you running before the switch?

@rgacogne doesn't error 4 in pdns_recursor[60be124e6000+605000] suggest the offset is 605000 ?

[CBbIH9I]

No real useful info in there, because the debug info is lacking.

No debug-enabled build unfortunately, so yeah.

Can you share the contents of /etc/powerdns/blacklist?

root@Telos:~# head /etc/powerdns/blacklist 
1cent.in
1cent.tv
1cupis.ru
1tv.com
1tv.ru
1tvcrimea.ru
1x-bet.com
1xbet.com
1xslots-online.com
1xstavka.ru
root@Telos:~# wc -l /etc/powerdns/blacklist 
1250 /etc/powerdns/blacklist

So it's just a plaintext list of one domain per line that is a dirty hack to prevent them from resolving (it was working okish on 4.5. branch)

It's also possble it's the RPZ handling, so sharing the info to downlod the RPZ would also be helpful.

I'll gladly share it with you, however, it won't let you XFER the zone, because there's some ACL on that end.

  rpzs:
    - name: 'fraud-rpz.ua.db'
      addresses: ['185.13.250.11']
      policyName: 'fraud-rpz.ua'
      tsig:
        name: 'rpz-general'
        algo: 'hmac-sha512'
        secret: 'EsVb6fnZXTipluu4sFh/wlD6aYZqRsphEBRNMVxCW3N2cSXy1cVk+ojF5LLnR61viE7EF5v/TAwE/4jFIcI+VA=='
      refresh: 600

Thank you.

[omoerbeek]

I added some new questions as well in the post above. You probably can instal the debug info package (we publish them), but I don't know from the top of my head how to do that on Ubuntu and I have no opportunity to find out right now. Will do tomorrow.

[CBbIH9I]

I added some new questions as well in the post above.

Thanks, answers below:

Additional info: did the crashes start the moment you switched to 5.2.0, or was it running fine before?

Yes, it started crashing once installed 5.2.0 on Ubuntu 24.04

What version were you running before the switch?

4.7.4-1pdns.bionic on Ubuntu 18.04, and it's running fine with the same 'blacklist' and RPZ (only made using custom LUA script)

You probably can instal the debug info package (we publish them), but I don't know from the top of my head how to do that on Ubuntu and I have no opportunity to find out right now. Will do tomorrow.

Thank you, any way would do, downloading .deb package, via apt, or whatever the way there is.

[CBbIH9I]

I could easily let you or someone else into that server with no problems, if you need a testing environment, for example, just drop me a message and provide public key.

[omoerbeek]

Installing the debug symbols: apt install pdns-recursor-dbgsym
In my test the gdb command above then gets me full symbols.

[CBbIH9I]

Installing the debug symbols: apt install pdns-recursor-dbgsym In my test the gdb command above then gets me full symbols.

it's odd, because i've installed the package, and it provides these files, i would assume it would install binary with debug symbols:

root@Telos:~# dpkg -L pdns-recursor-dbgsym
/.
/usr
/usr/lib
/usr/lib/debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/55
/usr/lib/debug/.build-id/55/72f059f845b26736420b523ccd36868b71d1bd.debug
/usr/lib/debug/.build-id/68
/usr/lib/debug/.build-id/68/1c175cbf9bd29c767286fd30c7566399c26dda.debug
/usr/share
/usr/share/doc
/usr/share/doc/pdns-recursor-dbgsym

anyways, i'll proceed with getting you the coredump and will reply to the message below

[omoerbeek]

gdb and other tools that need this info pick it up from those dirs.

[omoerbeek]

Can you share the coredump? I now have an ubuntu 24.04 system running. To get the last coredump:

# coredumpctl -o rec.coredump dump /usr/sbin/pdns_recursor
# gzip  rec.coredump

And make it available for download somewhere and send me the URL privately ([email protected]). Please be careful posting more info here, as this potentially could be a security issue.

[CBbIH9I]

Can you share the coredump? I now have an ubuntu 24.04 system running. To get the last coredump:

# coredumpctl -o rec.coredump dump /usr/sbin/pdns_recursor
# gzip  rec.coredump

And make it available for download somewhere and send me the URL privately ([email protected]). Please be careful posting more info here, as this potentially could be a security issue.

Thanks for all your efforts to help, i've emailed you a link to download the coredump.