zeronsd does not listen on port 53 on MacOS

[peterhoneder]

Hi!

I can see typical startup output:

/usr/local/bin/zeronsd start -t /private/var/lib/zerotier-one/token -d zerotier <network-id>
Oct 18 09:23:48.614  INFO zeronsd::init: Welcome to ZeroNS!
Oct 18 09:23:49.583  INFO zeronsd::init: Your IP for this network: 172.25.221.93
Oct 18 09:23:49.961  INFO zeronsd::authority: Adding new record <blanked>
Oct 18 09:23:49.961  INFO zeronsd::authority: Adding new record <blanked>
Oct 18 09:23:49.961  INFO zeronsd::authority: Adding/Replacing record <blanked>
...

The records that are added looked correct to me.

But then I tried to resolve using the IP (172.25.221.93) and nothing happened. Looking at the list of open ports (sudo lsof -i -P -n | grep LISTEN|grep 53) shows nothing listening on port 53.

I then tried to launch with debug and trace log levels, but did not see any other information that would point me to the issue. Do you have any other ideas on how to debug this?

OS: MacOS 12.4
installed: via brew (github repo instructions)

[laduke]

Hello,
Homebrew worked for me on macOS 12.6

zeronsd 63175 root 14u IPv4 0x7c52d7942caa8867 0t0 TCP 10.147.19.23:53 (LISTEN)

Are you on Arm? I'm on Intel.

[erikh]

I will try to comb over the issues in detail next week. Been away from the computer for a while.

A knee-jerk on this ticket however suggests that there is a firewall involved. I would look there first.

[erikh]

So, reviewing this again, can you run this command for me and paste the result? It should produce no secrets:

sudo lsof -p `pidof zeronsd`

It would help me diagnose your issue.

[laduke]

One way to reproduce this is to not use be root/sudo when your start zeronsd. For whatever reason macos doesn't complain or crash the program, but the program doesn't work. sudo lsof -P -iUDP | grep zero does not contain any zeronsd unless you start it with sudo.

[glimberg]

It should require root to run. Port 53 is < 1024, and thus a "privileged" port that requires root

[laduke]

apple got rid of privileged ports at some point, but -get this- it's buggy.

https://news.ycombinator.com/item?id=18302380
https://developer.apple.com/forums/thread/674179
(there aren't any actual docs on the topic)

But yeah, zeronsd should exit when it's not root, or not able to use port 53 I guess. I tried on linux and the same thing happens. It starts but doesn't work.

[jordanarpad]

I have a Raspberry Pi 1 and Pi Zero for 2 different zerotier networks running exactly the same zeronsd binary (built over 12 hours with 512M swap) and listens port 53 on one but not on the other. Both are running pi-hole besides zeronsd.
Here is the requested output for one that does not listen:

colos@jozsi:~ $ sudo lsof -p pidof zeronsd
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND  PID USER   FD      TYPE     DEVICE SIZE/OFF   NODE NAME
zeronsd 2343 root  cwd       DIR      179,2     4096      2 /
zeronsd 2343 root  rtd       DIR      179,2     4096      2 /
zeronsd 2343 root  txt       REG      179,2 15558984 391959 /home/colos/zeronsd
zeronsd 2343 root  mem       REG      179,2    17972 137627 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so
zeronsd 2343 root  mem       REG      179,2    75568 137744 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so
zeronsd 2343 root  mem       REG      179,2    13760 137632 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2
zeronsd 2343 root  mem       REG      179,2    42628 137628 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so
zeronsd 2343 root  mem       REG      179,2  1319784 137204 /usr/lib/arm-linux-gnueabihf/libc-2.31.so
zeronsd 2343 root  mem       REG      179,2    13864 137274 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so
zeronsd 2343 root  mem       REG      179,2   386572 137539 /usr/lib/arm-linux-gnueabihf/libm-2.31.so
zeronsd 2343 root  mem       REG      179,2   137364 137721 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so
zeronsd 2343 root  mem       REG      179,2   116324 137362 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
zeronsd 2343 root  mem       REG      179,2  2150824 137243 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1
zeronsd 2343 root  mem       REG      179,2   459032 137813 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1
zeronsd 2343 root  mem       REG      179,2     9512 137135 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
zeronsd 2343 root  mem       REG      179,2   146888 136992 /usr/lib/arm-linux-gnueabihf/ld-2.31.so
zeronsd 2343 root    0r      CHR        1,3      0t0      4 /dev/null
zeronsd 2343 root    1u     unix 0xcf31dc27      0t0  43155 type=STREAM
zeronsd 2343 root    2u     unix 0xcf31dc27      0t0  43155 type=STREAM
zeronsd 2343 root    3u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 2343 root    4u  a_inode       0,13        0   6967 [eventfd]
zeronsd 2343 root    5u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 2343 root    6u     unix 0x2cdb490e      0t0  43171 type=STREAM
zeronsd 2343 root    7u     unix 0x2d73d79d      0t0  43172 type=STREAM
zeronsd 2343 root    8u     unix 0x2cdb490e      0t0  43171 type=STREAM
zeronsd 2343 root    9u     IPv4      43193      0t0    TCP pi.hole:59370->151.101.1.91:https (ESTABLISHED)
colos@jozsi:~ $ 

And here is the requested output for the one that does listen:

jordana@pi1b:~ $ sudo lsof -p `pidof zeronsd`
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND   PID USER   FD      TYPE     DEVICE SIZE/OFF   NODE NAME
zeronsd 15574 root  cwd       DIR      179,2     4096      2 /
zeronsd 15574 root  rtd       DIR      179,2     4096      2 /
zeronsd 15574 root  txt       REG      179,2 15558984   3184 /home/jordana/zeronsd
zeronsd 15574 root  mem       REG      179,2    17972   8494 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so
zeronsd 15574 root  mem       REG      179,2    75568   8519 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so
zeronsd 15574 root  mem       REG      179,2    13760   7546 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2
zeronsd 15574 root  mem       REG      179,2    42628   8495 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so
zeronsd 15574 root  mem       REG      179,2  1319784   8471 /usr/lib/arm-linux-gnueabihf/libc-2.31.so
zeronsd 15574 root  mem       REG      179,2    13864   8477 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so
zeronsd 15574 root  mem       REG      179,2   386572   8480 /usr/lib/arm-linux-gnueabihf/libm-2.31.so
zeronsd 15574 root  mem       REG      179,2   137364   8512 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so
zeronsd 15574 root  mem       REG      179,2   116324   7299 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
zeronsd 15574 root  mem       REG      179,2  2150824  11580 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1
zeronsd 15574 root  mem       REG      179,2   459032  11584 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1
zeronsd 15574 root  mem       REG      179,2     9512   7093 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
zeronsd 15574 root  mem       REG      179,2   146888   7656 /usr/lib/arm-linux-gnueabihf/ld-2.31.so
zeronsd 15574 root    0r      CHR        1,3      0t0      4 /dev/null
zeronsd 15574 root    1u     unix 0x61a3094a      0t0 283766 type=STREAM
zeronsd 15574 root    2u     unix 0x61a3094a      0t0 283766 type=STREAM
zeronsd 15574 root    3u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 15574 root    4u  a_inode       0,13        0   6967 [eventfd]
zeronsd 15574 root    5u  a_inode       0,13        0   6967 [eventpoll]
zeronsd 15574 root    6u     unix 0xb7630605      0t0 283774 type=STREAM
zeronsd 15574 root    7u     unix 0x0adc4997      0t0 283775 type=STREAM
zeronsd 15574 root    8u     unix 0xb7630605      0t0 283774 type=STREAM
zeronsd 15574 root    9u     IPv4     283794      0t0    TCP pi.hole:41042->151.101.1.91:https (ESTABLISHED)
zeronsd 15574 root   10u     IPv4     283801      0t0    TCP 192.168.193.1:domain (LISTEN)
zeronsd 15574 root   11u     IPv4     283802      0t0    UDP 192.168.193.1:domain 
jordana@pi1b:~ $ 

There is slightly more logs in journal for the one that listens and here is the diff:

May 08 11:24:02 pi1b sudo[15582]:  jordana : TTY=pts/0 ; PWD=/home/jordana ; USER=root ; COMMAND=/usr/bin/lsof -p 15574
May 08 11:24:02 pi1b sudo[15582]: pam_unix(sudo:session): session opened for user root(uid=0) by jordana(uid=1000)
May 08 11:24:03 pi1b sudo[15582]: pam_unix(sudo:session): session closed for user root

[erikh]

Do you have the journalctl​ for the zeronsd logs?

…

------- Original Message -------

On Monday, May 8th, 2023 at 3:48 AM, jordanarpad ***@***.***> wrote: I have a Raspberry Pi 1 and Pi Zero for 2 different zerotier networks running exactly the same zeronsd binary (built over 12 hours with 512M swap) and listens port 53 on one but not on the other. Both are running pi-hole besides zeronsd. Here is the requested output for one that does not listen: ***@***.***:~ $ sudo lsof -p pidof zeronsd lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs Output information may be incomplete. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME zeronsd 2343 root cwd DIR 179,2 4096 2 / zeronsd 2343 root rtd DIR 179,2 4096 2 / zeronsd 2343 root txt REG 179,2 15558984 391959 /home/colos/zeronsd zeronsd 2343 root mem REG 179,2 17972 137627 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so zeronsd 2343 root mem REG 179,2 75568 137744 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so zeronsd 2343 root mem REG 179,2 13760 137632 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2 zeronsd 2343 root mem REG 179,2 42628 137628 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so zeronsd 2343 root mem REG 179,2 1319784 137204 /usr/lib/arm-linux-gnueabihf/libc-2.31.so zeronsd 2343 root mem REG 179,2 13864 137274 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so zeronsd 2343 root mem REG 179,2 386572 137539 /usr/lib/arm-linux-gnueabihf/libm-2.31.so zeronsd 2343 root mem REG 179,2 137364 137721 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so zeronsd 2343 root mem REG 179,2 116324 137362 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1 zeronsd 2343 root mem REG 179,2 2150824 137243 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1 zeronsd 2343 root mem REG 179,2 459032 137813 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1 zeronsd 2343 root mem REG 179,2 9512 137135 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so zeronsd 2343 root mem REG 179,2 146888 136992 /usr/lib/arm-linux-gnueabihf/ld-2.31.so zeronsd 2343 root 0r CHR 1,3 0t0 4 /dev/null zeronsd 2343 root 1u unix 0xcf31dc27 0t0 43155 type=STREAM zeronsd 2343 root 2u unix 0xcf31dc27 0t0 43155 type=STREAM zeronsd 2343 root 3u a_inode 0,13 0 6967 [eventpoll] zeronsd 2343 root 4u a_inode 0,13 0 6967 [eventfd] zeronsd 2343 root 5u a_inode 0,13 0 6967 [eventpoll] zeronsd 2343 root 6u unix 0x2cdb490e 0t0 43171 type=STREAM zeronsd 2343 root 7u unix 0x2d73d79d 0t0 43172 type=STREAM zeronsd 2343 root 8u unix 0x2cdb490e 0t0 43171 type=STREAM zeronsd 2343 root 9u IPv4 43193 0t0 TCP pi.hole:59370->151.101.1.91:https (ESTABLISHED) ***@***.***:~ $ And here is the requested output for the one that does listen: ***@***.***:~ $ sudo lsof -p `pidof zeronsd` lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs Output information may be incomplete. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME zeronsd 15574 root cwd DIR 179,2 4096 2 / zeronsd 15574 root rtd DIR 179,2 4096 2 / zeronsd 15574 root txt REG 179,2 15558984 3184 /home/jordana/zeronsd zeronsd 15574 root mem REG 179,2 17972 8494 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so zeronsd 15574 root mem REG 179,2 75568 8519 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so zeronsd 15574 root mem REG 179,2 13760 7546 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2 zeronsd 15574 root mem REG 179,2 42628 8495 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so zeronsd 15574 root mem REG 179,2 1319784 8471 /usr/lib/arm-linux-gnueabihf/libc-2.31.so zeronsd 15574 root mem REG 179,2 13864 8477 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so zeronsd 15574 root mem REG 179,2 386572 8480 /usr/lib/arm-linux-gnueabihf/libm-2.31.so zeronsd 15574 root mem REG 179,2 137364 8512 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so zeronsd 15574 root mem REG 179,2 116324 7299 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1 zeronsd 15574 root mem REG 179,2 2150824 11580 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1 zeronsd 15574 root mem REG 179,2 459032 11584 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1 zeronsd 15574 root mem REG 179,2 9512 7093 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so zeronsd 15574 root mem REG 179,2 146888 7656 /usr/lib/arm-linux-gnueabihf/ld-2.31.so zeronsd 15574 root 0r CHR 1,3 0t0 4 /dev/null zeronsd 15574 root 1u unix 0x61a3094a 0t0 283766 type=STREAM zeronsd 15574 root 2u unix 0x61a3094a 0t0 283766 type=STREAM zeronsd 15574 root 3u a_inode 0,13 0 6967 [eventpoll] zeronsd 15574 root 4u a_inode 0,13 0 6967 [eventfd] zeronsd 15574 root 5u a_inode 0,13 0 6967 [eventpoll] zeronsd 15574 root 6u unix 0xb7630605 0t0 283774 type=STREAM zeronsd 15574 root 7u unix 0x0adc4997 0t0 283775 type=STREAM zeronsd 15574 root 8u unix 0xb7630605 0t0 283774 type=STREAM zeronsd 15574 root 9u IPv4 283794 0t0 TCP pi.hole:41042->151.101.1.91:https (ESTABLISHED) zeronsd 15574 root 10u IPv4 283801 0t0 TCP 192.168.193.1:domain (LISTEN) zeronsd 15574 root 11u IPv4 283802 0t0 UDP 192.168.193.1:domain ***@***.***:~ $ There is slightly more logs in journal for the one that listens and here is the diff: May 08 11:24:02 pi1b sudo[15582]: jordana : TTY=pts/0 ; PWD=/home/jordana ; USER=root ; COMMAND=/usr/bin/lsof -p 15574 May 08 11:24:02 pi1b sudo[15582]: pam_unix(sudo:session): session opened for user root(uid=0) by jordana(uid=1000) May 08 11:24:03 pi1b sudo[15582]: pam_unix(sudo:session): session closed for user root — Reply to this email directly, [view it on GitHub](#199 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAAET23BQUOV3KO5CGKQSI3XFDFO7ANCNFSM6AAAAAARHYXDKQ). You are receiving this because you commented.Message ID: ***@***.***>

[jordanarpad]

Not exactly sure what you mean but I realised the diff is more, again in favour of the one that listens as the following lines are also there and they are not there for the one that does not listen.
This other diff snippet with a few lines before and after:

May 09 00:14:37 pi1b zeronsd[20722]: May 09 00:14:37.333  INFO zeronsd::init: Welcome to ZeroNS!
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.694  INFO zeronsd::init: Your IP for this network: 192.168.x.x
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.709  INFO trust_dns_server::store::forwarder::authority: loading forwarder config: .
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.716  INFO trust_dns_server::store::forwarder::authority: forward resolver configured: .:
May 09 00:14:39 pi1b zeronsd[20722]: May 09 00:14:39.003  INFO zeronsd::authority: Adding new record zt-xxxxxx.zeronsd.: (192.168.x.x)

It can be seen from this snippet that the initial post of this issue also misses these lines that refer to the forwarder.
A bit difficult to share more publicly from journalctl without exposing my network.

[erikh]

what I'm specifically looking for are any errors about bind() calls or listen() calls. If there's something like that in the log, please include it and mask your IP.

…

------- Original Message -------

On Tuesday, May 9th, 2023 at 1:34 AM, jordanarpad ***@***.***> wrote: Not exactly sure what you mean but I realised the diff is more, again in favour of the one that listens as the following lines are also there and they are not there for the one that does not listen. This other diff snippet with a few lines before and after: May 09 00:14:37 pi1b zeronsd[20722]: May 09 00:14:37.333 INFO zeronsd::init: Welcome to ZeroNS! May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.694 INFO zeronsd::init: Your IP for this network: 192.168.x.x May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.709 INFO trust_dns_server::store::forwarder::authority: loading forwarder config: . May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.716 INFO trust_dns_server::store::forwarder::authority: forward resolver configured: .: May 09 00:14:39 pi1b zeronsd[20722]: May 09 00:14:39.003 INFO zeronsd::authority: Adding new record zt-xxxxxx.zeronsd.: (192.168.x.x) It can be seen from this snippet that the initial post of this issue also misses these lines that refer to the forwarder. A bit difficult to share more publicly from journalctl without exposing my network. — Reply to this email directly, [view it on GitHub](#199 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAAET23UAPR5CYWXO5NBPNLXFH6THANCNFSM6AAAAAARHYXDKQ). You are receiving this because you commented.Message ID: ***@***.***>

[jordanarpad]

there are no zeronsd logs with bind or listen in neither journal unfortunately even after setting ZERONSD_LOG and RUST_LOG to debug.
do I need to set there environmental variables during the build process for a debug binary?

[erikh]

Not that I can recall. I'm not sure what is happening, unfortunately, and I don't have a testbed that can accurately replicate this. If you could, for the benefit of others, clearly detail the OS you're using and hardware platform it runs on, that may help down the road. I'm very sorry there is not a better solution here.

…

------- Original Message -------

On Tuesday, May 9th, 2023 at 4:09 AM, jordanarpad ***@***.***> wrote: there are no zeronsd logs with bind or listen in neither journal unfortunately even after setting ZERONSD_LOG and RUST_LOG to debug. do I need to set there environmental variables during the build process for a debug binary? — Reply to this email directly, [view it on GitHub](#199 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAAET27JUB35OEZELH5VRGTXFIQW7ANCNFSM6AAAAAARHYXDKQ). You are receiving this because you commented.Message ID: ***@***.***>

[jordanarpad]

I figured out the difference. On the one where zeronsd listens I had configured pi-hole to bind only on eth0. On the other one zeronsd could not bind because pi-hole bind was not limited to an interface so bind them all.

just for the record I used these commands to build zeronsd on Raspberry Pi 1 running latest Raspberry Pi OS:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt install librust-openssl-dev
cargo install --git https://github.com/zerotier/zeronsd --branch main --jobs 1

and I had to increase swap memory to 512M for which I used this article as guidance: increase swap memory
the build process took a painful 12 hours but I can happily use the binary accross all my Raspberry Pi Zero W and Raspberry Pi 1

[erikh]

This makes sense, so zeronsd could not bind to the port because it was already registered as bound. Travis, can you make a note of this? There's probably a check in the bind() calls in the startup routines that isn't being handled as a failure.

…

------- Original Message -------

On Tuesday, May 9th, 2023 at 3:53 PM, jordanarpad ***@***.***> wrote: I figured out the difference. On the one where zeronsd listens I had configured pi-hole to bind only on eth0. On the other one where zeronsd could not bind because pi-hole bind was not limited to any interface. just for the record I used these commands to build zeronsd on Raspberry Pi 1 running latest Raspberry Pi OS: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh sudo apt install librust-openssl-dev cargo install --git https://github.com/zerotier/zeronsd --branch main --jobs 1 and I had to increase swap memory to 512M for which I used this article as guidance: [increase swap memory](https://pimylifeup.com/raspberry-pi-swap-file/) the build process took a painful 12 hours but I can happily use the binary accross all my Raspberry Pi Zero W and Raspberry Pi 1 — Reply to this email directly, [view it on GitHub](#199 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAAET24YGBOKNPFGM7EDGLLXFLDFBANCNFSM6AAAAAARHYXDKQ). You are receiving this because you commented.Message ID: ***@***.***>