[1.2.0] NFSv4 xprtsec=mtls breakage

[leo9800]

after upgrading ktls-utils to 1.2.0, NFSv4 no longer works with xprtsec=mtls

mounting would results in:

$ sudo mount -t nfs4 -o defaults,sec=krb5,xprtsec=none,nosuid,noatime,nofail n.oxlab.org:/ /mnt
mount.nfs4: Operation not permitted for n.oxlab.org:/ on /mnt

$ sudo mount -vvvv -t nfs4 -o defaults,sec=krb5,xprtsec=none,nosuid,noatime,nofail n.oxlab.org:/ /mnt
mount.nfs4: timeout set for Thu Jul 17 22:45:44 2025
mount.nfs4: trying text-based options 'sec=krb5,xprtsec=none,vers=4.2,addr=10.0.0.10,clientaddr=10.0.10.1'
mount.nfs4: mount(2): Operation not permitted
mount.nfs4: Operation not permitted for n.oxlab.org:/ on /mnt

the configuration file (/etc/tlshd.conf) on both server and client remains unchanged, which works properly with 1.1.0

contents of /etc/exports on server is:

/mnt *(rw,no_root_squash,no_subtree_check,sec=krb5,xprtsec=mtls,fsid=0)

by invoking pcap, the NFSv4 server replys NFS4ERR_WRONGSEC when attempted mounting, check pcap file for more: faulty-nfsv4-mtls.pcapng.txt (strip .txt suffix in filename, github does not allow .pcapng, the suffix is just for decepting github)

besides, setting xprtsec=mtls:tls:none in server /etc/exports and mount with xprtsec=none (as control) or xprtsec=tls worked both like a charm, seems it is just the TLS mutual auth (mtls) does not working properly.

[chucklever]

You can increase the gnuTLS library debug output for tlshd to find out why the TLS handshake failed. See "man tlshd.conf" for details. Then attempt the mount and look in the system journal on both the client and server for the error reports from tlshd.

[leo9800]

You can increase the gnuTLS library debug output for tlshd to find out why the TLS handshake failed. See "man tlshd.conf" for details. Then attempt the mount and look in the system journal on both the client and server for the error reports from tlshd.

thanks for advice, I'll try this later

[leo9800]

dive deeper ...

i tried downgrading ktls-utils on both server-side and client-side, it yield results as below:

• server=1.2.0 client=1.2.0 -> NOT WORKING, current setup in this issue
• server=1.2.0 client=1.1.0 -> NOT WORKING
• server=1.1.0 client=1.2.0 -> WORKING
• server=1.1.0 client=1.1.0 -> WORKING, last working setup

is it safe to conclude that server=1.2.0 is the issue and server-side log from 1.2.0 should be collected?

[chucklever]

is it safe to conclude that server=1.2.0 is the issue and server-side log from 1.2.0 should be collected?

I wouldn't make that assumption yet. Please collect both.

[leo9800]

the server-side log is collected for both 1.1.0 (working) and 1.2.0 (not working)

tlshd_server_1.1.0.log

tlshd_server_1.2.0.log

to my interpretation, there are 2 major discrepancies between the logs from 2 versions:

• Failed to find keyring lines show up in 1.2.0 log
  • Failed to find keyring '.nfs' and Failed to find keyring '.nfsd' may be related to NFS server malfunctioning
• the log entries related to client verification differs
  • in 1.1.0 log, The certificate is trusted. and The peer offered 1 certificate(s).
  • in 1.2.0 log, Peer certificate: subject (client cert x509 fields)

[leo9800]

ooops, seems i caught the root cause of this issue!

tl;dr: TLS hostname (SNI, subjectAltName) must be resolved to IP

the client, hostnamed Leo-NUC (not a valid domain name) is not DNS-resolved, appending Leo-NUC 10.0.10.1 to server's /etc/hosts, reboot both systems and everything back into working order.

besides, I got another server runs NFSv4 server hostnamed r.oxlab.org which is DNS-resolved, (therefore no /etc/hosts modification required) when mounting NFSv4 on this server everything work as usual.

[leo9800]

sorry pls reopen this issue, i made an error that #108 (comment) was test with server=1.1.0, pls just disregard the refered comment

despite adding /etc/hosts entries it still refuse to work properly with server=1.2.0

[chucklever]

The 1.2.0 server log you sent me states "Handshake successful" at the bottom. So I need to see both the client and server logs.

Edit /etc/tlshd.conf on both the client and server as below:

[debug]
loglevel=1
tls=3
nl=0

Restart tlshd on both systems, then try your mount. Post both logs here.

[benh-debian]

I think this would be fixed by #109.

[chucklever]

Weird. mtls mounts worked fine for me before I pushed.

[chucklever]

@leo9800 please test Ben's proposed fix and let us know if it helps.

[leo9800]

The 1.2.0 server log you sent me states "Handshake successful" at the bottom. So I need to see both the client and server logs.

Edit /etc/tlshd.conf on both the client and server as below:

[debug]
loglevel=1
tls=3
nl=0

Restart tlshd on both systems, then try your mount. Post both logs here.

thanks for detailed instruction

the logs on both server and client are now collected, with debug level set.

tlshd_client.log

tlshd_server.log

[leo9800]

@chucklever @benh-debian thanks for investigate into this issue

i'll try rebuild with the patch later