ZFS send receive fails over LAN with "cannot receive incremental stream: incomplete stream"

[montanaviking]

Hi,
I am attempting to send all my snapshots from an old server (named sibyl) to a new server (named thor) via a 1G LAN using Netcat.
###########################
Source machine: Sibyl
Source OS: Ubuntu 18.04, Linux Sibyl 5.4.0-150-generic #16718.04.1-Ubuntu SMP Wed May 24 00:51:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Source ZFS version:
$ modinfo zfs | grep version
version: 2.1.6-0york118.04
srcversion: EBD93AD6D6AD43867A67BFA
vermagic: 5.4.0-150-generic SMP mod_unload modversions
Netcat version: OpenBSD netcat (Debian patchlevel 1.187-1ubuntu0.1)
###################################
Destination machine: thor
OS: Ubuntu 22.04, Linux thor 5.19.0-45-generic #4622.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 7 15:06:04 UTC 20 x86_64 x86_64 x86_64 GNU/Linux
ZFS version:
$ modinfo zfs | grep version
version: 2.1.5-1ubuntu622.04.1
srcversion: 5A94B4662A7A991696CC35F
vermagic: 5.19.0-45-generic SMP preempt mod_unload modversions
Netcat version: OpenBSD netcat (Debian patchlevel 1.218-4ubuntu1)
##################################################
Actions:
Send first (oldest) snapshot first
First do receive from thor (destination machine)
thor: $ nc -w 120 -l -p 8023 | sudo zfs receive -vduF datatank
Then send on Sibyl
Sibyl:$ sudo zfs send datatank/carbonics@zfs-auto-snap_monthly-2022-07-06-0617 | nc -w 20 thor 8023
Result: snapshot successfully copied from ZFS dataset on Sibyl to dataset on thor.
##########################################
Now try to send all the remaining snapshots up to the most recent. This is where the problem is:
First do receive from thor (destination machine)
thor: $ nc -w 120 -l -p 8023 | sudo zfs receive -vduF datatank
Then send incremental snapshots on Sibyl
Sibyl:$ sudo zfs send -I datatank/carbonics@zfs-auto-snap_monthly-2022-07-06-0617 datatank/carbonics@zfs-auto-snap_hourly-2023-06-22-0517 | nc -w 20 thor 8023

Result from thor:
receiving incremental stream of datatank/carbonics@zfs-auto-snap_monthly-2022-08-06-0627 into datatank/carbonics@zfs-auto-snap_monthly-2022-08-06-0627
snap datatank/carbonics@zfs-auto-snap_monthly-2022-08-06-0627 already exists; ignoring
received 0B stream in 2755 seconds (0B/sec)
receiving incremental stream of datatank/carbonics@zfs-auto-snap_monthly-2022-09-06-0216 into datatank/carbonics@zfs-auto-snap_monthly-2022-09-06-0216
snap datatank/carbonics@zfs-auto-snap_monthly-2022-09-06-0216 already exists; ignoring
received 0B stream in 2788 seconds (0B/sec)
receiving incremental stream of datatank/carbonics@zfs-auto-snap_monthly-2022-10-05-1545 into datatank/carbonics@zfs-auto-snap_monthly-2022-10-05-1545
snap datatank/carbonics@zfs-auto-snap_monthly-2022-10-05-1545 already exists; ignoring
received 0B stream in 2769 seconds (0B/sec)
receiving incremental stream of datatank/carbonics@zfs-auto-snap_monthly-2022-11-04-0638 into datatank/carbonics@zfs-auto-snap_monthly-2022-11-04-0638
cannot receive incremental stream: incomplete stream
#############################################
The snapshot transfer fails with "cannot receive incremental stream: incomplete stream"
Repeating the above simply replicates the same error.
Any ideas?
Thanks in advance!
Phil

[rincebrain]

The formatting on this makes it impossible to understand what's going on.

From what little I can make out, you should include the output of zfs version on both hosts, because Ubuntu has a nasty habit of shipping mixed versions that break very strangely, and that will report both the kernel module and userland tooling versions.

[montanaviking]

Thanks rincebrain for your comment.
It's appearing that my problem was timeout in Netcat. I increased the -w to -w 20000 for both send and receive (-w is the timeout in sec) and the send -receive of snapshots appears to be working so far.
Thanks,
Phil

[rincebrain]

Why use a -w at all? That seems like asking for trouble.

[montanaviking]

Hi Rich, I used it based on an example since this was the first time I'd started using Netcat. It appears that leaving the -w option off would give us no timeout so that's what I'll do. Thanks, Phil

…

On Thu, Jun 22, 2023 at 1:33 PM Rich Ercolani ***@***.***> wrote: Why use a -w at all? That seems like asking for trouble. — Reply to this email directly, view it on GitHub <#15006 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5GXWW7GSQJWGHHAOFQCM3XMSMZFANCNFSM6AAAAAAZQMU6TE> . You are receiving this because you authored the thread.Message ID: ***@***.***>