ZPOOL get limited speed when mount through network (both samba and NFS)

[Fmajor]

• I have host NAS and CLIENT connected by 10g Network.
  • I have the network properly set (e.g., mtu) and can get 10Gbps (tested by iperf3)
• I use these command to test read speed
  • fio command: F=./fio.dat J=1 BS=1m S=16G RW=read T=30 && fio -numjobs=${J} -filename=${F} -bs=${BS} -group_reporting -rw=${RW} -name=${BS}-${S}-${J} -runtime=${T} -size=${S}
• In NAS, I have a zfs zpool
  • in raidz2 made by 8 hard disks
  • zfs_arc_max is set to 4Gib to disable cache when I try to test pool speed using fio command (fio test file is 16GB)
  • in /zpool, using fio command, I get ~900M/s read speed
  • I also make a ramdisk in /ramdisk, and get ~10G M/S read speed
• In CLIENT, I mount NAS/zpool and NAS/ramdisk into /NAS/zpool and /NAS/ramdisk
  • in /NAS/ramdisk, fio give ~1100M/s read speed, so the samba is able to transfer data at network full speed
  • in /NAS/zpool
    • I expect fio to give ~900M/s, but I can only get ~300M/s
    • If I set zfs_arc_max (in NAS) to >16GB, then the zpool cache will start to work, and the fio could give ~1100M/s (which again confirm that samba is not the speed bottleneck)
    • If I set zfs_arc_max (in NAS) back to 4GB, and the fio result go down again to ~300M/s
  • I try to mount using NFS, get ~300M/s for /NAS/zpool and ~1100M/s for /NAS/ramdisk

Question:
I expect to get ~900M/s in CLIENT in /NAS/zpool, but get ~300M/s, with both NFS and samba, what could be the reason?
I will set a larger zfs_arc_max to improve performance, but I want to figure out why zpool over samba/NFS gets much lower speed (compared to fio result directly in NAS for /zpool)

More infos:

• when do fio test, the CPU of NAS is not full, the network flow is also ~300M/s (from bmon)

[Fmajor]

More info:

• write speed is as expected, only read is slow