low performance in sync=always （iops throughput）

[unkownerror1]

zfs version 2.1.5
the figure is show my zfs config

I think “sync =always ，logbias =throughput“ is the reason cause my low zfs performance

and for example we have made a stress test on 8 SSDs directly without going through zfs by using Vdbench（power workload） ， the outcome shows 10GB/s throughput

but with zfs the outcome shows 800MB/s

could someone tell me the reason why the outcome show such low performance？

[rkelleyrtp]

How did you assemble your pool? Please post "zpool status -v".

[unkownerror1]

[rkelleyrtp]

I am not a ZFS expert, but I think the way your pool was created is the issue. Essentially, you put all drives in the same vDev, and I believe, this gives you the performance of the slowest drive in the vDev. Check out this URL for a deep dive into creating multiple vdevs from single drives in your pool. That will probably provide the fastest performance.

[rincebrain]

That's a pool of 10 vdevs - 8 data disks, one special vdev, one log vdev, not all in one, unless that output is truncated in a way I don't see.

That said, sync=always is, I believe, basically going to force it all onto the single log vdev first, so that's going to be your bottleneck pretty quickly.

[rincebrain]

As I indirectly alluded to above, I think sync=always is your problem, since that means, I believe, all of your IO is going to the single log vdev first, and that's probably going to bottleneck no matter how nice a device it is.

I'm a little entertained by the combination of sync=always and logbias=throughput, since that seems like it's ambiguous which might win in that combination of settings, and my guess would be sync=always wins from your experience.

What are you trying to accomplish by setting those two?

[unkownerror1]

The sync=always is set to ensure data security and prevent data loss caused by power off，the logbias=throughput is set to get the maximum bandwidth. logbias=throughput will write user data to the data vdev, and then write the block pointer to the log vdev.

I have tried to remove the log vdev, but the performance has not improved.

[ryao]

Try sync=standard. If the application on top is properly written, you will not have a problem no matter how many power losses happen since it will fsync data as needed rather than on every IO operation.

[amotin]

What SSDs are you using? Not all SSDs are equal on synchronous write latency. And with logbias=throughput you depend on the latency of all SSDs, you likely don't want it. And unless your SLOG is dramatically better, you may not need it either. Check the huge collection of people experience on this topic: https://www.truenas.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/ .

[ryao]

When you are doing Vdbench, the benchmark is not being done with the equivalent of sync=always, which is extremely expensive and probably has no equivalent in vdbench.