Can ublk zero copy be used with non-io_uring interfaces?

[Kamillaova]

Can ublk zero copy be used with non-io_uring interfaces? I mean when the read/write operation is handled not by the io_uring (FIXED) operation, which internally writes to the kernel buffer, instead, using RDMA, for example. Or memset or something like that.

[ming1]

It can't be used in non-io_uring interfaces.

However, it is possible to cover some basic/generic operations by adding io_uring bpf OPs and exported kfuncs, such as:

• memcpy/memset
• checksum
• compression
• ...

[ming1]

AFAIK the main reason should be big cost of runtime updating page table.

is it really that big?

For any page table update, every CPU has to be notified, and its TLB cache has to
be flushed. The whole procedure could be long compared from IO lifetime viewpoint.

[Kamillaova]

@ming1 am I right that there is no way to do full ZC between TCP sockets and ublk on the RX path?
Or something like vmsplice (but in io_uring) can be used for doing that?

[ming1]

@ming1 am I right that there is no way to do full ZC between TCP sockets and ublk on the RX path? Or something like vmsplice (but in io_uring) can be used for doing that?

I think ublk zc can be done in RX path, and I did implement it before on ublk/nbd.
When nbd_reply packet is received, the request handle is parsed, and you can know if it is one READ or WRITE request, and its tag & hw_queue, then you can setup ublk zc recv for receiving the READ request in zero copy style.

So it depends on if the application protocol allows to recognize the READ IO. If it is yes, it can be supported.

[Kamillaova]

I think ublk zc can be done in RX path, and I did implement it before on ublk/nbd.

I mean 100% ZC, even on RX from NIC

[ming1]

I think ublk zc can be done in RX path, and I did implement it before on ublk/nbd.

I mean 100% ZC, even on RX from NIC

I remember that buffer is pre-allocated from userspace for io_uring net rx zc, then 100% net rx zc can't be supported, because ublk io buffer is from the client application side.

Thanks,

[vitalif]

P.S: Did some more tests on an Intel CPU, and ublk linear read/write is indeed slower than vduse. Not sure why - vduse and ublk both copy memory :-). Iops-wise it's still much faster though.
VDUSE vs UBLK linear read: 4249 MB/s vs 3140 MB/s
VDUSE vs UBLK linear write: 5062 MB/s vs 3674 MB/s
No idea why ¯\_(ツ)_/¯

[ming1]

OK, I will try to reproduce and see if I can reproduce.

I guess it is one 4k random read/write.

Can you share both ublk and vduse setting and the fio script?

Thanks,