Add note to README to improve read speed.

[the2nd]

You can improve read speed of a fuse filesystem by setting readahead to e.g. 10240:

1. Get bdi ID:
   cat /proc/self/mountinfo | grep /mnt

2. Set readahead (if bdi id is 0:107):
   echo 10240 > /sys/class/bdi/0:107/read_ahead_kb

Maybe it worth to add a note for this to the readme.

[mxmlnkn]

Sorry for the late answer, but did you test how much of a difference this makes?

[the2nd]

Hi,

this is the output of a test with OTPmeFS running on two VMs (Proxmox VE).

root@otpme-dev:/opt/otpme/otpme# dd if=/dev/urandom of=/mnt/test.img bs=1M count=100 status=progress
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.51595 s, 203 MB/s

root@otpme-dev:/opt/otpme/otpme# dd if=/mnt/test.img of=/dev/null bs=1M status=progress
99614720 bytes (100 MB, 95 MiB) copied, 2 s, 48.8 MB/s
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.0978 s, 50.0 MB/s

root@otpme-dev:/opt/otpme/otpme# grep /mnt /proc/self/mountinfo
209 28 0:107 / /mnt rw,nosuid,nodev,relatime shared:156 - fuse OTPmeFS:/management rw,user_id=0,group_id=0

root@otpme-dev:/opt/otpme/otpme# echo 10240 > /sys/class/bdi/0\:107/read_ahead_kb 

root@otpme-dev:/opt/otpme/otpme# dd if=/mnt/test.img of=/dev/null bs=1M status=progress
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.133065 s, 788 MB/s

OTPmeFS is a fuse network filesystem implemented for OTPme. It uses SSL sockets for network communication.

The client part can be found here: https://github.com/the2nd/otpme/blob/main/otpme/lib/fuse.py
The server part can be found here: https://github.com/the2nd/otpme/blob/main/otpme/lib/protocols/server/fs1.py

Using the loopback.py from the example directory shows a speedup of about 50-100M/s so i guess the biggest benefit get network filesystems because of the network delay.

[the2nd]

Same for sshfs:

root@otpme-dev:/opt/otpme/otpme# sshfs otpme-dev2:/tmp /mnt/

root@otpme-dev:/opt/otpme/otpme# dd if=/dev/urandom of=/mnt/test.img bs=1M count=300 status=progress
300+0 records in
300+0 records out
314572800 bytes (315 MB, 300 MiB) copied, 0.978351 s, 322 MB/s

root@otpme-dev:/opt/otpme/otpme# dd if=/mnt/test.img of=/dev/null bs=1M status=progress
300+0 records in
300+0 records out
314572800 bytes (315 MB, 300 MiB) copied, 0.563891 s, 558 MB/s

root@otpme-dev:/opt/otpme/otpme# grep /mnt /proc/self/mountinfo 
209 28 0:107 / /mnt rw,nosuid,nodev,relatime shared:156 - fuse.sshfs otpme-dev2:/tmp rw,user_id=0,group_id=0

root@otpme-dev:/opt/otpme/otpme# echo 10240 > /sys/class/bdi/0\:107/read_ahead_kb 

root@otpme-dev:/opt/otpme/otpme# dd if=/mnt/test.img of=/dev/null bs=1M status=progress
300+0 records in
300+0 records out
314572800 bytes (315 MB, 300 MiB) copied, 0.409776 s, 768 MB/s

[mxmlnkn]

Wow, that looks almost too good to be true. I have to benchmark this with ratarmount. libfuse has a max_readhead member in the fuse_conn_info struct. It should be possible to automatically configure this inside the mfusepy.Fuse.init_with_config callback. That should be more user-friendly than instructing the user to adjust it via /sys/class/....

[the2nd]

Wow, that looks almost too good to be true. I have to benchmark this with ratarmount. libfuse has a max_readhead member in the fuse_conn_info struct. It should be possible to automatically configure this inside the mfusepy.Fuse.init_with_config callback. That should be more user-friendly than instructing the user to adjust it via /sys/class/....

Can you tell me how to implement it with the init_with_config() callback? Or does this need changes on mfusepy?

[mxmlnkn]

It does not need changes in mfusepy, but it probably does not work as I expected it to because of this and this.

---

Just as in this example, but instead of nullpath_ok, you set max_readahead:

import mfusepy as fuse

class MyFuse(fuse.Operations):
    @fuse.overrides(fuse.Operations)
    def init_with_config(self, conn_info: Optional[fuse.fuse_conn_info], config_3: Optional[fuse.fuse_config]) -> None:
        # This only works for FUSE 3 while the flag_nullpath_ok and flag_nopath class members work for FUSE 2 and 3!
        if conn_info:
            conn_info.max_readahead = 10 * 1024 * 1024

    ...

I am unsure what exactly that value is. I am testing with FUSE_LIBRARY_NAME=fuse3 python3 examples/memory_nullpath.py mounted and by writing and reading a 1 MiB file with cat /dev/urandom | head -c $(( 1024 * 1024 )) > mounted/bar; cat mounted/bar | wc -c and printing out the parameters that the FUSE read method is called with:

• max_readahead = 1,2,3,4,8, ..., 4095: read gets called with size=4096 (4 KiB)
• max_readahead = 4096, 4097, 7*1024, 64*1024, 65*1024, 67*1024,68*1024-1: read gets called with size=131072 (128 KiB)
• max_readahead = 68*1024, 69*1024, 70*1024, 71*1024: read gets called with size=139264 (136 KiB) 7 times and with a remainder of size=73728
• max_readahead = 72*1024, 74*1024, 75*1024: read gets called with size=147456 (144 KiB) 7 times and with a remainder of size=16384
• max_readahead = 76*1024: read gets called with size=155648 (152 KiB) 6 times and with a remainder of size=114688
• max_readahead = 127*1024: read gets called with size=253952 (246 KiB) 5 times and with a remainder of size=32768
• max_readahead = 128*1024, 256*1024, 1024*1024: read gets called with size=262144 (256 KiB) 4 times
• Without specifying anything, the reads are in 256 KiB.

This seems somewhat random, but some thoughts:

• First off, max_readahead is not max_read, but it still seems to have some influence on what chunk size read is called with.
• The minimum granularity is 4 KiB, but after 8 KiB, it can only be configured with 8 KiB granularity for some reason.
• The specified max_readahead multiplied by two seems to specify the read chunk size ... Something seems to be wrong.
• I cannot explain the maximum of 256 KiB. A maximum of 128 KiB would have made sense as explained in the manual

  max_read=N
  With this option the maximum size of read operations can be set. The default is infinite. Note that the size of read requests is limited anyway to 32 pages (which is 128kbyte on i386).
  max_readahead=N
  Set the maximum number of bytes to read-ahead. The default is determined by the kernel. On linux-2.6.22 or earlier it's 131072 (128kbytes)

This comment also seems to apply somewhat. It might be that the max_readahead option inside the fuse_config can only reduce the upper limit. Increasing it to 10 MiB as you suggested would then still require your approach with the /sys filesystem.

Running with FUSE( ..., debug=True) and setting max_readahead = 33 * 1024 yields this output:

FUSE library version: 3.14.0
nullpath_ok: 0
Ignoring invalid max threads value 4294967295 > max (100000).
unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0
INIT: 7.42
flags=0x73fffffb
max_readahead=0x00020000
   INIT: 7.31
   flags=0x4040f039
   max_readahead=0x0001ec00
   max_write=0x00100000
   max_background=0
   congestion_threshold=0
   time_gran=1
   unique: 2, success, outsize: 80

The max_readahead=0x0001ec00 is 123 * 1024, so at least there seems to be nothing wrong with the mfusepy translation layer.