Slow performance in folder with a lot of files?

[JeffP07]

I was setting up mergerfs for someone on their NAS, the way they have it set up, they have a folder where they save files from a CAD program and over the years they have accumulated over 5000 files in a single folder. This folder is served through samba. Serving the drive directly through samba has no issues.

However, we found that trying to use mergerfs, whenever they try to save a new file from their CAD program, it would take a considerable amount of time, the save file dialog would just sit there for anywhere from 15-30 seconds before saving. Using strace, I found that it was calling newfstatat on every file in the directory and that is adding a ton of time to the save operation.

Is there any options I can enable/disable/change to try and speed this up? I've tried disabling security_capability, xattr=noattr and nosys, and enabling/disabling the various caches.

I'm on mergerfs 2.40.2, ubuntu 22.04 arm64

[trapexit]

You strace'd what? mergerfs or the app? Can you please provide details as requested in the support section of the docs?

[JeffP07]

Since removing mergerfs and having samba point directly to a drive doesn't have the same slowness, does that mean it might just be the overhead from mergerfs compounding over so many files being the issue then? and the real underlying issue is just samba making all the calls

[trapexit]

I don't know precisely what the app is doing or what samba is doing so I really can't make an informed claim. There are a lot of moving parts here. Have you done a baseline comparison?

Using a tmpfs mount: ls -1f vs ls -lh, mergerfs settings cache.files=off,cache.readdir=true,cache.entry=60,cache.attr=60,xattr=nosys

• tmpfs: 0.03s; 0.17s
• mergerfs - tmpfs: 0.03s; 0.7s not cached, 0.13s cached
• samba - tmpfs: 0.48s; 16.8s
• samba - mergerfs - tmpfs: 1.5s uncached, 0.48s cached; 1.84s uncached, 0.6s cached

As you can see it's complicated. When the kernel fully caches the dir contents and attr and entry values mergerfs isn't even hit.

[JeffP07]

Had time to revisit this a bit.. so I installed the Windows process monitor to look at what calls the CAD program is making, and noticed it was making like 15 or so QueryDirectory calls that were taking ~1.3-1.7 seconds each when going through Samba + mergerfs, and when taking mergerfs out of the equation the same calls were only taking ~0.01-0.03s.

Then on the NAS itself I tried timing ls without samba at all and just straight to the drive vs going through mergerfs. With mergerfs took about 1.6s without cache and without mergerfs took about 0.01s.

So then I tried increasing the cache time on mergerfs and that does greatly reduce the slowness of saving files from the CAD software. Seems that there were just a lot more calls than I thought so it was taking so long that it was hitting non-cached calls multiple times during a single save operation.

Regardless, there still seems to be considerable slowness in mergerfs when there's a lot of files, but it may just be as I was thinking earlier and it is just the bit of overhead multiplied by thousands of files causing it

[trapexit]

Need to know exactly what calls are made to comment but I find it very unlikely that Samba is getting thousands of getattr calls and it is so fast over the network. Something is almost certainly caching somewhere that for some reason doesn't with mergerfs added.

Think of the full round trip of a single getattr call, uncached:

client app -> windows kernel -> network -> samba -> linux kernel -> mergerfs -> linux kernel -> underlying filesystem -> mergerfs -> samba -> network -> windows kernel -> client app. Doing that thousands of times adds up.

[JeffP07]

I'm not sure if this helps at all but I did a few more tests with Process Monitor. I restarted the NAS between runs to try and get fresh first run times.

Here are a couple images of Process Monitor filtering to the CAD software during a save operation. The left side is with mergerfs, the right side is without. You can see that using mergerfs increases the time on some of the calls by several orders of magnitude. The first image shows when I first click "save as" and the save dialog is populating the files in the folder. The second image shows a sample of calls after I type a filename into the save dialog and click save. Clicking save waits on all those calls before it writes the file, which locks up the save dialog for the entirety of the time.

The left half is using mergerfs with this fstab where I don't use the cache to exacerbate the issue:

UUID=b73fb75b-43f9-425a-ba6d-3bc1717993b3   /mnt/nvme0n1    auto    defaults,noatime,nofail     0   0
/mnt/nvme*    /srv/samba/share    mergerfs    allow_other,noatime,moveonenospc=true,security_capability=false,xattr=nosys,category.create=mfs    0  0

The right half bypasses mergerfs with mounting the drive directly to the mount point:
UUID=b73fb75b-43f9-425a-ba6d-3bc1717993b3 /srv/samba/share auto defaults,noatime,nofail 0 0