Mountain Duck - Smart sync + Indexing - Endless disk operations / 100% busy

[lowtechthai]

Hello.

I have a long lasting issue with Mountain Duck. I'm connected to Backblaze B2 (which is basically same API as S3). I have about 45000 files in a thousands directories on B2.

First, I thought that this is not the issue, it just need to index files. Then, after a few months of intensive scans/reads/writes - it has rendered my SSD unusable, I had to to a secure erase then full system reinstall.

Next, I moved the cache to an external HDD, which is relatively fast as well. And I see Mduck - eating 100% of disk time most of the day time, slowing down everything. After a brief examination I've found that it has created 45000 almost empty files and 10000 directories - AND its only scanning the structure most of the time. In status Mduck showing its indexing files, however no details. After 3 months of such heavy usage it still not finished indexing. The only solution which helped is to delete all cache and switch mduch to online mode, which is slow and not much usable for me.

I tried to make cache folder compressed - no help. Object properties showing 45000 files with total size something 300MB, while disk space occupied about 450M.

So its clear, Mduck uses ineffective cache and indexing and unusable in production environment with a number of files.

Can anybody please suggest a fix for such a behaviour?

Thank you

[dkocher]

With B2 and other object storage with no notion of folders and no possibility to query for a possible change in a recursive directory structure by comparing a checksum (like a ETag response header) or modification date, the Index Files option will repeatedly query for changes of previously opened folders. I suggest that you disable the option Preferences → Sync → Index Files. You will still get notified for changes in folders currently opened in Finder on macOS or Windows Explorer.

[lowtechthai]

Thank you for details. However, nothing was mentioned about the issue in your docs and FAQ's. The thing rendered my SSD unusable.

The issue is NOT the lack of possible change query, but very very bad storage system with cache. What's the need to recreate the same structure with million of empty files? The system perform scan 99% of the time, digging onto directories and thousands files. The small compact database will do the job perfectly without damaging disks and getting around 100% of disk time.

This problem should be definitely sent to your developers, as this is a huge issue.

[dkocher]

Placeholder files should not be repeatedly being created on disk.