Maestral seems to ignore new files added to local storage - complex problem

[prentrodgers]

Is your feature request related to a problem? Please describe.
My setup is a bit complicated. I use the maestral docker image in a kubernetes pod targeting a filesystem on a Persistent Volume managed as a Persistent Volume Claim. This is maestral version 1.9.4. It successfully linked to Dropbox, and over an evening downloaded the requested directories that were not excluded. It successfully keeps track of changes on Dropbox to all the files in Dropbox. But if I add new files to the directory that is local to maestal, it doesn't upload new files to Dropbox. When I ask the filestatus of these files, maestral reports that they are unwatched.

maestral filestatus Dropbox/Uploads/ball9-t54u8_l47_r02_f2.50_c0176_sr1_p0_a1.00_w1.00_d03_46_t024.mp3
unwatched

This may be as designed, and I can work around it by uploading the unwatched files to a Dropbox managed file system on another system that has Dropbox installed. It's then noticed and updated.

Describe the solution you'd like
I would prefer if there were a way to have maestral notice that there are new files that need to be uploaded to Dropbox. Or if there were a command that would force it to notice newly added files in the local file system. I tried the maestral rebuild_index, but that didn't get the files to be discovered.

maestral rebuild-index
Rebuilding the index may take several minutes, depending on the size of your Dropbox. Any changes to local files will be synced once rebuilding has completed. If you
stop the daemon during the process, rebuilding will start again on the next launch. If the daemon is not currently running, a rebuild will be scheduled for the next
startup.

? Do you want to continue? Yes
✓ Rebuilding now. Run 'maestral status' to view progress.

Then I went for a walk and came back

bash-5.1$ maestral status

Account xxx.yyy@gmail.com (Pro)
Usage 6.8% of 2.2 TB used
Status Syncing ↓ 123/489
Sync errors 0
</code

Describe alternatives you've considered
I could copy them to a different server that runs Dropbox on linux.

Additional context
I have been searching for a long time for a solution that allows me to store a Dropbox directory in Kubernetes on a Persistent Volume using a Persistent Volume Claim, and maestral is the first solution that seems to be almost perfect for this. This is a really awesome program that does 99% of what I need. Perhaps I could cobble something together with the maestral API. I'd be happy to share the yaml file if there are others who would like to have it work in kubernetes.

[prentrodgers]

I noticed that the files that I just added to local filesystem from kubernetes were marked as root root:

ls -lth Dropbox/Uploads/ball9-*.mp3
-rw-r--r-- 1 root root 7.1M Apr 28 15:36 Dropbox/Uploads/ball9-t64u8.mp3

and the older ones that successfully synced with Dropbox are marked as dropbox dropbox:

-rw-r--r-- 1 dropbox dropbox 29.7M Sep 20 2023 Dropbox/Uploads/ball9-t37_1.25.mp3

I could change them to something other than root root. Perhaps that would solve the problem.

[prentrodgers]

I changed the ownership of the file and maestral picked it up:

chown dropbox:dropbox Dropbox/Uploads/ball9-t??u8*.mp3

It's no longer unwatched:

maestral filestatus Dropbox/Uploads/ball9-t54u8.mp3
up to date

I can handle that on my side.

[prentrodgers]

I have some new problems that have cropped up in the last few days. Now none of my files created in the file system local to maestral are getting synced to Dropbox. I had assumed that it was because they were owned as user: root group: root. But many that are marked as dropbox and dropbox for user and group are also failing to get uploaded. I wait for a while and maestral status says up to date, but they never get to Dropbox and thus never to my dropbox clients on Windows/Linux.

But if I chown them to dropbox explicitly, even though they are already in user and group dropbox and dropbox, maestral suddenly uploads them. These are all files that existed previously on the client dropbox directory. I explicitly delete them in the maestral pod before recreating them. That may be part of the problem. Maestral may not detect that they don't exist for the fraction of a second that they were deleted. I did that as a work-around for the problem of Maestral creating a conflicted copy of all of them. This comes because of the behavior of maestral thinking that any file created in local storage in its environment that matches the name in Dropbox, but has different content, must be a conflicted copy at acts accordingly.

I guess my work-around wasn't far enough around. I'll have to touch the files in the maestral environment after I've finished creating them.

These small files are new from maestral's perspective because they were deleted off of Dropbox before I created them with the same name in the maestral environment. I'm not saying this needs to be addressed in your code. Perhaps my use case with lots of small files being recreated several times a day in the maestral environment is unique. Most people create them on client machines and those get handled by Maestral perfectly. I thought I was being clever to create them in the maestral environment directly, since that's where they are accessed by my jupyter notebooks in kubernetes on a persistent volume claim. But they don't get up to the clients until I do the chown command. I could also use touch. That works also, without any reference to ownership.

[omichalek]

I installed Maestral from pip, inside WSL, and it claims my locally created files are unwatched as well. I think it has only downloaded files so far.

~$ cat /home/user/.config/maestral/maestral.ini
[auth]
account_id...
token_access_type = offline

[app]
notification_level = 15
log_level = 20
update_notification_interval = 604800
bandwidth_limit_up = 0.0
bandwidth_limit_down = 0.0
max_parallel_uploads = 6
max_parallel_downloads = 6

[sync]
path = /mnt/c/Users/user/Maestral
excluded_items = []
max_cpu_percent = 20.0
keep_history = 604800
upload = True
download = True

[main]
version = 20.0

~$ maestral resume
✓ Syncing resumed.


~$ maestral status
Account      ...@gmail.com (Basic)
Usage        98.0% of 17.7 GB used
Status       Up to date
Sync errors  0

~$ maestral filestatus /mnt/c/Users/user/Maestral/SHARE/DA
up to date

~$ maestral filestatus /mnt/c/Users/user/Maestral/SHARE/DA/250512
unwatched

But it should be watched! This is the locally (in Windows) created folder that does not get uploaded. It has same permissions as its parent:

~$ ls -al /mnt/c/Users/user/Maestral/SHARE/DA
total 10496
drwxrwxrwx 1 user user    4096 May 12 10:18  .
drwxrwxrwx 1 user user    4096 May 12 10:18  250512