Bug: file uploading fails when uploading many files (around 1.5 GB) "max parts reached"

[Benzeel]

Bug description

when uploading an obsurd amount of files to a folder (and maybe root) when finalizing, you get a status 400 error relating to numbers of parts reached being too many. i dont remember the wording verbatim.

How is Zipline being run?

On docker (docker, docker compose, etc.)

Zipline Version

4.4.2

If applicable, what browsers are you seeing this issue on?

Firefox based (Firefox, Zen Browser, Waterfox, etc.)

Relevant Logs

No response

Steps to Reproduce

upload ~ 1.5 gb of files around 2.8k to a folder with or without chunked uploading on

[Benzeel]

this still applies even with chunked uploads OFF.

video of this happening

[Benzeel]

note: this occurs whether you choose a subfolder or not.

[diced]

this still applies even with chunked uploads OFF.

video of this happening

weird.. when you do this can you have the dev tools open, then go to the network tab, and you should see a request to /api/upload (it might just say upload or whatever, or it will be red because it failed), click on it, and check the response

[Benzeel]

i get "payload too large" in devtools when its finalizing uploads. i think its not the file size, its the amount of files. when in the 1k files range, it seems to fail..

so ill go ahead and assume 1k files is the limit.

tried with chunked uploads both on and off.

[Benzeel]

maybe having a batched uploads option in the admin settings in the dash to upload files in batches (separate requests with a capped amount of files per request) would be a good idea?

[diced]

i get "payload too large"

not a zipline error, it's probably something with your reverse proxy.

[Benzeel]

update to this, i disabled forwarding https urls and logged in locally connected to the machine thru its local ip on the network

the upload just halts and gets stuck over 1000 files (im assuming)

then after a while retries, then gets stuck again

[Benzeel]

yeah, just confirmed, it doesnt get stuck under 1000 files

definitely see if you can reproduce on your end

[diced]

oh this is just a limit of fastify-multipart, which is what were using to handle multipart/form-data uploads, it just limits at 1000. i think the most sensible way to fix this is just introducing a config option that lets u bump that up