Download from portal not possible on Windows

[fedorov]

https://discourse.canceridc.dev/t/direct-download-from-the-portal-does-not-work/761

@s-paquette in https://nciimagingdat-m8a6349.slack.com/archives/C045VQUAZ0S/p1761345921749809:

confirmed the issue is a path length issue; if a user's max folder path length isn't incremented sufficiently the error is rather opaque. I'm going to put in a solution to detect it, but in this specific case, I suspect anyone on a Mac is unlikely to run into this because IIRC OSX's default path length max is generous (like 1024 characters or similar) but in Windows it's 260.
6:45
And the full path length of these files (collection+case+study+series+instace) is around 270+.
6:46
(I tried on 2 Windows 11 machines and also encountered the error; I suspect Steve is on Linux or OSX.)

@pieper:

Since we don't know the absolute path to the download folder (it's considered by the browser to be confidential, so it's not exposed to the web page code) there's no way for us to know in advance if any given suffix path we add will exceed the system's path limit. What we can and should do is check that the file was downloaded and saved correctly by re-reading it and verifying the checksum against the data in the bucket. This can be streamed so no need to hold the full file in memory and it is very fast compared to the download so virtually no overhead. If the hashes don't match we can let the user know and suggest that it might be the file path length.

You can get the checksum for objects in s3 automatically (from s3) and then compare them with what you calculate locally.

[s-paquette]

@pieper The issue is we can't even begin the download, so there's nothing to verify. The failure happens when the code goes to generate the file handle, resulting in a NotFoundError. so the good news is we can catch that and respond to it.

it's true we can't know the true, full path (because all that's available to the browser is the path we're initializing) but if that path is close to 256 characters we can already know we're going to have a problem.

the real question is how to fix the inability to save the files, and that can really only be fixed in two ways: we shorten the path, or, we concatenate/zip the downloads.

[fedorov]

we shorten the path

For that we can fall back to save all files as a flat list in the selected directory using UUIDs for names. Or we can be even more aggressive, and just number them consecutively: 0.dcm, 1.dcm, ... But I am not sure switching to the fallback is user-friendly. Maybe instead we can have a setting, or a popup, where we offer to save as hierarchical or flat, and hierarchical fails, we detect it, and suggest the user tries saving again using flat namespace?

[pieper]

Detecting right away when the file handle can't be created is even better, and gives us the chance to give better user feedback. It's an unfortunate feature of windows that we can't even suggest the idc-index as a fallback, since the same path length issue would come up there. Another option is to zip the files before download, but that also would fail at the unzip stage (and the download would be slower since it would not be overlapped).

I guess the flat fallback is the best option. We can't really solve their windows problem but we can at least give them the data.

[s-paquette]

@pieper Correct, the zip option still has this issue. Although Windows 11 can in most cases allow access into the zipfile from a lot of modern programs and apps, some might still have issues with it. So flattening the names is the best route.

@fedorov We can absolutely do this; at a minimum, we can still place the files under the collection (and case) folders, we just have to cut out series and study. Or we can let them choose? Any of this is doable, really.

[fedorov]

Per discussion today, let's implement automatic fallback when we detect that saving failed to save all files into a flat list, warning user first.