Unable to download a ZIM file

[kelson42]

Wanted to download a ZIM file which (from the UI) seems available

But when I try to click on the "Download ZIM". this is what I get:

Either the ZIM file should be available, or we should request to regenerate the ZIM. In worse case we should display a user-friendly error

[audiodude]

We hide the "download ZIM" link after 14 days, which is calculated by the WP1 server. This is based on the assumption that the fie lives in the Zimfarm S3 storage for that long. If the ZIM file is deleted by the Zimfarm for any reason before that, you will likely see this message.

We could alternately do a "background poll" of the ZIM file, maybe with a HEAD request to see if it is 404, and use that to decide whether to show the download link?

Alternately, this could have been caused by an error in Zimfarm not being reported as such to WP1 (and the file never being created).

[arnchlmcodes]

I think the issue is that WP1 assumes the ZIM file exists for 14 days, but sometimes it gets deleted earlier, so the download link still shows and users end up hitting an S3 NoSuchKey error.

My idea was to verify the file exists before redirecting (maybe via a HEAD request in the /zim/latest backend endpoint) so we can return a proper error and hide the download option instead of sending users to a broken link.

@audiodude does this sound like the right direction? Happy to implement it and open a PR if so.

[audiodude]

Yes I think that's right! You can go ahead an make a PR. I think the HEAD XHR should be done directly on the client if possible, and we can do away with the server side "timeout" altogether.

[arnchlmcodes]

@audiodude while I was working on this issue I added a function to fire a head request while the dedicated ZIM page loads then I realised MyLists.vue also has download links in the dashboard table, so should I do this check for all buttons in dashboard is that even feasible? or should i just fix the dedicated page as of now

[arnchlmcodes]

Also was your initial intent to do this check for all download links in the dashboard, Rereading the issue I realise it says nothing about doing this for the dedicated donwload page, I just assumed thats where to start

[audiodude]

Yes the main issue is the dashboard.

You don't need to "pre-check" all of the links. Instead, when the user clicks Download ZIM you can do the check then, and surface an error if the file is unavailable. This is preferable to showing a cryptic S3 XML page.

[arnchlmcodes]

Hi @audiodude, I've been working on this issue following your guidance in the comments. My initial idea was also to verify the file exists in the backend endpoint before redirecting, similar to #1090. However you mentioned the HEAD request should be done directly on the client if possible, and that we could do away with the server side timeout altogether, so I changed my approach accordingly and implemented it client side in both ZimFile.vue and MyLists.vue, intercepting the click and firing a HEAD request directly from the browser to the storage URL before redirecting.

I noticed #1090 ended up taking the backend approach instead. I just wanted to flag that my PR follows the client side direction you told me to try in this thread. Happy to keep it open for review or close it in favour of #1090 ,whatever you prefer.

[audiodude]

Yes, looking at #1090, it seems complete, correct and simple. Although it would be preferable to do the HEAD check from the user's browser, this isn't completely straightforward, because what I didn't realize is that the item.z_url field is not the URL of the file in wasabi storage, but actually the "local" WP1 API url that redirects to wasabi. So a client-only approach would involve re-architecting the data that is sent to the frontend to include the "real"/wasabi URL (picture something like item.z_storage_url or something).

I will comment here for context and link from the proposed PR, but as written #1093 doesn't work. Since the item.z_url is just a WP1 API URL, it will always return a 302 status (at current HEAD), instructing the browser to redirect to the actual wasabi URL. Therefore checking the response value of this item.z_url does not tell us anything about whether the ZIM file itself, which is at the URL returned in the Location: header of the WP1 API endpoint (item.z_url), is actually valid or present.

[Best2Two]

Thank you very much for the discussion @audiodude @arnchlmcodes I have read it and I have few comments.

@arnchlmcodes I saw your approach
But I would assume that we can't do it in the client side without re-architecturing because this will create kind of violation of data integrity. If applied only on frontend this will be like bypassing backend code where we check for the deletion of zim using is zim deleted function and then based on the the return this will update the status of the file. So doing it in the front end using HEAD request and timestamp and in the back end using ony timestamp will creata an illusion for the user which will provide a nice ux but this will affect other components because the status of the file in the back end is not deleted yet! So we need the HEAD request in the back end also for that reason.

So the simplest approach that works for now is that I submitted in #1090 (it also has this issue I just said but it's way simpler) We need to check is zim deleted using HEAD without making network overhead issues so this will also update the status of the file in the backend so check status functions can see it's current state but again without making network overhead. That's why i have created the follow up issue #1092 as @audiodude requested to discuss the our new re-architecturing.

Current valid approaches will always have an inconsistency in the ux as the HEAD requrst will occur after clicking the download zim that shouldn't appear from the beginning at all.

I am very interested in contributing to GSoC also for the "WP1 Combinator Builder" But I would suggest if there are another core refactoring for other component to add them and work on them all as an idea for GSoC, but I want to ask you first if this is valid suggestion or if there i another componets to refactor @audiodude .

[arnchlmcodes]

heyy @Best2Two, Just to clarify is_zim_file_deleted is a read only function that calculates whether the file is expired based on a timestamp, it doesn't actually update any status in the backend database. So a client-side HEAD check wouldn't bypass or conflict with any backend state, the two would just be checking different things independently. but yes architecture wise this might not be optimal, that said I do agree the broader re-architecture in #1099 is the right long term solution.