Improve Dependency Downloads for Users in China and with Restricted Network Access

[cary-rowen]

Hi Carter,

Following up on our email conversation, I'm opening this issue to discuss and resolve the dependency download problem for users with restricted network access. This is particularly relevant to my work on the VIVO provider in PR #66.

The Problem

A number of users, especially in China, face issues when the add-on attempts to download its dependencies from GitHub. The experience is often poor:

1. For some, the connection to GitHub is blocked entirely, leading to a quick failure.
2. For others, the connection is extremely slow, causing the download to hang for a long time before eventually timing out.

Waiting for a long timeout on a slow connection only to fall back to another URL isn't an ideal user experience.

A Potential Solution: Using a Mirror

To address this, the NVDA Chinese Community operates a mirror service that is highly available for users in the region.

Currently, this mirror is primarily used for hosting .nvda-addon files. However, we can add the required dependency .zip files (e.g., aic-py-3.11.zip) to our download whitelist, making them available through our service.

This would enable a more robust download strategy:

1. The add-on could first attempt to download from GitHub.
2. If the download fails or times out, it could automatically fall back to the NVDA-CN mirror URL.

Next Steps

This proposal is to make our mirror available for this purpose. If you're open to this approach, we can provide the stable mirror URLs for the dependency files.

Let's discuss if this is a viable path forward. I'm happy to provide any further details needed about the mirror service.

[cartertemm]

Hello @cary-rowen,

Thanks for your hard work toward making this add-on functional for the Chinese NVDA community. This is a huge help, and I'm grateful for the offer as I truthfully wouldn't know where to start.

The libs are contained within a single release (you can find it here).

Whenever there is a major security issue and/or we need another module, I repackage the zip files and replace the asset here. It isn't the greatest system in the world, but it's worked up to this point.

The upside is that you only have to look in one place for our dependencies, the downside is that there isn't a reliable way to detect when they have changed short of caching checksums.

If you'd like to mirror aic-py-3.11.zip and aic-py-3.7.zip, feel free. Once I have URLs, I will update the add-on to display a dialog when GitHub can't be contacted or times out. I'm thinking this would be a yes/no prompt asking the user whether they would like to download from a third-party mirror.

[cary-rowen]

Hi @cartertemm,

Here are the mirror URLs from the Chinese community.

Going forward, you don't need to notify me in advance when you update the binary files. I have configured our mirror to fully whitelist the AI-content-describer repository, which means all files under it are automatically passed through and made available.

https://github.mirror.nvdadr.com/cartertemm/AI-content-describer/releases/download/libs-release/aic-py-3.11.zip

https://github.mirror.nvdadr.com/cartertemm/AI-content-describer/releases/download/libs-release/aic-py-3.7.zip

Thank you for accepting this proposal

Cary,
Best

[cartertemm]

Brilliant idea to just proxy the request, that takes a lot of work off my plate so thank you!

As of this recent commit, the mirror should work for those in China or that are otherwise behind a restrictive firewall.

It functions properly on my side when I change the URL to something non-existent, thereby triggering a urllib.error. Would you be able to verify this on your side before I go ahead and publish the new release? I'm 99% sure this does what it should, but no harm in certainty. :)

[cartertemm]

@cary-rowen

Were you able to verify whether this works on your end? It's downloading no problem on my machine, for what it's worth.

It'd be cool to drop this in the next release.

[cary-rowen]

I'm sorry, I actually ran a test last time but I seem to have forgotten to comment or there was some other confusion and I did find some issues.

[cary-rowen]

I've noticed that sometimes GitHub is accessible in China, but downloading resources is extremely slow. In such cases, the plugin won't throw an error, so the mirror cannot be used, and users are left with no choice but to wait for extended periods.

What's even more frustrating is that it might fail while waiting for the download to reach 90%.

Do you have any suggestions for this situation? For example, the dialog for downloading dependencies has a button to switch download channels.

[cartertemm]

If it fails, the user should be presented with the alternative download option, right?
If not, that's definitely something I'd want to take a look at.

I could place a "download from a mirror" or similar button in the dependency dialog as a last resort, though I don't know how many users would know to click it. Preferably there would be a way to detect whether it was going to fail without having to wait an indeterminant amount of time.

How do other applications solve this problem, other than using a platform that isn't Github?

[cary-rowen]

If it fails, the user should be presented with the alternative download option, right?
Yes, this works as expected.

I could place a "download from a mirror" or similar button in the dependency dialog as a last resort, though I don't know how many users would know to click it. Preferably there would be a way to detect whether it was going to fail without having to wait an indeterminant amount of time.

I agree, is tracking download speed a feasible solution?

How do other applications solve this problem, other than using a platform that isn't Github?

For NVDA users in China, people generally use the NVDA Chinese community's mirror server to connect to the add-on store.
Perhaps we can detect whether the user is using NVDA's update mirror or add-onStore mirror, so as to automatically use the mirror to download dependencies?

Btw, Is it possible to include dependencies in add-ons in the future?

[cartertemm]

Is it possible to include dependencies in add-ons in the future?

This is what we did originally. However, a few of our dependencies take the form of PYD files which cannot be imported on releases of Python other than the one they were built on. NVDA runs on 3.7, 3.11, and most recently 3.13 (depending on version). bundling three (and counting) unique sets of dependencies with the add-on file would make it an extremely large download, 90MB at least. The second NVDA switches over to Python 3.15, that grows to 120MB, and so on unless we decide to drop support. I'd rather save deprecating older versions for the day that there is a technical limitation leaving us no other choice.

is tracking download speed a feasible solution?

It's possible, but I don't want people in other regions on hotspots or really slow connections to have to switch over to the mirror which likely wouldn't be faster for them.

Perhaps we can detect whether the user is using NVDA's update mirror or add-onStore mirror, so as to automatically use the mirror to download dependencies?

This sounds like our best bet. NVDA exposes config.conf["update"]["serverURL"] and config.conf["addonStore"]["baseServerURL"] for the update and store mirrors (respectively).
If one or both of these differ from the default, we could pop up a yes/no dialog: "We noticed you are using a mirror to download add-ons and/or new NVDA versions. Would you like to use a community-maintained mirror to download the dependencies for this add-on?"

Could this work?

[cary-rowen]

If one or both of these differ from the default, we could pop up a yes/no dialog: "We noticed you are using a mirror to download add-ons and/or new NVDA versions. Would you like to use a community-maintained mirror to download the dependencies for this add-on?"

Sounds good. On this basis, if the user does not use any of the above mirror services, we can also ask the user whether to try using a mirror when the download fails due to connection problems.

Thanks for your work.