Intermittent Google Drive rate-limit errors when running download_pg_data.py

[melek2]

Hi all,
I wanted to flag an issue I hit last week while running:
python download_pg_data.py
Most files downloaded fine, but one of the larger Google Drive files (onshorewind_rev_profiles_20240801.parquet) repeatedly failed with a gdown rate-limit error (“too many users have viewed or downloaded this file recently”; full error below). Trying again today worked.
It might be helpful for download_pg_data.py to skip files that already exist locally, and it may also be worth considering a more stable hosting option for the multi-GB files since Google Drive rate limits seem to hit fairly often.
Really appreciate all the amazing work that has gone into making this repo!! just sharing this in case it’s helpful. Happy to test any updates.

File ".../miniconda3/envs/switch-pg-reeds/lib/python3.10/site-packages/gdown/download.py", line 53, in get_url_from_gdriv e_confirmation raise FileURLRetrievalError(error) gdown.exceptions.FileURLRetrievalError: Too many users have viewed or downloaded this file recently. Please try accessing the file a gain later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator. During handling of the above exception, another exception occurred: Traceback (most recent call last): File ".../Switch-USA-PG-ReEDS/download_pg_data.py", line 105, in <module> main(filter=sys.argv[1:]) File ".../Switch-USA-PG-ReEDS/download_pg_data.py", line 81, in main filename = gdown.download(url, fuzzy=True, output=dest) File ".../miniconda3/envs/switch-pg-reeds/lib/python3.10/site-packages/gdown/download.py", line 278, in download raise FileURLRetrievalError(message) gdown.exceptions.FileURLRetrievalError: Failed to retrieve file url: Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator. You may still be able to access the file from the browser: https://drive.google.com/uc?id=1bSje2czbF6GMsIYbxLkNYRkeiS6eshon but Gdown can't. Please check connections and permissions.

[mfripp]

Yes, sounds like I should put them on Zenodo or somewhere, which may be slower to begin with, but more stable. I've also had trouble where gdown would download certain files but only very slowly, which was probably the last stage before the error you encountered.

There is an undocumented feature in download_pg_data.py that gives limited control of what is downloaded. If you specify one or more strings on the command line after download_pg_data.py, it will only download files whose name in pg_data.yml includes that string. This can be useful for only re-downloading one particular file or folder. Sometimes I also run python download_pg_data.py none to skip downloading any ('none' doesn't appear in any of the filenames) and just re-create the pg/settings/env.yml file.