Missing example data causes gdown script to fail

[chasemc]

There are some files that don't exist for all of the example datasets which causes the script to fail:

Access denied with the following error:

        Cannot retrieve the public link of the file. You may need to change
        the permission to 'Anyone with the link', or have had many accesses. 

You may still be able to access the file from the browser:



Traceback (most recent call last):
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 3090, in _get_loc_level
    return (self._engine.get_loc(key), None)
  File "pandas/_libs/index.pyx", line 782, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1920, in pandas._libs.hashtable.UInt64HashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1930, in pandas._libs.hashtable.UInt64HashTable.get_item
KeyError: 150

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chase/miniconda3/envs/autometa/bin/autometa-download-dataset", line 8, in <module>
    sys.exit(main())
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/autometa/validation/datasets.py", line 197, in main
    download(
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/autometa/validation/datasets.py", line 63, in download
    file_id = df.loc[(community_size, file_name), "file_id"]
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexing.py", line 1067, in __getitem__
    return self._getitem_tuple(key)
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexing.py", line 1247, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexing.py", line 941, in _getitem_lowerdim
    return self._getitem_nested_tuple(tup)
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexing.py", line 1047, in _getitem_nested_tuple
    obj = getattr(obj, self.name)._getitem_axis(key, axis=axis)
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexing.py", line 1312, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexing.py", line 1260, in _get_label
    return self.obj.xs(label, axis=axis)
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/generic.py", line 4049, in xs
    loc, new_index = index._get_loc_level(key, level=0)
  File "/home/chase/miniconda3/envs/autometa/lib/python3.10/site-packages/pandas/core/indexes/multi.py", line 3092, in _get_loc_level
    raise KeyError(key) from err
KeyError: ('2500Mbp', 'unclustered_recruitment.tsv.gz')

[rajnish107]

Hey, I wanna work on this issue can you guide me?

[evanroyrees]

Hello Rajnish, thank you for getting involved. I see you have submitted a pull request (#307), thanks so much! Generally we prefer new changes be pushed to the dev branch and then synchronize our branches with new releases. This way we can save ourselves from accidentally introducing bugs. It looks like you may have addressed all of the issues, but @chasemc and @ajlail98 may want to weigh in on next steps. Thanks again for your contributions! Feel free to tackle more! 😄

[rajnish107]

Well, thanks! 😅

Hello Rajnish, thank you for getting involved. I see you have submitted a pull request (#307), thanks so much! Generally we prefer new changes be pushed to the dev branch and then synchronize our branches with new releases. This way we can save ourselves from accidentally introducing bugs. It looks like you may have addressed all of the issues, but @chasemc and @ajlail98 may want to weigh in on next steps. Thanks again for your contributions! Feel free to tackle more! 😄

[chasemc]

Probably best to wrap

Autometa/autometa/validation/datasets.py

Lines 63 to 67 in b1ec370

	file_id = df.loc[(community_size, file_name), "file_id"]
	file_id_filepath = os.path.join(community_size_outdir, file_name)
	url = f"https://drive.google.com/uc?id={file_id}"
	gdown.download(url, file_id_filepath)

in a non-failing try/except

The amount of data the script downloads is quite large, I would suggest making the suggested change and PR but maybe not testing it on your end