needs a boost for timeout of hugging face downloads

[yarikoptic]

I just kept rerunning

(aind-ephys-pipeline) (base) yoh@typhon:/data/aind/try-aind-1$ rm -rf logs nextflow/; datalad run -m "Running pipeline via datalad + switching to hdf5 backend" '( export DATA_PATH=$PWD/sourcedata/nwb/ RESULTS_PATH=$PWD HF_HUB_DOWNLOAD_TIMEOUT=300;  nextflow -C code/aind-ephys-pipeline/pipeline/nextflow_local.config -log $RESULTS_PATH/logs/nextflow.log     run $PWD/code/aind-ephys-pipeline/pipeline/main_multi_backend.nf   --n_jobs 8 --job_dispatch_args "--input nwb" --nwb_ecephys_args "--backend hdf5" ; )'

until it succeeded. HF_HUB_DOWNLOAD_TIMEOUT did not have effect since when crash did happen -- it was saying that timeout=10. Unfortunately ATM I do not have captured output to show, will edit when I get one

[yarikoptic]

here is more of the context

  git -C capsule-repo -c core.fileMode=false checkout 4d8f38adecb21d640295a556a57446018249ab76 --quiet
  mv capsule-repo/code capsule/code
  rm -rf capsule-repo

  echo "[curation] running capsule..."
  cd capsule/code
  chmod +x run
  ./run   --n-jobs 8

  echo "[curation] completed!"

Command exit status:
  1

Command output:
  [curation] cloning git repo...
  [curation] running capsule...

  CURATION
  Curating recording: block0_acquisition-ElectricalSeriesRaw_recording1
  Curation query: isi_violations_ratio < 0.5 and presence_ratio > 0.8 and amplitude_cutoff < 0.1
        Passing default QC: 18 / 18
  Applying noise-neural classifier from SpikeInterface/UnitRefine_noise_neural_classifier

Command error:
      r = _request_wrapper(
          ^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 286, in _request_wrapper
      response = _request_wrapper(
                 ^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 309, in _request_wrapper
      response = http_backoff(method=method, url=url, **params, retry_on_exceptions=(), retry_on_status_codes=(429,))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 310, in http_backoff
      response = session.request(method=method, url=url, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
      resp = self.send(prep, **send_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
      r = adapter.send(request, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 96, in send
      return super().send(request, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
      raise ReadTimeout(e, request=request)
  requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: b4315909-26ed-49db-9231-2fefe98ae71e)')

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "capsule/code/run_capsule.py", line 194, in <module>
      noise_neuron_labels = scur.auto_label_units(
                            ^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/spikeinterface/curation/model_based_curation.py", line 268, in auto_label_units
      model, model_info = load_model(
                          ^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/spikeinterface/curation/model_based_curation.py", line 321, in load_model
      model, model_info = _load_model_from_huggingface(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/spikeinterface/curation/model_based_curation.py", line 347, in _load_model_from_huggingface
      full_path = hf_hub_download(repo_id=repo_id, filename=filename)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
      return fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1010, in hf_hub_download
      return _hf_hub_download_to_cache_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1117, in _hf_hub_download_to_cache_dir
      _raise_on_head_call_error(head_call_error, force_download, local_files_only)
    File "/opt/conda/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1661, in _raise_on_head_call_error
      raise LocalEntryNotFoundError(
  huggingface_hub.errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

Work dir: