Retraction script failing #50
Description
The retraction script has been failing regularly for about 11 days now. There seem to be two separate issues.
- One is a known issue of a search node timing out (because the amount of returned items is less than the expected items - this seems to be happening only infrequently in the past for reasons I do not fully understand.
- But there are also a bunch of failures like this which show a trace similar to here
Details
```shell Backing up catalog [260](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:261) Traceback (most recent call last): [261](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:262) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/retry.py", line 115, in retry_request [262](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:263) return await func(*args, **kwargs) [263](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:264) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 374, in _request [264](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:265) headers=self._get_headers(headers), [265](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:266) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 353, in _get_headers [266](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:267) self.credentials.apply(out) [267](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:268) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 182, in apply [268](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:269) self.maybe_refresh() [269](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:270) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 177, in maybe_refresh [270](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:271) self.credentials.refresh(req) [271](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:272) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/service_account.py", line 410, in refresh [272](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:273) access_token, expiry, _ = _client.jwt_grant( [273](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:274) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 289, in jwt_grant [274](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:275) response_data = _token_endpoint_request( [275](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:276) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 260, in _token_endpoint_request [276](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:277) _handle_error_response(response_data, retryable_error) [277](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:278) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response [278](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:279) raise exceptions.RefreshError( [279](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:280) google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', ***'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'***) [280](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:281) Traceback (most recent call last): [281](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:282) File "create_filtered_catalog.py", line 75, in [282](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:283) gcs.put_file(local_filename, f'cmip6/***backup_filename***') [283](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:284) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/fsspec/asyn.py", line 111, in wrapper [284](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:285) return sync(self.loop, func, *args, **kwargs) [285](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:286) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/fsspec/asyn.py", line 96, in sync [286](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:287) raise return_result [287](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:288) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/fsspec/asyn.py", line 53, in _runner [288](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:289) result[0] = await coro [289](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:290) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 1061, in _put_file [290](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:291) location = await initiate_upload( [291](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:292) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 1559, in initiate_upload [292](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:293) headers, _ = await fs._call( [293](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:294) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 392, in _call [294](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:295) status, headers, info, contents = await self._request( [295](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:296) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/decorator.py", line 221, in fun [296](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:297) return await caller(func, *(extras + args), **kw) [297](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:298) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/retry.py", line 147, in retry_request [298](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:299) raise e [299](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:300) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/retry.py", line 115, in retry_request [300](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:301) return await func(*args, **kwargs) [301](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:302) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 374, in _request [302](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:303) headers=self._get_headers(headers), [303](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:304) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/core.py", line 353, in _get_headers [304](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:305) self.credentials.apply(out) [305](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:306) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 182, in apply [306](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:307) self.maybe_refresh() [307](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:308) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/gcsfs/credentials.py", line 177, in maybe_refresh [308](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:309) self.credentials.refresh(req) [309](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:310) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/service_account.py", line 410, in refresh [310](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:311) access_token, expiry, _ = _client.jwt_grant( [311](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:312) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 289, in jwt_grant [312](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:313) response_data = _token_endpoint_request( [313](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:314) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 260, in _token_endpoint_request [314](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:315) _handle_error_response(response_data, retryable_error) [315](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:316) File "/opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/google/oauth2/_client.py", line 69, in _handle_error_response [316](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:317) raise exceptions.RefreshError( [317](https://github.com/pangeo-data/pangeo-cmip6-cloud/actions/runs/3232057041/jobs/5292250131#step:6:318) google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', ***'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'***) ```I think the latter might be caused by our quite outdated version of google-github-actions/auth.
I will try to activate dependabot for this repo and see if that gets rid of these failures at least.