CI fails to authenticate causing `notebooks` and `integration` failures

[jonbrenas]

When CI runs the notebooks and integration tests, it fails to authenticate properly to GCS resulting in failures.

It replicates with all PRs I tried so my guess is that something changed with the way CI is supposed to be authenticating.

[leehart]

My first guess is that it might be related to this in workflows/notebooks.yml and workflows/integration_tests.yml:

      - id: 'auth'
        name: 'Set up Google Cloud authentication'
        uses: 'google-github-actions/auth@v2'
        with:
          credentials_json: '${{ secrets.GOOGLE_CREDENTIALS }}'

I'll see whether GOOGLE_CREDENTIALS has simply expired and needs refreshing.

[leehart]

The only info I can see on GOOGLE_CREDENTIALS is that it was last updated last year, so perhaps it had a one-year expiry date. Anyway, I'll try to recreate it as soon as possible, but I will need to make sure I do that correctly, obviously.

https://github.com/malariagen/malariagen-data-python/settings/secrets/actions

[leehart]

Here are the docs for google-github-actions https://github.com/google-github-actions/auth that I'll be reading through.

[leehart]

It appears that we have a non-expired service account for this purpose, which stopped being used 1 day ago (20th May at 11:50 then sporadically until 18:33). So I don't think the problem is an expired account or key. One of the keys was created in March 2024, which I gather is the one being used here, because the other one was only created in January. (I don't know why we have two keys on the same service account.)

There is a note from Google that:

Service account keys could pose a security risk if compromised. We recommend that you avoid downloading service account keys and instead use the workload identity federation.

So maybe there are two keys because someone wanted to use the same service account but didn't have the first key, so created a new one. (The service account key can be downloaded as JSON, which should be minified into a single line before being stored as the GitHub Secret for this purpose.)

If it's not simply an expired credential that is causing this sudden failure, then I wonder what else might have changed. 🤔

[leehart]

Considering the error

TypeError: can't subtract offset-naive and offset-aware datetimes

I wonder if this is simply a bug in GCSFS, that has surfaced, perhaps because of some kind of environmental change, such as a minor version bump in one of our dependencies. This looks like a common gotcha around using datetime, so I'll check the gcsfs issues to see if it's a known bug there.

[leehart]

As suspected, I reckon this is an issue in gcsfs which was reported a couple of days ago and looks like it's been fixed there:

Issue fsspec/gcsfs#682

I'm not sure how we can implement that fix though.

PR fsspec/gcsfs#683

Looks like we need at least gcsfs-2025.5.0.post1, which we should already be on pypi.

[leehart]

Judging from re-running the notebooks and integration tests on PR #704 and them now passing GCS authentication (integration_tests has finished but notebooks takes an absurdly long time), it looks like this has already resolved itself due to the upstream fix to gcsfs.

[jonbrenas]

Yeahhh! Issues that solve themselves are my favourite!