Fix CI breakages + questions on support policy

[Billyzhang1229]

Hi sgkit community, I want to more actively maintain this toolkit and would like to start by fixing the CI. I’ve fixed some issues, but there are a few points I’m not sure about and would like feedback.

What I’ve done

1. Pinned pandas < 3.0 to fix CI.
   Dask is not fully supported with pandas 3.0 yet, so this is likely a temporary pin until the ecosystem stabilizes.

2. Apple M4 + NumPy:
   On Apple M4 chips, NumPy < 2.3.5 can emit runtime warnings in matmul.
   I split NumPy constraints because:

   • Python 3.10 compatibility only works with NumPy < 2.3.0
   • Apple M4 issue is resolved by numpy >= 2.3.5
     Reference:

   BUG: "divide by zero encountered in matmul" on MacOS M4 with numpy v2.3.3 numpy/numpy#29820
   ENH: Make FPE blas check a runtime check for all apple arm systems numpy/numpy#30102
   https://numpy.org/doc/stable/release/2.3.0-notes.html

3. xarray xr.concat FutureWarning (data_vars=None default change)
   I made the test explicit with data_vars="all".
   Reference:

   Stricter defaults for concat, combine, open_mfdataset, merge pydata/xarray#8778
   New defaults for concat, merge, combine_* pydata/xarray#10062

4. Zarr error regex update
   Updated the regex to match the new “contains data” wording in Zarr v3.
   Reference:

   https://zarr.readthedocs.io/en/v3.0.9/_modules/zarr/storage/_common.html

Questions:

• Based on the NumPy/Python support tables, are we planning to drop Python 3.10 and NumPy < 2 soon?
  Reference:

  https://devguide.python.org/versions/#supported-versions
  https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
  https://numpy.org/neps/nep-0029-deprecation_policy.html#drop-schedule

• How should we handle warnings in CI for Zarr > 3 (ignore in tests vs. require code changes)?

[tomwhite]

Thanks @Billyzhang1229, this all looks very sensible.

Based on the NumPy/Python support tables, are we planning to drop Python 3.10 and NumPy < 2 soon?

We can certainly drop NumPy 1.x, as we've done that in bio2zarr and vcztools recently without any issues.

I would be OK with dropping Python 3.10 too, as we're not planning an sgkit release soon. What do you think @jeromekelleher?

How should we handle warnings in CI for Zarr > 3 (ignore in tests vs. require code changes)?

I would ignore the warnings in the tests for now unless there is a clear fix in the code. Running sgkit on Zarr 3 is still experimental.

[jeromekelleher]

I agree we can drop numpy 1.x and python 3.10.

[Billyzhang1229]

Thanks @tomwhite and @jeromekelleher.

Updates so far:

• Dropped Python 3.10 and NumPy 1.x support; removed relevant files/workflows and updated docs.
• Filtered two warnings for the Zarr 3 build workflow.
• Split NumPy requirements by sys_platform: Apple uses NumPy >=2.3.5 (Apple ARM chip fix).
• Fixed Windows build by installing most deps via pip (conda can’t parse PEP 508 markers): https://peps.python.org/pep-0508/
• Docs: relaxed sphinx / sphinx_autodoc_typehints and used PEP 604 unions to avoid typing.Union nitpicks: https://peps.python.org/pep-0604/

Current blocker: docs build still fails because dataset links are no longer available (#1332).

Data URLs referenced by docs notebooks

• https://storage.googleapis.com/sgkit-data/tutorial/1kg.vcf.bgz
• https://storage.googleapis.com/sgkit-data/tutorial/1kg.vcf.bgz.tbi
• https://storage.googleapis.com/sgkit-gwas-tutorial/1kg_annotations.txt
• https://storage.googleapis.com/sgkit-data/tutorial/Cleveland2012_pedigree.txt.gz
• https://storage.googleapis.com/sgkit-data/tutorial/Cleveland2012_snps1000.txt.gz
• https://storage.googleapis.com/sgkit-data/validation/hapmap_JPT_CHB_r23a_filtered.zip

Question:
Any thoughts on possibly using uv and ruff to manage environment and formatting?

[jeromekelleher]

Any thoughts on possibly using uv and ruff to manage environment and formatting?

Happy to consider these, we've used them on some tskit projects for a bit with some success. Ruff is over-opinionated by default though, and I prefer to add some exceptions to make it more Black-like.

[Billyzhang1229]

Thanks, that sounds reasonable to me.

Maybe we could open a separate issue focused on upgrading to more modern tooling (e.g. uv for environment / dependency management and ruff for linting/formatting), so it doesn’t get mixed in with the CI fixes. That would also be a good opportunity to simplify things by consolidating or removing the multiple requirements*.txt files, which have been a bit of a maintenance burden.

[jeromekelleher]

Yeah, I think a good approach would be "get things working again" first as easily as we can (if that entails temporarily removing the bit of the docs build that's broken so be it), and then we can hopefully "make things better" incrementally from there.

[Billyzhang1229]

I think I can temporarily disable execution of the notebooks during the docs build, which should avoid the broken external dataset links and unblock the pipeline. We can leave a clear TODO/comment pointing to #1332 so this doesn’t get forgotten.