Skip to content

WIP: Test lowest versions of all required and optional dependencies #3639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 34 commits into
base: main
Choose a base branch
from
Draft

WIP: Test lowest versions of all required and optional dependencies #3639

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
3a5d6da
Use uv package manager to test lowest versions in ci_tests_legacy.yaml
weiji14 Nov 21, 2024
f303964
Temporarily enable ci_tests_legacy.yaml
weiji14 Nov 21, 2024
3f6909d
Pin to netCDF>=1.7
weiji14 Nov 21, 2024
788fc46
Pin contextily>=1, IPython>=8, pyarrow>=14
weiji14 Nov 21, 2024
4139f4e
Pin geopandas>=0.14, rioxarray>=0.14
weiji14 Nov 21, 2024
1263075
Add --color=yes to test_no_images addopts
weiji14 Nov 21, 2024
d1fa38d
Pin packaging>=22.0
weiji14 Nov 21, 2024
1c69406
Test with pytest-mpl and pytest-doctestplus
weiji14 Nov 21, 2024
3b7b777
Install into uv venv
weiji14 Nov 21, 2024
6120bbe
Pin contextily>=1.2
weiji14 Nov 21, 2024
3299229
Pin pyarrow>=16
weiji14 Nov 21, 2024
0584459
Merge branch 'main' into uv/resolution-lowest
weiji14 Nov 22, 2024
7d542a9
Update wording on ci_tests_legacy.yaml header comment
weiji14 Nov 22, 2024
4733bda
Pin pyarrow>=13 and use pytest.mark.skipif on string_view type test
weiji14 Nov 22, 2024
e24a6c2
Merge branch 'main' into uv/resolution-lowest
weiji14 Dec 2, 2024
52c78fa
Remove unused import _get_module_version
weiji14 Dec 2, 2024
d64087e
Merge branch 'main' into uv/resolution-lowest
weiji14 Jan 2, 2025
af01f75
Bump astral-sh/setup-uv from 3 to 5.1.0
weiji14 Jan 2, 2025
3cb017c
Merge branch 'main' into uv/resolution-lowest
weiji14 Mar 23, 2025
fe0caf2
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 14, 2025
15da090
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 14, 2025
170fb09
Remove note about testing min versions on optional deps in ci_tests.yaml
weiji14 Apr 14, 2025
b1e26da
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 29, 2025
b079acc
Bump astral-sh/setup-uv from 5.1.0 to 6.0.0
weiji14 Apr 29, 2025
b7814ed
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 29, 2025
fe20756
Merge branch 'main' into uv/resolution-lowest
weiji14 Jun 9, 2025
2cc4ba2
Bump astral-sh/setup-uv from 6.0.0 to 6.1.0
weiji14 Jun 9, 2025
f54d002
Temporarily pin to rioxarray>=0.19
weiji14 Jun 9, 2025
834be64
Revert "Temporarily pin to rioxarray>=0.19"
weiji14 Jun 9, 2025
8ce0755
Merge branch 'main' into uv/resolution-lowest
weiji14 Jun 13, 2025
4540a5d
Try install pyproj from conda-forge instead of PyPI
weiji14 Jun 13, 2025
668cbe1
Merge branch 'main' into uv/resolution-lowest
weiji14 Sep 15, 2025
f5db0ad
:arrow_up: Bump astral-sh/setup-uv from 6.1.0 to 6.7.0
weiji14 Sep 15, 2025
0d8ec91
Edit comment in ci_tests.yaml
weiji14 Sep 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 15 additions & 24 deletions .github/workflows/ci_tests_legacy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Test PyGMT with GMT legacy versions on Linux/macOS/Windows
# Test PyGMT with GMT legacy versions and old Python dependencies on Linux/macOS/Windows
#
# This workflow runs regular PyGMT tests with GMT legacy versions. Due to the minor
# baseline image changes between GMT versions, the workflow only runs the tests but
# doesn't do image comparisons.
# This workflow runs regular PyGMT tests with GMT legacy versions and old versions of
# all optional Python dependencies. Due to the minor baseline image changes between GMT
# versions, the workflow only runs the tests but doesn't do image comparisons.
#
# It is scheduled to run every Tuesday on the main branch.
#
Expand All @@ -12,7 +12,7 @@ on:
# push:
# branches: [ main ]
# Uncomment the 'pull_request' line below to trigger the workflow in PR
# pull_request:
pull_request:
# types: [ready_for_review]
# paths:
# - 'pygmt/**'
Expand Down Expand Up @@ -62,23 +62,6 @@ jobs:
python=3.10
gmt=${{ matrix.gmt_version }}
ghostscript<10
numpy
pandas
xarray
netCDF4
packaging
contextily
geopandas
ipython
pyarrow
rioxarray
sphinx-gallery
make
pip
python-build
pytest
pytest-doctestplus
pytest-mpl

# Download cached remote files (artifacts) from GitHub
- name: Download remote data from GitHub
Expand All @@ -95,10 +78,18 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}

# Install uv package manager
- name: Install uv
uses: astral-sh/setup-uv@v3

# Install the package that we want to test
- name: Install the package
run: make install
run: |
uv run --with pip==23 --resolution lowest-direct --all-extras --dev make install
Copy link
Member Author

@weiji14 weiji14 Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using --resolution lowest-direct for now, because --resolution lowest grabs many other transitive dependencies that may be hard to install (e.g. missing wheels for newer Python versions, so requires compilation from source). We could switch to --resolution lowest once the Python ecosystem does lower bound pins a bit more thoroughly (which may take years).

uv pip list

# Run the tests but skip images
- name: Run tests
run: make test_no_images PYTEST_EXTRA="-r P"
run: uv run --with pytest==8 --resolution lowest-direct --all-extras --dev make test_no_images PYTEST_EXTRA="-r P"
env:
GMT_LIBRARY_PATH: $CONDA_PREFIX/lib
Copy link
Member Author

@weiji14 weiji14 Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting this error at https://github.com/GenericMappingTools/pygmt/actions/runs/11945150103/job/33297317606?pr=3639#step:7:35:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/runner/work/pygmt/pygmt/pygmt/__init__.py", line 24, in <module>
    from pygmt import datasets
  File "/home/runner/work/pygmt/pygmt/pygmt/datasets/__init__.py", line 7, in <module>
    from pygmt.datasets.earth_age import load_earth_age
  File "/home/runner/work/pygmt/pygmt/pygmt/datasets/earth_age.py", line 12, in <module>
    from pygmt.datasets.load_remote_dataset import _load_remote_dataset
  File "/home/runner/work/pygmt/pygmt/pygmt/datasets/load_remote_dataset.py", line 9, in <module>
    from pygmt.clib import Session
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/__init__.py", line 9, in <module>
    from pygmt.clib.session import Session, __gmt_version__
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/session.py", line 111, in <module>
    _libgmt = load_libgmt()
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/loading.py", line 62, in load_libgmt
    raise GMTCLibNotFoundError("\n".join(error_msg))
pygmt.exceptions.GMTCLibNotFoundError: Error loading GMT shared library at '/home/runner/micromamba/envs/pygmt/lib/libgmt.so'.
/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/runner/micromamba/envs/pygmt/lib/./libgdal.so.34)
Error loading GMT shared library at 'libgmt.so'.
/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/runner/micromamba/envs/pygmt/lib/python3.10/lib-dynload/../.././libgdal.so.34)
make: *** [Makefile:35: _runtest] Error 1

Similar to issues mentioned at GenericMappingTools/try-gmt#50 (comment) and GenericMappingTools/try-gmt#37 (comment). Seems to only be on the Ubuntu/Linux build.

12 changes: 6 additions & 6 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,18 @@ dependencies = [
"numpy>=1.24",
"pandas>=2.0",
"xarray>=2023.04",
"netCDF4",
"netCDF4>=1.7",
Copy link
Member Author

@weiji14 weiji14 Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed making NetCDF an optional dependency before in #429. Maybe we could revisit that discussion?

Copy link
Member

@seisman seisman Nov 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. NetCDF4 was needed by xarray when reading and writing netCDF files. As mentioned in #239 (comment), previously, our load_xxx functions (e.g., load_earth_relief) called xarray.load_dataarray to load local netCDF grids into an xarray.DataArray object, making netCDF a highly "required" dependency.

After we refactor our load_xxx function in #3120, we no longer call xarray.load_dataarray, so netCDF should be no longer highly needed by PyGMT.

I've opened PR #3643 to see how the PyGMT relies on the netCDF package.

Edit: There are only 14 failures in the Python 3.11 + Ubuntu CI job after removing netCDF entirely (see https://github.com/GenericMappingTools/pygmt/actions/runs/11968416461/job/33367210857?pr=3643) and most of them are caused by load_static_earth_relief.

Copy link
Member Author

@weiji14 weiji14 Apr 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is another error that popped up on Ubuntu-22.04-ARM64 at https://github.com/GenericMappingTools/pygmt/actions/runs/14457975280/job/40545076044?pr=3639#step:7:1079, related to static_earth_relief too:

==================================== ERRORS ====================================
__________________ ERROR at setup of test_clib_read_data_grid __________________

self = CachingFileManager(<class 'netCDF4._netCDF4.Dataset'>, '/home/runner/.gmt/cache/static_earth_relief.nc', mode='r', kwa...r': True, 'diskless': False, 'persist': False, 'format': 'NETCDF4'}, manager_id='26bc1ee9-cb26-4f6f-bcea-ac669a2faa62')
needs_lock = True

    def _acquire_with_cache_info(self, needs_lock=True):
        """Acquire a file, returning the file and whether it was cached."""
        with self._optional_lock(needs_lock):
            try:
>               file = self._cache[self._key]

../.venv/lib/python3.11/site-packages/xarray/backends/file_manager.py:210: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <xarray.backends.lru_cache.LRUCache object at 0xff802d7a2b80>
key = [<class 'netCDF4._netCDF4.Dataset'>, ('/home/runner/.gmt/cache/static_earth_relief.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '26bc1ee9-cb26-4f6f-bcea-ac669a2faa62']

    def __getitem__(self, key: K) -> V:
        # record recent use of the key by moving it to the front of the list
        with self._lock:
>           value = self._cache[key]
E           KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/runner/.gmt/cache/static_earth_relief.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '26bc1ee9-cb26-4f6f-bcea-ac669a2faa62']

../.venv/lib/python3.11/site-packages/xarray/backends/lru_cache.py:56: KeyError

During handling of the above exception, another exception occurred:

    @pytest.fixture(scope="module", name="expected_xrgrid")
    def fixture_expected_xrgrid():
        """
        The expected xr.DataArray object for the static_earth_relief.nc file.
        """
>       return load_dataarray(which("@static_earth_relief.nc"))

../pygmt/tests/test_clib_read_data.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../pygmt/io.py:44: in load_dataarray
    with xr.open_dataarray(filename_or_obj, **kwargs) as dataarray:
../.venv/lib/python3.11/site-packages/xarray/backends/api.py:686: in open_dataarray
    dataset = open_dataset(
../.venv/lib/python3.11/site-packages/xarray/backends/api.py:525: in open_dataset
    backend_ds = backend.open_dataset(
../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:588: in open_dataset
    store = NetCDF4DataStore.open(
../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:389: in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:336: in __init__
    self.format = self.ds.data_model
../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:398: in ds
    return self._acquire()
../.venv/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:392: in _acquire
    with self._manager.acquire_context(needs_lock) as root:
../../../../.local/share/uv/python/cpython-3.11.12-linux-aarch64-gnu/lib/python3.11/contextlib.py:137: in __enter__
    return next(self.gen)
../.venv/lib/python3.11/site-packages/xarray/backends/file_manager.py:198: in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
../.venv/lib/python3.11/site-packages/xarray/backends/file_manager.py:216: in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
src/netCDF4/_netCDF4.pyx:2489: in netCDF4._netCDF4.Dataset.__init__
    ???
src/netCDF4/_netCDF4.pyx:2092: in netCDF4._netCDF4._get_vars
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   RuntimeError: only endian='native' allowed for NETCDF3 files, got 'big' (variable 'lon', group '/')

src/netCDF4/_netCDF4.pyx:4062: RuntimeError

This seems to be fixed in Unidata/netcdf4-python#1355, released in netcdf4=1.7.2. So we can either set the minimum version to netCDF4>=1.7.2 and/or remove netCDF4 in #3643. Would prefer the latter, because netCDF4=1.7.2 is the latest version released on Oct 2024, so not very old.

Edit: netCDF4 is no longer a required dependency of PyGMT since #3643, so no need to worry about this anymore.

"packaging",
]
dynamic = ["version"]

[project.optional-dependencies]
all = [
"contextily",
"geopandas",
"IPython", # 'ipython' is not the correct module name.
"pyarrow",
"rioxarray",
"contextily>=1",
"geopandas>=0.14",
"IPython>=8", # 'ipython' is not the correct module name.
"pyarrow>=14",
"rioxarray>=0.14",
Copy link
Member Author

@weiji14 weiji14 Nov 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting lower bounds here, otherwise uv will try to install super old versions that may not work anymore, some which actually go back to Python 2! For e.g., it tried to install rioxarray=0.0.1 at https://github.com/GenericMappingTools/pygmt/actions/runs/11945100244/job/33297179010#step:7:45. Happy to reconsider some of these version pins if needed.

Edit see comments at #3639 (comment) and #3639 (comment) below.

Copy link
Member Author

@weiji14 weiji14 Jun 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting this test failure on windows-2022 at https://github.com/GenericMappingTools/pygmt/actions/runs/15524973929/job/43703740279?pr=3639#step:7:1095:

_______________ [doctest] pygmt.datasets.tile_map.load_tile_map _______________
118     >>> raster.sizes
119     Frozen({'band': 3, 'y': 256, 'x': 512})
120     >>> raster.coords  # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
121     Coordinates:
122       * band         (band) uint8... 1 2 3
123       * y            (y) float64... -7.081e-10 -7.858e+04 ... -1.996e+07 -2.004e+07
124       * x            (x) float64... -2.004e+07 -1.996e+07 ... 1.996e+07 2.004e+07
125         spatial_ref  int... 0
126     >>> # CRS is set only if rioxarray is available
127     >>> if hasattr(raster, "rio"):
Expected:
    'EPSG:3857'
Got:
    'PROJCS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Mercator_1SP"],PARAMETER["central_meridian",0],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +nadgrids=@null +wktext +no_defs"],AUTHORITY["EPSG","3857"]]'

D:\a\pygmt\pygmt\pygmt\datasets\tile_map.py:127: DocTestFailure

Similar situation to the one reported at #3575. We could perhaps pin to a rioxarray version that requires rasterio>=1.4.2? E.g. rioxarray=0.19.0 pins to rasterio>=1.4.3 (corteva/rioxarray#850). However, that version was released too recently on 22 Apr 2025 (<2 months).

Suggestions on what to do?

Copy link
Member

@seisman seisman Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual error messages are

ERROR 1: PROJ: proj_create_from_database: C:\Users\runneradmin\micromamba\envs\pygmt/Library/share/proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 3 whereas a number >= 4 is expected. It comes from another PROJ installation.
ERROR 1: PROJ: proj_identify: C:\Users\runneradmin\micromamba\envs\pygmt/Library/share/proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 3 whereas a number >= 4 is expected. It comes from another PROJ installation.

Some related discussions are in corteva/rioxarray#447. I'm unsure if pin rioxarray>=0.19.0 can solve the issue, but please give it a try first.

Copy link
Member Author

@weiji14 weiji14 Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm you're right, got the same ERROR 1: PROJ: ... message even with rioxarray>0.19.0. Probably because we are using uv/PyPI installed packages with incompatible PROJ versions, instead of conda-forge packages where packages are compiled with the same PROJ lib.

Copy link
Member

@seisman seisman Jun 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dependency tree GMT->GDAL->PROJ installs PROJ 9.4.0, while uv/pypi installs pyproj 3.7.1 which contains PROJ 9.5.1.

Perhaps we can install pyproj via micromamba instead?

Copy link
Member Author

@weiji14 weiji14 Jun 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can install pyproj via micromamba instead?

Tried at commit 4540a5d, same error 😢

Copy link
Member

@seisman seisman Jun 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if setting env PROJ_WHEEL=YES?

xref: https://pyproj4.github.io/pyproj/dev/installation.html#setup-pyproj

Copy link
Member Author

@weiji14 weiji14 Jun 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PROJ_WHEEL seems to be relevant only when building the pyproj wheel, but since we're pulling a compiled version of pyproj from PyPI/conda-forge, it probably doesn't do anything.

]

[project.urls]
Expand Down
Loading