Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
3a5d6da
Use uv package manager to test lowest versions in ci_tests_legacy.yaml
weiji14 Nov 21, 2024
f303964
Temporarily enable ci_tests_legacy.yaml
weiji14 Nov 21, 2024
3f6909d
Pin to netCDF>=1.7
weiji14 Nov 21, 2024
788fc46
Pin contextily>=1, IPython>=8, pyarrow>=14
weiji14 Nov 21, 2024
4139f4e
Pin geopandas>=0.14, rioxarray>=0.14
weiji14 Nov 21, 2024
1263075
Add --color=yes to test_no_images addopts
weiji14 Nov 21, 2024
d1fa38d
Pin packaging>=22.0
weiji14 Nov 21, 2024
1c69406
Test with pytest-mpl and pytest-doctestplus
weiji14 Nov 21, 2024
3b7b777
Install into uv venv
weiji14 Nov 21, 2024
6120bbe
Pin contextily>=1.2
weiji14 Nov 21, 2024
3299229
Pin pyarrow>=16
weiji14 Nov 21, 2024
0584459
Merge branch 'main' into uv/resolution-lowest
weiji14 Nov 22, 2024
7d542a9
Update wording on ci_tests_legacy.yaml header comment
weiji14 Nov 22, 2024
4733bda
Pin pyarrow>=13 and use pytest.mark.skipif on string_view type test
weiji14 Nov 22, 2024
e24a6c2
Merge branch 'main' into uv/resolution-lowest
weiji14 Dec 2, 2024
52c78fa
Remove unused import _get_module_version
weiji14 Dec 2, 2024
d64087e
Merge branch 'main' into uv/resolution-lowest
weiji14 Jan 2, 2025
af01f75
Bump astral-sh/setup-uv from 3 to 5.1.0
weiji14 Jan 2, 2025
3cb017c
Merge branch 'main' into uv/resolution-lowest
weiji14 Mar 23, 2025
fe0caf2
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 14, 2025
15da090
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 14, 2025
170fb09
Remove note about testing min versions on optional deps in ci_tests.yaml
weiji14 Apr 14, 2025
b1e26da
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 29, 2025
b079acc
Bump astral-sh/setup-uv from 5.1.0 to 6.0.0
weiji14 Apr 29, 2025
b7814ed
Merge branch 'main' into uv/resolution-lowest
weiji14 Apr 29, 2025
fe20756
Merge branch 'main' into uv/resolution-lowest
weiji14 Jun 9, 2025
2cc4ba2
Bump astral-sh/setup-uv from 6.0.0 to 6.1.0
weiji14 Jun 9, 2025
f54d002
Temporarily pin to rioxarray>=0.19
weiji14 Jun 9, 2025
834be64
Revert "Temporarily pin to rioxarray>=0.19"
weiji14 Jun 9, 2025
8ce0755
Merge branch 'main' into uv/resolution-lowest
weiji14 Jun 13, 2025
4540a5d
Try install pyproj from conda-forge instead of PyPI
weiji14 Jun 13, 2025
668cbe1
Merge branch 'main' into uv/resolution-lowest
weiji14 Sep 15, 2025
f5db0ad
:arrow_up: Bump astral-sh/setup-uv from 6.1.0 to 6.7.0
weiji14 Sep 15, 2025
0d8ec91
Edit comment in ci_tests.yaml
weiji14 Sep 15, 2025
bcb0e32
Merge branch 'main' into uv/resolution-lowest
weiji14 Feb 3, 2026
2fcde39
:arrow_up: Bump astral-sh/setup-uv from 6.7.0 to 7.2.1
weiji14 Feb 3, 2026
eed225c
Bump pip from 23 to 24
weiji14 Feb 3, 2026
d560c53
Revert "Try install pyproj from conda-forge instead of PyPI"
weiji14 Feb 3, 2026
c37404b
Merge branch 'main' into uv/resolution-lowest
weiji14 Feb 4, 2026
11fea3e
Relax ghostscript upper pin to <10.04
weiji14 Feb 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 19 additions & 24 deletions .github/workflows/ci_tests_legacy.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Test PyGMT with GMT legacy versions on Linux/macOS/Windows
# Test PyGMT with GMT legacy versions and old Python dependencies on Linux/macOS/Windows
#
# This workflow runs regular PyGMT tests with GMT legacy versions. Due to the minor
# baseline image changes between GMT versions, the workflow only runs the tests but
# doesn't do image comparisons.
# This workflow runs regular PyGMT tests with GMT legacy versions and old versions of
# all optional and required Python dependencies. Due to minor baseline image changes
# between GMT versions, the workflow runs the tests but doesn't do image comparisons.
#
# It is scheduled to run every Tuesday on the main branch.
#
Expand All @@ -12,7 +12,7 @@ on:
# push:
# branches: [ main ]
# Uncomment the 'pull_request' line below to trigger the workflow in PR
# pull_request:
pull_request:
# types: [ready_for_review]
# paths:
# - 'pygmt/**'
Expand Down Expand Up @@ -60,23 +60,7 @@ jobs:
create-args: >-
python=3.12
gmt=${{ matrix.gmt_version }}
ghostscript
numpy=2.0
pandas=2.2
xarray=2024.5
packaging=24.2
contextily=1.5
geopandas=1.0
ipython
pyarrow-core=16
rioxarray
sphinx-gallery
make
pip
python-build
pytest
pytest-doctestplus
pytest-mpl
ghostscript<10.04

# Download cached remote files (artifacts) from GitHub
- name: Download remote data from GitHub
Expand All @@ -89,10 +73,21 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}

# Install uv package manager
- name: Install uv
uses: astral-sh/setup-uv@803947b9bd8e9f986429fa0c5a41c367cd732b41 # v7.2.1
with:
activate-environment: true
python-version: 3.12

# Install the package that we want to test
- name: Install the package
run: make install
run: |
uv run --with pip==24 --resolution lowest-direct --all-extras --dev make install
uv pip list

# Run the tests but skip images
- name: Run tests
run: make test_no_images PYTEST_EXTRA="-r P"
run: uv run --with pytest==8,pytest-mpl==0.17,pytest-doctestplus==1.2 --resolution lowest-direct --all-extras --dev make test_no_images PYTEST_EXTRA="-r P"
env:
GMT_LIBRARY_PATH: $CONDA_PREFIX/lib
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof, it's not liking the GMT from conda-forge now. Traceback from https://github.com/GenericMappingTools/pygmt/actions/runs/21656150792/job/62431201928?pr=3639#step:7:39:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/runner/work/pygmt/pygmt/pygmt/__init__.py", line 24, in <module>
    from pygmt import datasets
  File "/home/runner/work/pygmt/pygmt/pygmt/datasets/__init__.py", line 28, in <module>
    from pygmt.datasets.samples import list_sample_data, load_sample_data
  File "/home/runner/work/pygmt/pygmt/pygmt/datasets/samples.py", line 11, in <module>
    from pygmt.src import which
  File "/home/runner/work/pygmt/pygmt/pygmt/src/__init__.py", line 5, in <module>
    from pygmt.src.basemap import basemap
  File "/home/runner/work/pygmt/pygmt/pygmt/src/basemap.py", line 9, in <module>
    from pygmt.clib import Session
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/__init__.py", line 9, in <module>
    from pygmt.clib.session import Session, __gmt_version__
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/session.py", line 116, in <module>
    _libgmt = load_libgmt()
              ^^^^^^^^^^^^^
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/loading.py", line 62, in load_libgmt
    raise GMTCLibNotFoundError("\n".join(error_msg))
pygmt.exceptions.GMTCLibNotFoundError: Error loading GMT shared library at '/home/runner/micromamba/envs/pygmt/lib/libgmt.so'.
/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found (required by /home/runner/micromamba/envs/pygmt/lib/./libgdal.so.37)
Error loading GMT shared library at 'libgmt.so'.
libgmt.so: cannot open shared object file: No such file or directory
make: *** [Makefile:35: _runtest] Error 1

GMT was installed via micromamba. Not sure if that libstdc++.so.6: version CXXABI_1.3.15' not found` error is from libgdal or elsewhere?

4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ dynamic = ["version"]
all = [
"contextily>=1.5",
"geopandas>=1.0",
"IPython", # 'ipython' is not the correct module name.
"IPython>=8", # 'ipython' is not the correct module name.
"pyarrow>=16",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinning to pyarrow>16 here to fix this error:

________ test_to_numpy_pyarrow_array_pyarrow_dtypes_string[string_view] ________
>   ???
E   KeyError: 'string_view'
pyarrow/types.pxi:5025: KeyError
During handling of the above exception, another exception occurred:
dtype = 'string_view'
    @pytest.mark.skipif(not _HAS_PYARROW, reason="pyarrow is not installed")
    @pytest.mark.parametrize(
        "dtype",
        [
            None,
            "string",
            "utf8",  # alias for string
            "large_string",
            "large_utf8",  # alias for large_string
            "string_view",
        ],
    )
    def test_to_numpy_pyarrow_array_pyarrow_dtypes_string(dtype):
        """
        Test the _to_numpy function with PyArrow arrays of PyArrow string types.
        """
>       array = pa.array(["abc", "defg", "12345"], type=dtype)
../pygmt/tests/test_clib_to_numpy.py:333: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/array.pxi:230: in pyarrow.lib.array
    ???
pyarrow/types.pxi:5040: in pyarrow.lib.ensure_type
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   ValueError: No type alias for string_view
pyarrow/types.pxi:5027: ValueError

Code was added in #3608:

"string_view",
],
)
def test_to_numpy_pyarrow_array_pyarrow_dtypes_string(dtype):
"""
Test the _to_numpy function with PyArrow arrays of PyArrow string types.
"""
array = pa.array(["abc", "defg", "12345"], type=dtype)
result = _to_numpy(array)
_check_result(result, np.str_)
npt.assert_array_equal(result, array)

The pyarrow.StringViewArray class was added in apache/arrow#39652 that was released in pyarrow v16.0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyArrow v16.0 was released in April, 2024. Instead of pinning pyarrow>=16.0, we can just skip string_view if pyarrow<v16.0.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I just tested older versions from pyarrow v12 to v15, and think we can pin to pyarrow>=13 (v13 was released on 24 Aug 2023) if ignoring string_view. With v12, there is another error on datetime.

_________________________________________________________________ test_to_numpy_pyarrow_array_pyarrow_dtypes_date[date64[ms]] __________________________________________________________________

dtype = 'date64[ms]', expected_dtype = 'datetime64[ms]'

    @pytest.mark.skipif(not _HAS_PYARROW, reason="pyarrow is not installed")
    @pytest.mark.parametrize(
        ("dtype", "expected_dtype"),
        [
            pytest.param("date32[day]", "datetime64[D]", id="date32[day]"),
            pytest.param("date64[ms]", "datetime64[ms]", id="date64[ms]"),
        ],
    )
    def test_to_numpy_pyarrow_array_pyarrow_dtypes_date(dtype, expected_dtype):
        """
        Test the _to_numpy function with PyArrow arrays of PyArrow date types.

        date32[day] and date64[ms] are stored as 32-bit and 64-bit integers, respectively,
        representing the number of days and milliseconds since the UNIX epoch (1970-01-01).

        Here we explicitly check the dtype and date unit of the result.
        """
        data = [
            date(2024, 1, 1),
            datetime(2024, 1, 2),
            datetime(2024, 1, 3),
        ]
        array = pa.array(data, type=dtype)
        result = _to_numpy(array)
        _check_result(result, np.datetime64)
>       assert result.dtype == expected_dtype  # Explicitly check the date unit.
E       AssertionError: assert dtype('<M8[D]') == 'datetime64[ms]'
E        +  where dtype('<M8[D]') = array(['2024-01-01', '2024-01-02', '2024-01-03'], dtype='datetime64[D]').dtype

../pygmt/tests/test_clib_to_numpy.py:364: AssertionError

The main change appears to be in apache/arrow#33321, when pyarrow supported preserving the datetime64 temporal resolution.

Maybe we can keep the string_view test, but add a skip_if(pyarrow.__version__ < 16) marker or something like that.

Copy link
Member Author

@weiji14 weiji14 Nov 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjusted pin to pyarrow>=13 in commit 4733bda

Edit: pin is now back at pyarrow>=16 with PR #3917.

"rioxarray",
"rioxarray>=0.14",
Copy link
Member Author

@weiji14 weiji14 Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting this test failure on windows-2022 at https://github.com/GenericMappingTools/pygmt/actions/runs/15524973929/job/43703740279?pr=3639#step:7:1095:

_______________ [doctest] pygmt.datasets.tile_map.load_tile_map _______________
118     >>> raster.sizes
119     Frozen({'band': 3, 'y': 256, 'x': 512})
120     >>> raster.coords  # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
121     Coordinates:
122       * band         (band) uint8... 1 2 3
123       * y            (y) float64... -7.081e-10 -7.858e+04 ... -1.996e+07 -2.004e+07
124       * x            (x) float64... -2.004e+07 -1.996e+07 ... 1.996e+07 2.004e+07
125         spatial_ref  int... 0
126     >>> # CRS is set only if rioxarray is available
127     >>> if hasattr(raster, "rio"):
Expected:
    'EPSG:3857'
Got:
    'PROJCS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Mercator_1SP"],PARAMETER["central_meridian",0],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1],AXIS["Easting",EAST],AXIS["Northing",NORTH],EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +nadgrids=@null +wktext +no_defs"],AUTHORITY["EPSG","3857"]]'

D:\a\pygmt\pygmt\pygmt\datasets\tile_map.py:127: DocTestFailure

Similar situation to the one reported at #3575. We could perhaps pin to a rioxarray version that requires rasterio>=1.4.2? E.g. rioxarray=0.19.0 pins to rasterio>=1.4.3 (corteva/rioxarray#850). However, that version was released too recently on 22 Apr 2025 (<2 months).

Suggestions on what to do?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual error messages are

ERROR 1: PROJ: proj_create_from_database: C:\Users\runneradmin\micromamba\envs\pygmt/Library/share/proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 3 whereas a number >= 4 is expected. It comes from another PROJ installation.
ERROR 1: PROJ: proj_identify: C:\Users\runneradmin\micromamba\envs\pygmt/Library/share/proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 3 whereas a number >= 4 is expected. It comes from another PROJ installation.

Some related discussions are in corteva/rioxarray#447. I'm unsure if pin rioxarray>=0.19.0 can solve the issue, but please give it a try first.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm you're right, got the same ERROR 1: PROJ: ... message even with rioxarray>0.19.0. Probably because we are using uv/PyPI installed packages with incompatible PROJ versions, instead of conda-forge packages where packages are compiled with the same PROJ lib.

Copy link
Member

@seisman seisman Jun 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dependency tree GMT->GDAL->PROJ installs PROJ 9.4.0, while uv/pypi installs pyproj 3.7.1 which contains PROJ 9.5.1.

Perhaps we can install pyproj via micromamba instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can install pyproj via micromamba instead?

Tried at commit 4540a5d, same error 😒

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if setting env PROJ_WHEEL=YES?

xref: https://pyproj4.github.io/pyproj/dev/installation.html#setup-pyproj

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PROJ_WHEEL seems to be relevant only when building the pyproj wheel, but since we're pulling a compiled version of pyproj from PyPI/conda-forge, it probably doesn't do anything.

]

[project.entry-points."xarray.backends"]
Expand Down
Loading