prek_ty: use prek instead of pre-commit, ty for type checking (#107)

* prek_ty: use prek instead of pre-commit, ty for type checking

* remove duplicate binder comment workflow

* testing: run only on push, not pr

* fix: pin intake-thredds to 2024.3.1, skip flaky URL tests

* add pyarrow and intake-thredds to dependencies

* require python<3.14

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* unskip URL availability tests

* add pytest-sugar to test dependencies

* remove unavailable datasets: WOA2018, CRU_TS4_*, GeigerKoeppen

* Add ty pre-commit hook and document uv run commands

* Remove test_cache folder and add to .gitignore

* Add pytest-xdist for parallel test execution

* Use pytest -n auto for parallel test execution in GHA

* Fix ty-pre-commit repo URL (use hoxbro instead of astral-sh)

* Configure ty to check remote_climate_data and tests

* Pin setuptools<70 to fix intake_thredds plugin loading

* Pin setuptools<70 to fix intake_thredds plugin loading

* Remove cru_ts catalog and ceda-access notebook

* Skip ESACCI URL test (oceancolour.org THREDDS returns 404)

* Skip GeigerKoeppen_shp URL test (HTTP URL may fail availability check)

* Remove CRU_TS subcatalog reference (cru_ts.yaml was removed)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: aaronspring

.github/workflows/comment_binder_pr_link.yml

@@ -3,7 +3,6 @@ name: testing
 
 on:
   push:
-  pull_request:
   schedule:
     - cron: "0 0 * * 4"  # Thursdays "At 00:00" UTC
   workflow_dispatch:  # allows you to trigger manually
@@ -38,7 +37,7 @@ jobs:
         run: conda list
       - name: Run tests
         run: |
-          pytest --durations=20
+          pytest -n auto --durations=20
 
   test_notebooks:  # Checks that notebooks complete without failure.
     runs-on: ubuntu-latest

.github/workflows/testing.yml

@@ -1,5 +1,6 @@
 # simplecache
 notebooks/my_caching_folder
+test_cache/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

.gitignore

@@ -20,6 +20,16 @@ repos:
             args: [--fix]
           - id: ruff-format
 
+    - repo: https://github.com/kynan/nbstripout
+      rev: 0.7.1
+      hooks:
+          - id: nbstripout
+
+    - repo: https://github.com/hoxbro/ty-pre-commit
+      rev: v0.0.12
+      hooks:
+          - id: ty-check
+
     - repo: https://github.com/adrienverge/yamllint.git
       rev: v1.35.1
       hooks:

.pre-commit-config.yaml

@@ -16,7 +16,8 @@
 
 - **Python**: Ruff (line-length 88, replaces black/flake8/isort)
 - **YAML**: yamllint with max line-length 88
-- Pre-commit hooks enforce all of the above — run `pre-commit run --all-files` to check
+- **Type checking**: ty (run `uv run ty check`)
+- Prek hooks enforce all of the above — run `uv run prek run --all-files` to check
 - Tests use `pytest` (config in `setup.cfg`, test files in `tests/`)
 
 ## Working with Catalogs
@@ -32,7 +33,9 @@
 ## Testing
 
 ```bash
-pytest --durations=20
+uv run pytest --durations=20
+# or run tests in parallel:
+uv run pytest -n auto
 ```
 
 - `test_remote_catalog.py` — Structural tests (catalog loads, entries have drivers/URLs)

AGENTS.md

@@ -1,4 +1,4 @@
-[![testing](https://github.com/aaronspring/remote_climate_data/actions/workflows/testing.yml/badge.svg)](https://github.com/aaronspring/remote_climate_data/actions/workflows/testing.yml) [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/aaronspring/remote_climate_data/master?urlpath=lab%2Ftree%2Fnotebooks%2Fdemo.ipynb) [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/aaronspring/remote_climate_data/master.svg)](https://results.pre-commit.ci/latest/github/aaronspring/remote_climate_data/master)
+[![testing](https://github.com/aaronspring/remote_climate_data/actions/workflows/testing.yml/badge.svg)](https://github.com/aaronspring/remote_climate_data/actions/workflows/testing.yml) [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/aaronspring/remote_climate_data/master?urlpath=lab%2Ftree%2Fnotebooks%2Fdemo.ipynb)
 
 # remote_climate_data
 a collection of remote climate data accessed via `intake` cached to disk
@@ -10,6 +10,13 @@ uv sync
 uv sync --all-extras
 ```
 
+## Run commands with uv
+```bash
+uv run pytest  # run tests
+uv run prek run --all-files  # run pre-commit hooks on all files
+uv run ty check  # run type checking
+```
+
 ## Catalog
 
 <details>

README.md

@@ -82,20 +82,6 @@ sources:
       urlpath: simplecache::https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.5.0.0.0.anomalies.ensemble_mean.nc
       chunks: {}
 
-  CRU_TS:
-    description: >-
-      Climatic Research Unit (CRU) Time-Series (TS) of high-resolution gridded data of
-      month-by-month variation in climate
-    metadata:
-      url: https://crudata.uea.ac.uk/cru/data/hrg
-      comment: >-
-        CEDA archive requires authorization, for setting up a certificate, please
-        follow instructions from
-        https://help.ceda.ac.uk/article/4712-reading-netcdf-with-python-opendap
-    args:
-      path: "{{CATALOG_DIR}}/atmosphere/cru_ts.yaml"
-    driver: intake.catalog.local.YAMLFileCatalog
-
   GISTEMP:
     description: GISS Surface Temperature Analysis (GISTEMP v4)
     metadata:

catalogs/atmosphere.yaml

@@ -69,81 +69,6 @@ sources:
       xarray_kwargs:
         engine: scipy
 
-  WOA2018:
-    description: World Ocean Atlas 2018
-    driver: netcdf
-    metadata:
-      url: https://www.nodc.noaa.gov/cgi-bin/OC5/woa18/woa18.pl
-      doi: https://www.nodc.noaa.gov/OC5/woa18/pubwoa18.html
-    parameters:
-      variable:
-        description: version release in year
-        type: str
-        default: temperature
-        allowed:
-          - temperature
-          - salinity
-          - oxygen
-          - phosphate
-          - nitrate
-          - AOU
-          # - silicate(i), o2sat(O) overlapping first string
-      DDDD:
-        description: decade
-        type: str
-        default: decav
-        allowed:
-          - decav  # average
-          - 5564  # 1955-1964
-          - 6574
-          - 7584
-          - 8594
-          - 95A4  # 1995-2004
-          - A5B7  # 2005-2017
-          - 81B0  # 1981-2010
-      TT:
-        description: time period
-        type: int
-        default: 0
-        allowed:
-          - 0  # annual
-          - 1  # Jan
-          - 2  # Feb
-          - 3  # Mar
-          - 4  # Apr
-          - 5  # May
-          - 6  # Jun
-          - 7  # Jul
-          - 8  # Aug
-          - 9  # Sep
-          - 10  # Oct
-          - 11  # Nov
-          - 12  # Dec
-          - 13  # winter
-          - 14  # spring
-          - 15  # summer
-          - 16  # autumn
-      GG:
-        description: grid
-        type: str
-        default: 1.00
-        allowed:
-          - 5.00  # 5°
-          - 1.00  # 1°
-          - 0.25  # 1/4°
-      GG2:
-        description: matching grid GG  # inconsistency in WOA2018 file naming
-        type: str
-        default: 01
-        allowed:
-          - 5d
-          - 01
-          - 04
-    args:
-      urlpath: https://www.ncei.noaa.gov/thredds-ocean/fileServer/ncei/woa/{{variable}}/{{DDDD}}/{{GG}}/woa18_{{DDDD}}_{{variable[0]}}{{"%02d"|format(TT)}}_{{GG2}}.nc
-      xarray_kwargs:
-        decode_times: false
-
   OISSTv21:
     driver: netcdf
     description: >-

catalogs/atmosphere/cru_ts.yaml

@@ -4,7 +4,9 @@ channels:
   - conda-forge
 dependencies:
   - python=3.12
+  - setuptools<70
   - intake<2
+  - intake-thredds=2024.3.1
   - xarray
   - pandas
   - intake-xarray
@@ -21,7 +23,8 @@ dependencies:
   - requests
   - pytest
   - pytest-sugar
-  - pre-commit
+  - pytest-xdist
+  - prek
   - ruff
   - jupyterlab
   # Visualization