WIP: Support icechunk stores #96

jbusecke · 2025-09-18T21:08:15Z

I am working towards icechunk (for now likely just native) support in titler-multidim here.

Depends on #95 and developmentseed/titiler#1235

Tasks

Updated zarr to v3
Modified the test fixture scripts to write v2 and v3 zarr stores explicitly (I retriggered them and committed the new fixtures)
Add scripts/fixtures for native icechunk
Deploy and test on real-world native icechunk data
Add scripts/fixtures for virtual icechunk
Deploy and test on real-world virtual icechunk data

Co-authored-by: Henry Rodman <[email protected]>

jbusecke · 2025-09-18T21:14:18Z

I noticed that some tests fail with

E           TypeError: cannot pickle 'module' object

in the XarrayReader

This seems to only happen with caching is enabled (and only for the netcdf store)! To show this I have added an additional parametrization to the app test fixture which runs all tests both with and without cache.

I did confirm that this error is introduced with the upgrade of either xarray or zarr here. When I just add the changes to conftest.py to main, all tests pass, so this was not a silent failure.

Ill move on here for now, but I would love to fix this issue here. @hrodmn do you have any intuition why the caching would not play nice with a new xarray or zarr version?

…to dev deps

jbusecke · 2025-09-18T22:07:25Z

Ok making some progress here, but Ill need some input from folks.

I have refactored the test datasets for zarr v2 and v3. The reading in seems to work (since the tests to get the variables only do pass), but I have some questions about the expected responses.

when I run the test_get_info pytest I am not getting the expected response (left is returned and right is expected).

FAILED tests/test_app.py::test_get_info[without_cache-zarr_store_v3] - AssertionError: assert {'bounds': [-180.0, -90.0, 180.0, 90.0], 'crs': 'http://www.opengis.net/def/crs/EPSG/0/4326', 'band_metadata': [['b1', {}]], 'band_descriptions': [['b1', '0']], 'dtype': 'float64', 'nodata_type': 'Nodata', 'name': 'CDD0', 'count': 1, 'width': 72, 'height': 36, 'dimensions': ['y', 'x'], 'attrs': {}} == {'bounds': [-180, -90.0, 180.0, 90.0], 'band_metadata': [], 'band_descriptions': [], 'dtype': 'float64', 'nodata_type': 'None', 'height': 36, 'count': 1, 'width': 72, 'attrs': {}, 'name': 'CDD0'}
  
  Common items:
  {'attrs': {},
   'bounds': [-180.0, -90.0, 180.0, 90.0],
   'count': 1,
   'dtype': 'float64',
   'height': 36,
   'name': 'CDD0',
   'width': 72}
  Differing items:
  {'nodata_type': 'Nodata'} != {'nodata_type': 'None'}
  {'band_descriptions': [['b1', '0']]} != {'band_descriptions': []}
  {'band_metadata': [['b1', {}]]} != {'band_metadata': []}
  Left contains 2 more items:
  {'crs': 'http://www.opengis.net/def/crs/EPSG/0/4326', 'dimensions': ['y', 'x']}
  
  Full diff:
    {
        'attrs': {},
  -     'band_descriptions': [],
  ?                           --
  +     'band_descriptions': [
  +         [
  +             'b1',
  +             '0',
  +         ],
  +     ],
  -     'band_metadata': [],
  ?                       --
  +     'band_metadata': [
  +         [
  +             'b1',
  +             {},
  +         ],
  +     ],
        'bounds': [
  -         -180,
  +         -180.0,
  ?             ++
            -90.0,
            180.0,
            90.0,
        ],
        'count': 1,
  +     'crs': 'http://www.opengis.net/def/crs/EPSG/0/4326',
  +     'dimensions': [
  +         'y',
  +         'x',
  +     ],
        'dtype': 'float64',
        'height': 36,
        'name': 'CDD0',
  -     'nodata_type': 'None',
  ?                       ^^
  +     'nodata_type': 'Nodata',
  ?                       ^^^^
        'width': 72,
    }

I am not exactly sure how these come about TBH.

Are 'band_metadata', nodata_type, dimensions... changes that have been made to main recently? Or is this something that comes from xarray?

Happy to just rewrite the expected output, but I first wanted to understand exactly what is going on.

All I could gather here was that these response fields are expected for e.g. the netcdf output.

hrodmn · 2025-10-08T16:12:54Z

pyproject.toml

    "s3fs",
-    "xarray",
-    "zarr>=2,<3",
+    "xarray>2025.07.1",


The pickling problem is solved if we bump this up to 2025.10.1!

jbusecke · 2025-10-08T18:59:24Z

Ok I was able to pass the simple tests (including native icechunk) when running from a local modified version of titiler.xarray (now exists as PR here).

Just a note to myself how to link uv to a local file.

In pyproject.toml:

[tool.uv.sources]
titiler-xarray = { path = "../titiler/src/titiler/xarray", editable = true }

jbusecke · 2025-10-08T19:32:29Z

Almost have all tests passing, but I am not sure how to interpret these:

Is there any way to enable the full error trace to be displayed?

jbusecke · 2025-10-08T19:48:08Z

Ah thanks to @hrodmn I was able to look at the response detail and get:

{"detail":"Bounds and transform are inconsistent"}

currently stuck here. I suspect that this is somehow related to upstream changes ( for example here: #96 (comment)) or the way I have regenerated some of the datasets (less likely since this is happening for every test dataset and I only replaced the zarr/icechunk ones).

Julius Busecke and others added 5 commits September 18, 2025 14:02

Parametrize test parameters across test functions

441063e

Fix pre-commit failure

5d760bc

Fix pyramid store detection in get_tile test

b87582d

Update tests/test_app.py

7890228

Co-authored-by: Henry Rodman <[email protected]>

updated zarr; added zarr v3 test fixture; parametrize caching for tests

ad28eab

Julius Busecke added 2 commits September 18, 2025 17:20

Remove dask dependency

9161be4

updated zarr fixtures+scripts; minor test changes+additions;add dask …

a018066

…to dev deps

update tilejson expected responses

3e57c11

jbusecke mentioned this pull request Sep 22, 2025

Zarr v3 + updated xarray cause TypeError: cannot pickle 'module' object #97

Open

jbusecke self-assigned this Sep 23, 2025

hrodmn reviewed Oct 8, 2025

View reviewed changes

Julius Busecke added 2 commits October 8, 2025 13:14

All tests passing with zarr v2 and v3

4d20b59

add icechunk fixture and generation script

21f33d1

jbusecke mentioned this pull request Oct 8, 2025

WIP: Enable icechunk reading - rough first try developmentseed/titiler#1235

Draft

Julius Busecke and others added 3 commits October 8, 2025 15:19

Remove debugs, fix most tests

e6d86ea

Merge branch 'main' into support-icechunk

d99859c

Bump xarray + renable testing with cache

791d618

Some more debugging of the tile test

e1d90bf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Support icechunk stores #96

WIP: Support icechunk stores #96

jbusecke commented Sep 18, 2025 •

edited

Loading

Uh oh!

jbusecke commented Sep 18, 2025 •

edited

Loading

Uh oh!

jbusecke commented Sep 18, 2025 •

edited

Loading

Uh oh!

hrodmn Oct 8, 2025

Uh oh!

jbusecke commented Oct 8, 2025

Uh oh!

jbusecke commented Oct 8, 2025

Uh oh!

jbusecke commented Oct 8, 2025

Uh oh!

Uh oh!

WIP: Support icechunk stores #96

Are you sure you want to change the base?

WIP: Support icechunk stores #96

Conversation

jbusecke commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbusecke commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbusecke commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hrodmn Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

jbusecke commented Oct 8, 2025

Uh oh!

jbusecke commented Oct 8, 2025

Uh oh!

jbusecke commented Oct 8, 2025

Uh oh!

Uh oh!

jbusecke commented Sep 18, 2025 •

edited

Loading

jbusecke commented Sep 18, 2025 •

edited

Loading

jbusecke commented Sep 18, 2025 •

edited

Loading