Skip to content

Add chunks='auto' support for cftime datasets #10527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
eb1a967
All works, just need to satisfy mypy and whatnot now
charles-turner-1 Jul 13, 2025
852476d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 13, 2025
c921c59
Merge branch 'main' into autochunk-cftime
charles-turner-1 Jul 13, 2025
1aba531
Fix moving import to be optional
charles-turner-1 Jul 13, 2025
9429c3d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 13, 2025
3c9d27e
Make mypy happy
charles-turner-1 Jul 13, 2025
5153d2d
Add some clarifying comments about what we need to do to optimise this
charles-turner-1 Jul 13, 2025
62e71e6
Merge branch 'autochunk-cftime' of https://github.com/charles-turner-…
charles-turner-1 Jul 14, 2025
cfdc31b
@dcherian's suggestions. Just need to update chunking strategy to res…
charles-turner-1 Jul 14, 2025
2f16bc7
Merge branch 'main' of https://github.com/charles-turner-1/xarray
charles-turner-1 Jul 14, 2025
ce720fa
Merge branch 'main' into autochunk-cftime
charles-turner-1 Jul 14, 2025
4fa58c1
Merge branch 'main' into autochunk-cftime
charles-turner-1 Jul 23, 2025
e58d6d7
Can now load cftime arrays with auto-chunking. Implementation still k…
charles-turner-1 Jul 23, 2025
590e503
Merge branch 'autochunk-cftime' of https://github.com/charles-turner-…
charles-turner-1 Jul 23, 2025
f953976
Test for autochunking when reading from disk
charles-turner-1 Jul 25, 2025
6706524
replace `build_chunkspec` with faking the dtype of a cftime array & a…
charles-turner-1 Jul 25, 2025
4e56acd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 25, 2025
0d008cd
Merge branch 'autochunk-cftime' of https://github.com/charles-turner-…
charles-turner-1 Jul 25, 2025
49c4e9c
Merge branch 'main' into autochunk-cftime
charles-turner-1 Jul 25, 2025
4594099
Merge branch 'autochunk-cftime' of https://github.com/charles-turner-…
charles-turner-1 Jul 25, 2025
5d00b0a
Remove redundant comments, rename things to make them clearer, add mo…
charles-turner-1 Jul 25, 2025
80421ef
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 25, 2025
d1f7ad3
Refactor to move most of the changes into the DaskManager
charles-turner-1 Jul 25, 2025
1b7de62
Merge branch 'autochunk-cftime' of https://github.com/charles-turner-…
charles-turner-1 Jul 25, 2025
4407185
bare-min tests should pass now?
charles-turner-1 Jul 25, 2025
d8f45b2
Deepak's suggestions (think mypy is still going to be angry for now)
charles-turner-1 Jul 28, 2025
20226c1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 28, 2025
11ac9f0
Merge branch 'autochunk-cftime' of https://github.com/charles-turner-…
charles-turner-1 Jul 28, 2025
8485df5
Fix errant line
charles-turner-1 Jul 28, 2025
2c27877
Clean up `DaskManager.rechunk` a bit - maybe possible to remove more …
charles-turner-1 Jul 28, 2025
0983261
Remove unused import
charles-turner-1 Jul 28, 2025
c4ec31f
Merge branch 'main' into autochunk-cftime
charles-turner-1 Aug 8, 2025
adbf5b2
Merge branch 'autochunk-cftime' of https://github.com/charles-turner-…
charles-turner-1 Aug 8, 2025
6c93bc4
Fix a couple of type errors
charles-turner-1 Aug 8, 2025
74bc0ea
Mypy & tests passing locally
charles-turner-1 Aug 8, 2025
0b9bbd0
Merge branch 'main' into autochunk-cftime
charles-turner-1 Aug 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions xarray/namedarray/daskmanager.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

import numpy as np

from xarray.core.common import _contains_cftime_datetimes
from xarray.core.indexing import ImplicitToExplicitIndexingAdapter
from xarray.namedarray.parallelcompat import ChunkManagerEntrypoint, T_ChunkedArray
from xarray.namedarray.utils import is_duck_dask_array, module_available
Expand All @@ -16,6 +17,7 @@
_NormalizedChunks,
duckarray,
)
from xarray.namedarray.parallelcompat import _Chunks

try:
from dask.array import Array as DaskArray
Expand Down Expand Up @@ -264,3 +266,56 @@ def shuffle(
if chunks != "auto":
raise NotImplementedError("Only chunks='auto' is supported at present.")
return dask.array.shuffle(x, indexer, axis, chunks="auto")

def rechunk( # type: ignore[override]
self,
data: T_ChunkedArray,
chunks: _NormalizedChunks | tuple[int, ...] | _Chunks,
**kwargs: Any,
) -> Any:
"""
Changes the chunking pattern of the given array.

Called when the .chunk method is called on an xarray object that is already chunked.

Parameters
----------
data : dask array
Array to be rechunked.
chunks : int, tuple, dict or str, optional
The new block dimensions to create. -1 indicates the full size of the
corresponding dimension. Default is "auto" which automatically
determines chunk sizes.

Returns
-------
chunked array

See Also
--------
dask.array.Array.rechunk
cubed.Array.rechunk
"""

if _contains_cftime_datetimes(data):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this can be deleted

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a play and I don't think I can fully get rid of it, I've reused as much of the abstracted logic as possible though.

# Preprocess chunks if they're cftime
cftime_nbytes_approx = 64
from dask import config as dask_config
from dask.utils import parse_bytes

target_chunksize = parse_bytes(dask_config.get("array.chunk-size"))

# Calculate total elements per chunk
elements_per_chunk = target_chunksize // cftime_nbytes_approx

# Distribute elements across dimensions
# Simple approach: try to make chunks roughly cubic
ndim = data.ndim # type:ignore
shape = data.shape # type:ignore
if ndim > 0:
chunk_size_per_dim = int(elements_per_chunk ** (1.0 / ndim))
chunks = tuple(min(chunk_size_per_dim, dim_size) for dim_size in shape)
else:
chunks = ()

return data.rechunk(chunks, **kwargs)
31 changes: 31 additions & 0 deletions xarray/tests/test_dask.py
Original file line number Diff line number Diff line change
Expand Up @@ -1060,6 +1060,29 @@ def make_da():
return da


def make_da_cftime():
yrs = np.arange(2000, 2120)
cftime_dates = xr.date_range(
start=f"{yrs[0]}-01-01",
end=f"{yrs[-1]}-12-31",
freq="1YE",
use_cftime=True,
)
yr_array = np.tile(cftime_dates.values, (10, 1))
da = xr.DataArray(
yr_array,
dims=["x", "t"],
coords={"x": np.arange(10), "t": cftime_dates},
name="a",
).chunk({"x": 4, "t": 5})
da.x.attrs["long_name"] = "x"
da.attrs["test"] = "test"
da.coords["c2"] = 0.5
da.coords["ndcoord"] = da.x * 2

return da


def make_ds():
map_ds = xr.Dataset()
map_ds["a"] = make_da()
Expand Down Expand Up @@ -1141,6 +1164,14 @@ def test_auto_chunk_da(obj):
assert actual.chunks == expected.chunks


@pytest.mark.parametrize("obj", [make_da_cftime()])
def test_auto_chunk_da_cftime(obj):
actual = obj.chunk("auto").data
expected = obj.data.rechunk({0: 10, 1: 120})
np.testing.assert_array_equal(actual, expected)
assert actual.chunks == expected.chunks


def test_map_blocks_error(map_da, map_ds):
def bad_func(darray):
return (darray * darray.x + 5 * darray.y)[:1, :1]
Expand Down
Loading