Chunking requirements are too restrictive for optimal performance

[emmanuelmathot]

Summary

The current specification requires strict 1:1 mapping between Zarr chunks and tile matrix tiles (Section 9.7.4), which may prevent optimal chunking strategies for different data types and storage backends.

Current Problem

• Section 9.7.4 mandates "Chunks MUST match the tileWidth and tileHeight declared in the TileMatrix"
• This prevents optimization for different access patterns
• May not be optimal for cloud storage or specific analysis workflows
• Doesn't account for data-dependent optimal chunk sizes

Proposed Solution

1. Relax strict tile alignment requirements while maintaining compatibility
2. Allow flexible chunking that optimizes for the specific use case
3. Provide guidance on when stronger alignment is beneficial vs. when flexibility is preferred
4. Document chunk alignment strategies for different scenarios

Implementation Evidence

The EOPF implementation includes chunk alignment logic:

def calculate_aligned_chunk_size(dimension_size: int, target_chunk_size: int) -> int:
    """Calculate a chunk size that divides evenly into the dimension size."""
    if target_chunk_size >= dimension_size:
        return dimension_size
    
    # Find the largest divisor that is <= target_chunk_size
    for chunk_size in range(target_chunk_size, 0, -1):
        if dimension_size % chunk_size == 0:
            return chunk_size
    return 1

This approach:

• Prevents chunk overlap issues with Dask
• Optimizes for data dimensions rather than arbitrary tile sizes
• Maintains compatibility while improving performance

Specification Sections to Update

• Section 9.7.4 (Chunk Layout Alignment)
• Add performance considerations section
• Include guidance on chunk size optimization

cc @vincentsarago, @maxrjones, @d-v-b, @briannapagan

[geospatial-jeff]

From an IO perspective it is most optimal if the chunking of the Zarr does align with the TileMatrix it advertises. I do agree that the requirement that "Chunks MUST match the tileWidth and tileHeight declared in the TileMatrix" is too restrictive. Notably it prevents the writer from implementing "metatiling" where the physical chunks stored on disk align with many tiles, not just one.

A simple example with XYZ tiling scheme (2x decimation, 256x256 tile size). Rather than storing a single 256x256 chunk at zoom n you can store a 512x512 chunk at zoom n+1, where the chunk aligns with all 4 256x256 tiles at zoom n. Or a 1024x1024 chunk at zoom n+2, where the chunk aligns with all 16 256x256 tiles at zoom n.

Metatiling is the primary reason COGs are more performant than Zarr at scale. Zarr V3 has the notion of metatiling through chunk codecs and it would be a shame if GeoZarr couldn't take advantage of this by requiring that chunks always align with tiles one-to-one.

I believe this is slightly different than the proposed solution which allows the the user to define any chunking regardless of how well that chunking aligns with the tile matrix advertised by the Zarr file.

[d-v-b]

As zarr apis are generally robust to sub-optimal chunking (you will just do some unnecessary IO), it seems unnecessary to require a specific chunking arrangement in the spec. This might be better off in a non-normative section on performance.

[geospatial-jeff]

Minimizing extra IO really matters at scale! For both performance and cost!

Those reasons aside, it seems confusing to me if a GeoZarr file says "I conform to this tile matrix" but doesn't actually have chunks that conform to that tile matrix.