Projection task pyramid building: downsampling error in edge-case

[jluethi]

For a Zarr with some non-typical dimensions, the current projection task fails in building pyramids for the MIP.

Error:

2025-01-17 21:58:15,818; INFO; START projection task
2025-01-17 21:58:15,818; INFO; init_args.origin_url='/path/to/fractal_output_more/R0/day3p5_p1_C04/r0.zarr/C/04/0'
2025-01-17 21:58:15,818; INFO; zarr_url='/path/to/fractal_output_more/R0/day3p5_p1_C04/r0_mip.zarr/C/04/0'
2025-01-17 21:58:15,818; INFO; method=<DaskProjectionMethod.MIP: 'mip'>
2025-01-17 21:58:15,872; INFO; Source on_disk_shape=(4, 146, 1945, 3260)
2025-01-17 21:58:15,872; INFO; Destination dest_on_disk_shape=[4, 1, 1945, 3260]
2025-01-17 21:58:15,872; INFO; New shape: dest_on_disk_shape=(4, 1, 1945, 3260)
/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/zarr/storage.py:455: FutureWarning: missing object_codec for object array; this will raise a ValueError in version 3.0
  _init_array_metadata(
2025-01-17 21:58:16,568; INFO; New Projection image created - new_ngff_image=NGFFImage(group_path=/path/to/fractal_output_more/R0/day3p5_p1_C04/r0_mip.zarr/C/04/0/, 
          paths=['0', '1', '2', '3', '4'], 
          labels=[], 
          tables=['FOV_ROI_table', 'well_ROI_table'], 
)
Traceback (most recent call last):
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/fractal_tasks_core/tasks/projection.py", line 144, in <module>
    run_fractal_task(
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/fractal_tasks_core/tasks/_utils.py", line 79, in run_fractal_task
    metadata_update = task_function(**pars)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/pydantic/validate_call_decorator.py", line 60, in wrapper_function
    return validate_call_wrapper(*args, **kwargs)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/pydantic/_internal/_validate_call.py", line 96, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/fractal_tasks_core/tasks/projection.py", line 110, in projection
    new_image.consolidate()
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/ngio/core/image_handler.py", line 228, in consolidate
    self._consolidate(order=order)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/ngio/core/image_like_handler.py", line 542, in _consolidate
    on_disk_zoom(
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/ngio/pipes/_zoom_utils.py", line 152, in on_disk_zoom
    target_array.to_zarr(target)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/array/core.py", line 2978, in to_zarr
    return to_zarr(self, *args, **kwargs)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/array/core.py", line 3825, in to_zarr
    return arr.store(
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/array/core.py", line 1777, in store
    r = store([self], [target], **kwargs)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/array/core.py", line 1245, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/base.py", line 397, in compute_as_if_collection
    return schedule(dsk2, keys, **kwargs)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/threaded.py", line 91, in get
    results = get_async(
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/local.py", line 516, in get_async
    raise_exception(exc, tb)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/local.py", line 324, in reraise
    raise exc
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/local.py", line 229, in execute_task
    result = task(data)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/_task_spec.py", line 745, in __call__
    return self.func(*new_argspec)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/_task_spec.py", line 171, in _execute_subgraph
    res = execute_graph(final, keys=[outkey])
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/_task_spec.py", line 984, in execute_graph
    cache[key] = node(cache)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/_task_spec.py", line 745, in __call__
    return self.func(*new_argspec)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/array/core.py", line 4539, in store_chunk
    return load_store_chunk(x, out, index, lock, return_stored, False)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/dask/array/core.py", line 4521, in load_store_chunk
    out[index] = x
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/zarr/core.py", line 1447, in __setitem__
    self.set_orthogonal_selection(pure_selection, value, fields=fields)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/zarr/core.py", line 1636, in set_orthogonal_selection
    self._set_selection(indexer, value, fields=fields)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/zarr/core.py", line 1962, in _set_selection
    check_array_shape("value", value, sel_shape)
  File "/path/to/FRACTAL_TASK_DIR/7/fractal-tasks-core/1.4.2/venv/lib/python3.10/site-packages/zarr/util.py", line 561, in check_array_shape
    raise ValueError(
ValueError: parameter 'value': expected array with shape (1, 1, 122, 204), got (1, 1, 121, 204)

Zarr specification:

{
    "chunks": [
        1,
        10,
        1000,
        1000
    ],
    "compressor": {
        "blocksize": 0,
        "clevel": 5,
        "cname": "lz4",
        "id": "blosc",
        "shuffle": 1
    },
    "dimension_separator": "/",
    "dtype": "<u2",
    "fill_value": 0,
    "filters": null,
    "order": "C",
    "shape": [
        4,
        146,
        1945,
        3260
    ],
    "zarr_format": 2
}

[jluethi]

It's confirmed to run fine with 1.3.3 of tasks core, but fails with 1.4.2.

Also, initial tests by @lorenzocerrone show that it appears to be chunking dependent, e.g. the same size OME-Zarr only runs into this issue in ngio with certain chunking configurations. More analysis of this to come later :)

[lorenzocerrone]

This issue was addressed in PR #28 and a patch was released in ngio==0.1.6.

I do not yet fully understand the root cause of the bug. It appears to be related to an issue with Dask array chunking, where chunk sizes become unknown due to indexing. My current fix is to explicitly call compute_chunk_sizes() on the array.

[jluethi]

=> Any new installation of task core should not have this issue anymore. But existing installations of 1.4.2 task-core are still using ngio 0.1.5
@nrepina this should get auto resolved the next time we install a new task core version at FMI :)