fix(py): fix large pyramidal TIFF conversion performance and crashes #342
+737
−81
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes excessive memory usage and slow graph construction when converting large tiled TIFF files (e.g. 3GB JPEG2000 OME-TIFF with 512x512 tiles). The root cause was that the default 128px output chunks (for 3D images) are smaller than typical input tiles (512px), causing a 16x task count explosion during dask rechunking. A 100K-task input became 1.6M+ tasks, consuming 2GB RAM just for graph construction. Changes: - Enable the existing task_count check (was commented out) to trigger disk caching when task count exceeds config.task_target (50K) - Auto-detect tiled input sources and use max(default, input_chunk) for spatial dimensions to avoid unnecessary rechunking - Preserve channel dimension chunking (don't split RGB from 3 to 1) - Extract _find_optimal_chunk_size to module-level function - Add _cache_2d_strips for strip-based caching of large 2D images (replaces the TODO at the old line 289) - Add _cache_1d_segments for 1D image caching edge case - Add comprehensive tests for all new functionality Refs: #310, dask/dask#8570
When tifffile opens a multi-level pyramidal TIFF (e.g., OME-TIFF with pyramid levels), it returns a zarr Group instead of an Array. This caused an AttributeError on 'Group' object has no attribute 'ndim' in to_ngff_image(). Extract the full-resolution array from the Group automatically, using multiscales metadata when available or falling back to the largest array by size.
When converting pyramidal OME-TIFFs that already contain multiple resolution levels, reuse the pre-built pyramid directly instead of regenerating all levels from scratch via expensive downsampling. This avoids Gaussian blur + downsample computation entirely and eliminates redundant reads of the full-resolution data. For a 3GB JPEG2000 OME-TIFF with 4 pyramid levels, this reduces total work from ~131B pixels (10 recomputed levels) to ~84B pixels (4 existing levels, read-and-write only). Also refactors CLI metadata override logic into a reusable _apply_cli_metadata_overrides() function.
thewtex
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes conversion of large pyramidal TIFF files (e.g., JPEG2000 OME-TIFF) that previously caused excessive memory usage, task graph explosion, and hour-long processing times.
Resolves #310
Problem
Converting a 3GB JPEG2000 OME-TIFF (
602a12_z_stack.qupath.j2k.ome.tif) with 512×512 tiles was:AttributeError: 'Group' object has no attribute 'ndim'— pyramidal TIFFs return azarr.Group, not anArrayChanges
1. Handle zarr Groups from pyramidal TIFFs (
to_ngff_image.py)_extract_array_from_group()to extract the full-resolution array from azarr.Group, usingmultiscalesmetadata when available or falling back to the largest arrayto_ngff_image()now transparently handleszarr.Groupinputs2. Reduce dask task explosion (
to_multiscales.py)task_count()guard that was commented out_find_optimal_chunk_size()to module level for reuse_cache_2d_strips()and_cache_1d_segments()for strip-based caching of 2D/1D large images (previously a TODO)3. Reuse existing pyramid levels (
cli.py)_multiscales_from_tifffile_pyramid(): when a TIFF already contains multiple resolution levels, buildsMultiscalesdirectly from them instead of regenerating viato_multiscales()_next_scale_metadata()_apply_cli_metadata_overrides()out of_ngff_image_to_multiscales()for reuse by both code pathsto_multiscales()pathPerformance impact
For the 3GB test TIFF (shape: 3×57128×153122×3, 4 pyramid levels, 512×512 JPEG2000 tiles):
Testing
test_large_image_chunking.pycovering input-aligned chunks, channel preservation, 2D strip caching, and 1D segment caching