Will V3 compressor codecs require all parameters in array metadata?

[jhamman]

Zarr version

v2.14.3

Numcodecs version

v0.11.0

Python Version

3.10

Operating System

Mac

Installation

pip

Description

The current V3 implementation seems to require a more specific set of parameters for codecs/compressors than V2 did. The code sample below demonstrates how all configuration parameters are extracted separately:

zarr-python/zarr/meta.py

Lines 443 to 460 in b3cafa9

	if meta['codec'].startswith(uri + 'gzip/'):
	codec = numcodecs.GZip(level=conf['level'])
	elif meta['codec'].startswith(uri + 'zlib/'):
	codec = numcodecs.Zlib(level=conf['level'])
	elif meta['codec'].startswith(uri + 'blosc/'):
	codec = numcodecs.Blosc(clevel=conf['clevel'],
	shuffle=conf['shuffle'],
	blocksize=conf['blocksize'],
	cname=conf['cname'])
	elif meta['codec'].startswith(uri + 'bz2/'):
	codec = numcodecs.BZ2(level=conf['level'])
	elif meta['codec'].startswith(uri + 'lz4/'):
	codec = numcodecs.LZ4(acceleration=conf['acceleration'])
	elif meta['codec'].startswith(uri + 'lzma/'):
	codec = numcodecs.LZMA(format=conf['format'],
	check=conf['check'],
	preset=conf['preset'],
	filters=conf['filters'])

Whereas in V2, the compressor parameters were simply passed on to numcodecs.get_config

zarr-python/zarr/meta.py

Lines 169 to 171 in b3cafa9

	dtype=cls.encode_dtype(dtype),
	compressor=meta["compressor"],
	fill_value=cls.encode_fill_value(meta["fill_value"], dtype, object_codec),

Question:

Was this an intentional change? If so, I foresee a number of downstream projects that will need to adjust their expectations (e.g. Kerchunk, Tifffile).

Steps to reproduce

I ran into this working on a V2->V3 conversion tool for Kerchunk. A minimal example is not available at this time.

xref: cgohlke/tifffile#211

[rabernat]

Joe could you show the codec metadata for your specific use case? What is the codec name and configuration parameters?

[jhamman]

Coming from Kerchunk/tifffile, the compressor metadata only includes non-default fields. An example:

"compressor": {\n  "id": "lzma"\n }

whereas Zarr is trying to unpack this as:

             numcodecs.LZMA(format=conf['format'], 
                            check=conf['check'], 
                            preset=conf['preset'], 
                            filters=conf['filters'])

[rabernat]

This would be easily fixed by just relaxing the V3 implementation's expectations back to how V2 does it.

[jhamman]

cc @grlee77 (and others) - I've put a PR up for discussion that reverts this strict codec unpacking to something that looks closer to the V2 implementation - #1447. Comments welcome!