Limit on number of VlMeta entries

[leoyala]

Hello,

I have been using NDArrays to store my data, and I have also been using the variable metadata layer to store additional information related to my data. I started to run into an error that I don't understand where it comes from, in some conditions, when I add more than 16 entries to the variable length metadata layer, I get a Runtime Error (see details below). I am adding below a minimal example for reproduction. This only seems to be the case when storing to disk. When the array is kept in memory, the error does not occur. Is this intended behavior? Or am I perhaps doing something wrong?

General info:

python version: 3.10.11
blosc2 version: 2.6.2
msgpack version: 1.1.0
System: Ubuntu 22.04

Other things that I tried:

pip install blosc2==2.6.2 --force-reinstall --no-cache

This does not work:

import blosc2
import numpy as np
cparams = dict(
    codec=blosc2.Codec.ZSTD,
    clevel=1,
    filters=[blosc2.Filter.SHUFFLE, blosc2.Filter.BYTEDELTA],
)
x = np.random.rand(100)
a = blosc2.asarray(
                x,
                chunks=(10,),
                urlpath="test.b2nd",
                mode="w",
                cparams=cparams,
            )
for i in range(20):
    try:
        a.schunk.vlmeta[f"metadata_{i}"] = 0
    except RuntimeError as err:
        print(f"Error at iteration: {i}")
        raise err

Error message:

Error at iteration: 16

Cell In[3], line 18
     16 for i in range(20):
     17     try:
---> 18         a.schunk.vlmeta[f"metadata_{i}"] = 0
     19     except RuntimeError as err:
     20         print(f"Error at iteration: {i}")

File ~/anaconda3/envs/sds/lib/python3.10/site-packages/blosc2/schunk.py:35, in vlmeta.__setitem__(self, name, content)
     28 cparams = {"typesize": 1}
     29 content = packb(
     30     content,
     31     default=blosc2_ext.encode_tuple,
     32     strict_types=True,
     33     use_bin_type=True,
     34 )
---> 35 super().set_vlmeta(name, content, **cparams)

File blosc2_ext.pyx:1613, in blosc2.blosc2_ext.vlmeta.set_vlmeta()

This works:

import blosc2
import numpy as np

x = np.random.rand(100)
a = blosc2.asarray(
                x,
                chunks=(10,),
                cparams=cparams,
            )
for i in range(100):
    try:
        a.schunk.vlmeta[f"metadata_{i}"] = 0
    except RuntimeError as err:
        print(f"Error at iteration: {i}")
        raise err