Error when compressing a large tree sequence: Codec does not support buffers of > 2147483647 bytes

[Jesson-mark]

Hi tskit-dev,

I am using tszip v0.2.6 to compress a tree sequence from the entire chromosome 1 in humans. The raw (uncompressed) tree sequence is 26 GB; but an error occured as shown below:

Traceback (most recent call last):
  File "/opt/miniforge3/envs/py3.10/bin/tszip", line 10, in <module>
    sys.exit(tszip_main())
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/tszip/cli.py", line 180, in tszip_main
    main(args)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/tszip/cli.py", line 174, in main
    run_compress(args)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/tszip/cli.py", line 128, in run_compress
    tszip.compress(ts, outfile, variants_only=args.variants_only)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/tszip/compression.py", line 103, in compress
    compress_zarr(ts, root, variants_only=variants_only)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/tszip/compression.py", line 259, in compress_zarr
    ).compress(root, compressor)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/tszip/compression.py", line 157, in compress
    compressed_array[:] = self.array
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/zarr/core.py", line 1449, in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/zarr/core.py", line 1545, in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/zarr/core.py", line 1935, in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/zarr/core.py", line 1988, in _set_selection
    self._chunk_setitem(chunk_coords, chunk_selection, chunk_value, fields=fields)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/zarr/core.py", line 2261, in _chunk_setitem
    self._chunk_setitem_nosync(chunk_coords, chunk_selection, value, fields=fields)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/zarr/core.py", line 2271, in _chunk_setitem_nosync
    self.chunk_store[ckey] = self._encode_chunk(cdata)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/zarr/core.py", line 2395, in _encode_chunk
    cdata = self._compressor.encode(chunk)
  File "numcodecs/blosc.pyx", line 559, in numcodecs.blosc.Blosc.encode
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/numcodecs/compat.py", line 151, in ensure_contiguous_ndarray
    ensure_contiguous_ndarray_like(buf, max_buffer_size=max_buffer_size, flatten=flatten)
  File "/opt/miniforge3/envs/py3.10/lib/python3.10/site-packages/numcodecs/compat.py", line 117, in ensure_contiguous_ndarray_like
    raise ValueError(msg)
ValueError: Codec does not support buffers of > 2147483647 bytes

Do you have any insights regarding this issue? Any input would be greatly appreciated.

Best regards,

Jie

[jeromekelleher]

This is a dupe of #69.

Can you give a bit more information about the tree sequence @Jesson-mark ?

The output of tskit info [trees file] and kastore ls -h [trees file] would help us see which table is causing the issue and help see if we can work around.

This is a bug that we should fix, though, and should be easy enough (we pick a different codec for arrays that have more the 2GiB of data).

[benjeffery]

We could also use the same codec but chunk the data? I think this is quite straight forward, although annoying to test. The chunking size could be configurable so tests can be meaningful on small data I guess.

[jeromekelleher]

Oh, so we would just need to make sure the Zarr chunks are less than 2G? That would be a good solution. I'm sure there's nothing to be gained from having them more than say 10M anyway, so would be easy enough to test.

[Jesson-mark]

Hi Jerome,

I used the tskit info command to obtain the information of my tree sequence. The output is as below:

╔═════════════════════════════╗
║TreeSequence                 ║
╠═══════════════╤═════════════╣
║Trees          │      144,950║
╟───────────────┼─────────────╢
║Sequence Length│2.4921201e+08║
╟───────────────┼─────────────╢
║Time Units     │      unknown║
╟───────────────┼─────────────╢
║Sample Nodes   │        2,148║
╟───────────────┼─────────────╢
║Total Size     │     31.4 GiB║
╚═══════════════╧═════════════╝
╔═══════════╤════════════╤════════╤════════════╗
║Table      │Rows        │Size    │Has Metadata║
╠═══════════╪════════════╪════════╪════════════╣
║Edges      │6.224153e+08│18.5 GiB│          No║
╟───────────┼────────────┼────────┼────────────╢
║Individuals│       2,148│58.8 KiB│          No║
╟───────────┼────────────┼────────┼────────────╢
║Migrations │           0│ 8 Bytes│          No║
╟───────────┼────────────┼────────┼────────────╢
║Mutations  │   1,151,700│40.6 MiB│          No║
╟───────────┼────────────┼────────┼────────────╢
║Nodes      │3.112098e+08│ 8.1 GiB│          No║
╟───────────┼────────────┼────────┼────────────╢
║Populations│           0│ 8 Bytes│          No║
╟───────────┼────────────┼────────┼────────────╢
║Provenances│           0│16 Bytes│          No║
╟───────────┼────────────┼────────┼────────────╢
║Sites      │   1,165,297│27.8 MiB│          No║
╚═══════════╧════════════╧════════╧════════════╝

Do you have any idea about this information?

[jeromekelleher]

Yep, that's a big ole tree sequence!

There's nothing to be done here other than fix the bug I think. It should be easy enough to do and hopefully we can get a release of tszip out with a fix quickly.

In the meantime, I think you will just have to use another utility to compress. Something like 7z should do pretty well.

[Jesson-mark]

That's great! Besides, if I compress this tree sequence using other softwares like 7z or zstd, can it be loaded into Python using tskit without decompressing it before?

[jeromekelleher]

No, you'll have to decompress it first I'm afraid.

[Jesson-mark]

Oh, thank you for your helpful suggestions!