write10xCounts() appears to rely on HDF5's default auto-chunking for the matrix/data and matrix/indices datasets, which for a large 1-D array resolves to a single chunk spanning nearly the whole dataset. Since HDF5's chunk cache defaults to 1 MiB, any reader that streams the matrix in bounded slices (rather than loading it all at once) can't retain a decompressed chunk between calls, so every small read redecompresses a possible multi-GB chunk from scratch.
A possible fix: When creating the matrix/data, matrix/indices (and similarly the barcodes/features columns), explicitly pass a bounded chunk size, e.g. chunk_dims = min(length(x), 1e6), instead of leaving it to HDF5's default auto-chunking, which for a single large 1-D vector tends to pick a chunk size close to the full dataset length.
An example for this would be anndataR's fix to address a similar issue scverse/anndataR#424
write10xCounts() appears to rely on HDF5's default auto-chunking for the matrix/data and matrix/indices datasets, which for a large 1-D array resolves to a single chunk spanning nearly the whole dataset. Since HDF5's chunk cache defaults to 1 MiB, any reader that streams the matrix in bounded slices (rather than loading it all at once) can't retain a decompressed chunk between calls, so every small read redecompresses a possible multi-GB chunk from scratch.
A possible fix: When creating the matrix/data, matrix/indices (and similarly the barcodes/features columns), explicitly pass a bounded chunk size, e.g. chunk_dims = min(length(x), 1e6), instead of leaving it to HDF5's default auto-chunking, which for a single large 1-D vector tends to pick a chunk size close to the full dataset length.
An example for this would be anndataR's fix to address a similar issue scverse/anndataR#424