Skip to content

add suport for zstd compression of h5sc files #345

@sophiamaedler

Description

@sophiamaedler

I ran a quick benchmark for writing h5ad files with lzf compression vs zstd compression:

Dataset Write LZF (s) Write ZSTD (s) Read LZF (s) Read ZSTD (s) Read LZF (backed) Read ZSTD (backed) LZF .h5ad (GB) ZSTD .h5ad (GB)
Dataset A 63.79 74.60 31.14 19.30 28.94 18.03 2.9 2.1
Dataset B 64.28 70.41 30.83 19.37 28.46 18.04 2.8 2.0
Dataset C 61.01 72.50 30.68 18.93 28.31 17.40 2.8 2.0
Mean ± SD 63.03 ± 1.7 72.50 ± 2.1 30.88 ± 0.2 19.20 ± 0.2 28.57 ± 0.3 17.82 ± 0.3 2.83 ± 0.05 2.03 ± 0.05
Metric ZSTD / LZF ratio Relative difference Interpretation
Write time 1.15× +15 % slower Slight write-speed penalty
Read time 0.62× −38 % faster Much faster read
Read (backed) 0.62× −38 % faster Same gain in backed mode
File size 0.72× −28 % smaller ~30 % storage saving

zstd compression has the downside that we are dependent on an additional plugin (hdf5plugin) for both reading and writing making the files slightly less broadly compatible. Nevertheless it would be interesting to add zstd support to scPortrait as this could be quite relevant to some users.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions