-
Notifications
You must be signed in to change notification settings - Fork 7
Open
Labels
enhancementNew feature or requestNew feature or request
Description
I ran a quick benchmark for writing h5ad files with lzf compression vs zstd compression:
| Dataset | Write LZF (s) | Write ZSTD (s) | Read LZF (s) | Read ZSTD (s) | Read LZF (backed) | Read ZSTD (backed) | LZF .h5ad (GB) | ZSTD .h5ad (GB) |
|---|---|---|---|---|---|---|---|---|
| Dataset A | 63.79 | 74.60 | 31.14 | 19.30 | 28.94 | 18.03 | 2.9 | 2.1 |
| Dataset B | 64.28 | 70.41 | 30.83 | 19.37 | 28.46 | 18.04 | 2.8 | 2.0 |
| Dataset C | 61.01 | 72.50 | 30.68 | 18.93 | 28.31 | 17.40 | 2.8 | 2.0 |
| Mean ± SD | 63.03 ± 1.7 | 72.50 ± 2.1 | 30.88 ± 0.2 | 19.20 ± 0.2 | 28.57 ± 0.3 | 17.82 ± 0.3 | 2.83 ± 0.05 | 2.03 ± 0.05 |
| Metric | ZSTD / LZF ratio | Relative difference | Interpretation |
|---|---|---|---|
| Write time | 1.15× | +15 % slower | Slight write-speed penalty |
| Read time | 0.62× | −38 % faster | Much faster read |
| Read (backed) | 0.62× | −38 % faster | Same gain in backed mode |
| File size | 0.72× | −28 % smaller | ~30 % storage saving |
zstd compression has the downside that we are dependent on an additional plugin (hdf5plugin) for both reading and writing making the files slightly less broadly compatible. Nevertheless it would be interesting to add zstd support to scPortrait as this could be quite relevant to some users.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request