Skip to content
This repository was archived by the owner on Jun 2, 2024. It is now read-only.

Conversation

cosmicexplorer
Copy link

@cosmicexplorer cosmicexplorer commented Sep 30, 2023

This is another response to zip-rs/zip2#165.

After demonstrating an approach to parallelize extracting zip archives using rayon threadpools in #407, I wanted to further investigate whether introducing an async API might provide a more scalable approach to introducing parallelism (as I noted in https://github.com/zip-rs/zip/issues/403#issuecomment-1728747153). This PR prototypes such an API.

Example Usage

See the doctest: https://github.com/zip-rs/zip/blob/bc601d744bf8f61cb3eda88d39610de55098b120/src/read/tokio.rs#L554-L576

Performance & Benchmarking

cargo bench -- io demonstrates that this async implementation is faster to extract for large archives!
> cargo bench -- io
io/big archive(84642745 bytes)/<async copy_buf>
                        time:   [29.463 ms 29.661 ms 29.865 ms]
                        thrpt:  [2.6396 GiB/s 2.6577 GiB/s 2.6756 GiB/s]
io/big archive(84642745 bytes)/<async copy (no buf)>
                        time:   [72.790 ms 73.435 ms 74.127 ms]
                        thrpt:  [1.0634 GiB/s 1.0735 GiB/s 1.0830 GiB/s]
io/big archive(84642745 bytes)/<sync copy (no buf)>
                        time:   [35.737 ms 35.870 ms 36.016 ms]
                        thrpt:  [2.1887 GiB/s 2.1976 GiB/s 2.2058 GiB/s]
io/small archive(126517 bytes)/<async copy_buf>
                        time:   [113.44 µs 114.66 µs 115.86 µs]
                        thrpt:  [1.0170 GiB/s 1.0276 GiB/s 1.0386 GiB/s]
io/small archive(126517 bytes)/<async copy (no buf)>
                        time:   [167.58 µs 169.65 µs 171.53 µs]
                        thrpt:  [703.41 MiB/s 711.20 MiB/s 720.00 MiB/s]
io/small archive(126517 bytes)/<sync copy (no buf)>
                        time:   [60.929 µs 61.357 µs 61.835 µs]
                        thrpt:  [1.9055 GiB/s 1.9204 GiB/s 1.9339 GiB/s]
io/random archive(10106802 bytes)/<async copy_buf>
                        time:   [4.2838 ms 4.3385 ms 4.4067 ms]
                        thrpt:  [2.1360 GiB/s 2.1696 GiB/s 2.1973 GiB/s]
io/random archive(10106802 bytes)/<async copy (no buf)>
                        time:   [9.4273 ms 9.5448 ms 9.6646 ms]
                        thrpt:  [997.31 MiB/s 1009.8 MiB/s 1022.4 MiB/s]
io/random archive(10106802 bytes)/<sync copy (no buf)>
                        time:   [3.9525 ms 3.9812 ms 4.0152 ms]
                        thrpt:  [2.3443 GiB/s 2.3643 GiB/s 2.3814 GiB/s]
extract/big archive(84642745 bytes)/<async extraction>
                        time:   [779.27 ms 801.90 ms 824.78 ms]
                        thrpt:  [97.870 MiB/s 100.66 MiB/s 103.59 MiB/s]
extract/big archive(84642745 bytes)/<sync extraction>
                        time:   [802.74 ms 803.94 ms 805.23 ms]
                        thrpt:  [100.25 MiB/s 100.41 MiB/s 100.56 MiB/s]
extract/small archive(126517 bytes)/<async extraction>
                        time:   [9.5155 ms 9.5789 ms 9.6337 ms]
                        thrpt:  [12.524 MiB/s 12.596 MiB/s 12.680 MiB/s]
extract/small archive(126517 bytes)/<sync extraction>
                        time:   [2.3227 ms 2.3490 ms 2.3713 ms]
                        thrpt:  [50.881 MiB/s 51.365 MiB/s 51.946 MiB/s]
extract/random archive(10106802 bytes)/<async extraction>
                        time:   [160.18 ms 164.26 ms 169.09 ms]
                        thrpt:  [57.004 MiB/s 58.681 MiB/s 60.173 MiB/s]
extract/random archive(10106802 bytes)/<sync extraction>
                        time:   [37.677 ms 37.850 ms 38.045 ms]
                        thrpt:  [253.35 MiB/s 254.65 MiB/s 255.82 MiB/s]

Notably, this approach is already faster than the sync implementation, without even applying the optimizations requiring a cloneable file handle that were necessary in #407!

TODO

This is left as a draft because it adds a lot of extra dependencies, which I would like to avoid. It also directly depends on tokio, instead of using the async-executors crate to apply to any async executor. I'm going to spend some time to break out the extra code I had to write to make this async API work into a separate wrapper crate, which should allow this change to become much smaller and easier for maintainers to accept.

@Pr0methean
Copy link
Member

Replaced with zip-rs/zip2#73.

@Pr0methean Pr0methean closed this May 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants