Concurrent reading

[bicarlsen]

Is there a way to read a ZipArchive concurrently, whether from multiple threads or async?

[bicarlsen]

I guess async will be tough as most methods require io::Read + io::Seek, which it doesn't seem the async versions (e.g. tokio::fs::File) implement.

[Its-Just-Nans]

Hi

What would be your usage of multi thread reading (see https://stackoverflow.com/a/7632336 btw)

What would be your usage of async read?

[Its-Just-Nans]

Tokio has AsyncWrite and AsyncRead

[bicarlsen]

My archive consists of about 100k small files which are independent and need some light processing, so my thought is that by splitting the read operatons across multiple threads or using async I could process it faster.

I assumed that any time I call a reading function -- e.g. by_path -- it essentially acts as an io::Read + io::Seek.

Or is the entire file read into memory when the ZipArchive is created? Or am I completely misunderstanding everything? :P

[bicarlsen]

ping on this in case anybody has advise.

[bicarlsen]

After some discussion, I think I have some things clarified. It seems like a good way to allow parallelism would be to allow creating a new ZipZrchive with a new reader but sharing the ZipArchive::shared data so it doesn't have to be parsed again.

Perhaps something like

impl<R> ZipArchive<R> {
  fn metadata(&self) -> Arc<Shared> {
    self.shared.clone()
  }
  
  fn new_with_metadata<R: Read + Seek>(&self, reader: R, shared: Arc<Shared>) -> Self {
    Self {
      reader,
      shared,
    }
  }
}

This would leave it to the caller to ensure that the new reader is reading over something consistent with the shared metadata. It would enable something like

use rayon::prelude::*;
use std::fs::File;

fn read_data_par(path: impl AsRef<Path>) {
  let reader = File::open(&path).unwrap(); // some large archive, so parsing the shared data takes time
  let archive = ZipArchive::new(reader);
  let paths: Vec<PathBuf> = get_data_paths(); // some large vec of paths in the archive
  let data = paths
    .into_par_iter()
    .map_init(
      || {
        let reader = File::open(&path).unwrap();
        ZipArchive::new_with_metadata(reader, archive.metadata())
      },
      |archive, path| {
        let mut file = archive.by_path(path).unwrap();
        parse_data(&mut file)
      }
    )
    .collect();
    // ...
}

[bicarlsen]

@Its-Just-Nans, just wanted to ping this to get your thoughts.

[Its-Just-Nans]

Hi

Yes that seems a good idea. But I see two points

• I don't know if this will really improve the performances since it will be bound to I/O of the reading part
• It looks like the repository https://github.com/zip-rs/zip2 is in slow maintenance mode so I'm not sure if this can be added easily

I thought about something like on my side but your example seems better

use rayon::prelude::*;
use std::fs::File;

fn read_data_par(path: impl AsRef<Path>) {
  let reader = File::open(&path).unwrap(); // some large archive, so parsing the shared data takes time
  let archive = ZipArchive::new(reader);
  let data = archive.metadata().files
    .into_par_iter()
    .map(|(path_file, file)| {
        let mut file = archive.by_path(path_file).unwrap();
        parse_data(&mut file)
      }
    )
    .collect();
    // ...
}