RFC: Persistent Cache Snapshot

[jerrykingxyz]

Feature Name: Design the Snapshot module at persistent cache
Start Date: 2024/11/18

Summary

Design the Snapshot module at persistent cache

Context

There is a big difference in functionality between rspack Snapshot and webpack Snapshot.
For webpack Snapshot, it is mainly used to check whether the cache is available when accessing.
For rspack Snapshot is responsible for calculating the files updated or deleted during downtime. (PS. rspack cache availability check is implemented by incremental rebuild in the corresponding step)

User Guide

Option

Snaphot configuration items support immutable_paths , unmanaged_paths and managed_paths.

pub enum PathMatcher {
    String(String),
    Regexp(RspackRegex),
}

pub struct SnapshotOption {
    immutable_paths: Vec<PathMatcher>,
    unmanaged_paths: Vec<PathMatcher>,
    managed_paths: Vec<PathMatcher>,
}

Verification Strategy

Snapshot needs to support two types of file validation

• PackageVersion: Verify with the version field in package.json. When the file path is in managed_paths, PackageVersion will be used for verification.
• CompileTime: Verify the last compile time and file modification time, default strategy.

pub enum Strategy {
    PackageVersion(String),
    CompileTime(u64),
}

API

The Snapshot module supports adding/removing files and folders to the listening list. Through calc_changed_files, we can calculate which files have been modified and deleted.

struct Snapshot {
    storage: Arc<Storage>,
    option: SnapshotOption
}

impl Snapshot {
    pub fn add(&self, files: Vec<&PathBuf>) {
        ...
    }
    pub fn remove(&self, files: Vec<&PathBuf>) {
        ...
    }
    pub fn calc_changed_files(&self) -> (HashSet<PathBuf>, HashSet<PathBuf>) {
        // return modified file and deleted file
        ...
    }
}

Detailed design

Option

SnapshotOption provides is_immutable_path and is_managed_path to quick validation.

struct SnapshotOption {
    immutable_paths: Vec<PathMatcher>,
    unmanaged_paths: Vec<PathMatcher>,
    managed_paths: Vec<PathMatcher>,
}

impl SnapshotOption {
  pub fn is_immutable_path(&self, path_str: &str) -> bool {
    for item in &self.immutable_paths {
      if item.try_match(path_str) {
        return true;
      }
    }
    false
  }

  pub fn is_managed_path(&self, path_str: &str) -> bool {
    for item in &self.unmanaged_paths {
      if item.try_match(path_str) {
        return false;
      }
    }

    for item in &self.managed_paths {
      if item.try_match(path_str) {
        return true;
      }
    }
    false
  }
}

Strategy

StrategyHelper encapsulates Strategy related operations, providing compile_time , package_version generation functions, and also supports validate them.

struct StrategyHelper {
    fs: Arc<ReadableFileSystem>
}

pub enum ValidateResult {
  Deleted,
  Modified,
  NoChanged,
}

impl StrategyHelper {
    fn compile_time() -> Strategy {
        Strategy::CompileTime(SystemTime::now())
    }
    fn package_version(&self, path: &Path) -> Option<Strategy> {
        if let Ok(data) = self.fs.read(path.join("package.json")) {
            if let Ok(json) = serde::from_slice(data) {
                if let Some(v) = json.get("version") {
                    return Some(Strategy::PackageVersion(v));
                }
            }
        }
        
        let Some(parent) = path.parent() else {
            return None;
        }
        
        self.package_version(parent)
    }
    
    fn validate(&self, path: &Path, strategy: Strategy) -> ValidateResult {
        match strategy {
            Strategy::CompileTime(time) => {
                let meta = self.fs.metadata(path) else {
                    return ValidateResult::Deleted
                };
                if meta.mtime_ms > time {
                    ValidateResult::Modified
                } else {
                    ValidateResult::NoChanged
                }
            },
            Strategy::PackageVersion(version) => {
                let Some(Strategy::PackageVersion(cur)) = self.package_version(path) else {
                    return ValidateResult::Deleted
                }
                if cur != version {
                    ValidateResult::Modified
                } else {
                    ValidateResult::NoChanged
                }
            }
        }
    }
}

API

The implementation of Snapshot is relatively simple. It requires associating the Option and Strategy to achieve the following methods:

• add: Need to generate the corresponding verification strategy for the file and save them to Storage.
• remove: Delete file strategy directly from Storage
• calc_changed_files: Get all files in Storage, verify these files according to their verification strategy, and collect a list of changed and deleted files.

impl Snapshot {
    pub fn add(&self, files: Vec<&PathBuf>) {
        let helper = StrategyHelper::default();
        let compile_time = StrategyHelper::compile_time();
        for path in files {
            if self.option.is_immutable_path(path) {
                continue;
            }
            if self.option.is_managed_path(path) {
                if let Some(v) = helper.package_version() {
                    let bytes = rspack_cacheable.to_bytes(&v, &()).unwrap();
                    self.storage.set(path.as_bytes(), bytes);
                }
            }
            // default
            let bytes = rspack_cacheable.to_bytes(&compile_time, &()).unwrap();
            self.storage.set(path.as_bytes(), bytes);
        }
    }
    pub fn remove(&self, files: Vec<&PathBuf>) {
        for item in files {
            self.storage.remove(item.as_bytes())
        }
    }
    pub fn calc_changed_files(&self) -> (HashSet<PathBuf>, HashSet<PathBuf>) {
        let modified = HashSet::default();
        let removed = HashSet::default();
        let helper = StrategyHelper::default();
        for (key, value) in self.storage.get_all() {
            let path = PathBuf::from(String::from_utf8(key).unwrap());
            let strategy: Strategy = from_bytes::<Strategy, ()>(&value, &mut ()).unwrap();
            match helper.validate(&path, &strategy) {
                REMOVED => removed.insert(path),
                MODIFIED => modified.insert(path),
                _ => {}
            }
        }
        (modified, removed)
    }
}

CacheTrait

Snapshot needs to support incremental save(only save the newly added and delete dependent files when HMR finish), specifically, we need to call the add & remove method according to this incremental information when call after_compile.

impl Cache for PersistentCache {
    before_compile(&self, comp: &mut Compilation) {
        if comp.modified_file.is_empty() && comp.removed_file.is_empty() {
            let (m, r) = self.snapshot.calc_changed_files();
            comp.modified_file = m;
            comp.removed_file = r;
        }
    }
    
    after_compile(&self, comp: &Compilation) {
        let (_, added1, removed1) = comp.file_dependencies();
        let (_, added2, removed2) = comp.context_dependencies();
        ...
        let added = comp.modified_file.iter().chain(added1).chain(added2)...;
        let removed = comp.removed_file.iter().chain(removed1).chain(removed2)...;
        // should remove first
        self.snapshot.remove(removed);
        self.snapshot.add(added);
    }
}