Skip to content

[Bug]: speed up zstash update #409

@chengzhuzhang

Description

@chengzhuzhang

What happened?

Based on @wlin7 report:
When an archiving is stopped prematurely, zstash can resume with the updatefunction, but the zstash update can spend many hours (even days depending on simulation size) just to gather the files that have not been archived in previous runs of zstashfor the same simulation before resuming actual archiving. We need to dramatically cut down the time for this step by improving the inner working of zstash update, or adjusting the on-disk file management.

What machine were you running on?

chrysalis

Environment

latest in e3sm unified 1.12

Minimal Complete Verifiable Example (MCVE)

Relevant log output

Anything else we need to know?

@golaz point out changes need to be made at

def get_files_to_archive(cache: str, include: str, exclude: str) -> List[str]:

Metadata

Metadata

Assignees

No one assigned

    Labels

    semver: new featureNew feature (will increment minor version)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions