Can zstash transfer tars without extracting?

[forsyth2]

Question criteria

• I added a descriptive title here.
• I searched the zstash GitHub Discussions to find a similar question and didn't find it.
• I searched the zstash documentation.

What is the deadline?

N/A

Describe your question

From @TonyB9000:

Does zstash (automatically) provide a way to bring over only the tarfiles needed (containing land and ocean/sea-ice) data? I don’t want to extract the data (yet).
Put another way, can zstash act (automatically) to first transfer the index.db, determine the required tar-files, and then globus-transfer ONLY those tarfiles? This seems unlikely to me, as it would require providing a list of file-matching patterns to determine the needed tar-flles, while not acting to extract the data on arrival of the tar-files.
I suppose I could have manually transferred only the index.db, plied it for each of the “patterns” of interest (archive/lnd/hist/elm.h0, archive/ocean/hist/timeseriesMonthly, etc etc.) and then determined which of the 180 tarfiles I need to transfer.
I have been under the impression that zstash would pull over ALL tarfiles for an archive, and only thereafter use the index.db to extract what the user specified. This is likely a misconception on my part.

Are there are any possible answers you came across?

Usage docs are here. I was thinking some combination of --tars and --include could handle this, but the fundamental issue is that there's no way to transfer tars without extracting from them.

Potential workflow:

# 1. Get index.db and find the relevant tars. 
# index.db is automatically retrieved on initial run of `zstash ls` to a HPSS path. 
# Use `--tars` to list the tars containing these files.
zstash ls --hpss=<path> --tars "archive/lnd/hist/*elm.h0*" > land_files.txt
zstash ls --hpss=<path> --tars "archive/ocean/hist/*timeseriesMonthly*" > ocean_files.txt

# 2. Transfer the tars
# Fundamental limitation: you'll need to manually transfer just those tars 
# (listed in the "Tars:" section of the output) via Globus/hsi.
# Transfer both the tar files AND index.db to your cache directory.

# 3. Once desired, you can extract from local tars:
zstash extract --hpss=none --cache=<cache> "archive/lnd/hist/*elm.h0*"
zstash extract --hpss=none --cache=<cache> "archive/ocean/hist/*timeseriesMonthly*"

What machine were you running on?

N/A

Environment

N/A

Minimal Complete Verifiable Example (MCVE)

No response

Relevant log output

No response

Anything else we need to know?

No response

[forsyth2]

@TonyB9000 Further comments:

Does zstash (automatically) provide a way to bring over only the tarfiles needed (containing land and ocean/sea-ice) data? I don’t want to extract the data (yet).

See workflow described above. Unfortunately, it requires a manual step.

can zstash act (automatically) to first transfer the index.db, determine the required tar-files, and then globus-transfer ONLY those tarfiles?

No, that is the fundamental limitation described above. It's always going to extract requested files after transferring the tars.

I suppose I could have manually transferred only the index.db, plied it for each of the “patterns” of interest (archive/lnd/hist/elm.h0, archive/ocean/hist/timeseriesMonthly, etc etc.) and then determined which of the 180 tarfiles I need to transfer.

This is essentially the workflow described above, except there is some automation ability using zstash ls --tars.

I have been under the impression that zstash would pull over ALL tarfiles for an archive, and only thereafter use the index.db to extract what the user specified

No, that can't be the case, otherwise zstash extract --tars would offer no performance benefit.

[TonyB9000]

@forsyth2 You wrote "It's always going to extract requested files after transferring the tars."

I thought that "zstash check" (with --keep) would pull over an archive of tar-files without extracting any files, and leave the tarfiles alone. But without supplying specific file-patterns, there is no way to determine which tar-files to include.

To make the "--tars" useful, there needs to be a function that converts an extraction (file-matching) pattern to a list of tarfiles that would be involved. Even a two-step process could be a big time-saver, if one needs only a few of the (say) 200 tarfiles of an archive.

The real drawback of forcing file extraction is that there is little opportunity to direct the results of each extraction pattern to a different file-system destination. You are at the mercy of the tarred path/file names. I need to be able to trust that a set of patterns have no intersection. This is reasonable for a single archive, but may not hold up across multiple archives.

[forsyth2]

I thought that "zstash check" (with --keep) would pull over an archive of tar-files without extracting any files, and leave the tarfiles alone.

Oh yes of course, that's correct. I was too focused on extract.

Does this work for your purposes?

# Download only the tars containing land files (without extracting)
zstash check --hpss=<path> --keep "archive/lnd/hist/*elm.h0*"

# Download only the tars containing ocean files (without extracting) 
zstash check --hpss=<path> --keep "archive/ocean/hist/*timeseriesMonthly*"

But without supplying specific file-patterns, there is no way to determine which tar-files to include.

I'm not following why this is an issue -- something would need to be supplied for zstash to figure out which files the user wants.

little opportunity to direct the results of each extraction pattern to a different file-system destination

I think that would require a large change to zstash's structure. It seems most convenient to simply mv post-extract.

[TonyB9000]

@forsyth2 This would work, in principle. The only issue that remains is as follows: Suppose both "land" and "ocean" files are (somehow) mixed within some of the same tar-files. I first ask for "land", obtain the needed tar files, and separately issue "zstash extract" afterwards. All is well. Then, I want the "ocean" files. If I apply this to the "--kept" tar files, I may extract some - but not all of the ocean files. If instead I issue a "zstash check --hpss=NERSC archive/ocean/hist/timeseriesMonthly" I will certainly receive all needed tarfiles, but may have redundantly transferred some tar files.

This is nit-picking, perhaps, but I seek to minimize transfer and extraction cycles.

What would be "ideal", from my perspective, would be the ability to supply a list of patterns (or else, one at a time) and obtain the names of the associated tarfiles, take their union, and then conduct a "zstash check --tars " for just those tar files.

Honestly, it never occurred to me until this moment that I might be able to issue:

zstash --check --hpss=NERSC <pattern_1> <pattern_2> ... <pattern_n>

and have zstash conduct the "union" internally. Then, I would be sure that the arriving tar-files contain ALL of the files satisfying the patterns. Is this possible?

[forsyth2]

zstash --check --hpss=NERSC <pattern_1> <pattern_2> ... <pattern_n>

In an effort to keep a ground truth, I set up #418 to test behavior of workflows described in this discussion. That test (specifically test_method2_check_keep_union) confirms this is indeed possible/effective.

Just keep in mind you'd also need to specify --keep so the tars stay in the directory.

[TonyB9000]

@forsyth2 Awesome! Thank you for taking the time to test this. The documentation should mention this as a feature (if it is there - supporting a space-delimited list of file patterns - I'm sure I missed it).

[forsyth2]

Yes, agreed. The zstash docs could use a number of updates to answer user questions. (To that end, I have a doc update PR up at #419)