How do I handle partially completed archives?

[forsyth2]

Question criteria

• I added a descriptive title here.
• I searched the zstash GitHub Discussions to find a similar question and didn't find it.
• I searched the zstash documentation.

What is the deadline?

N/A

Describe your question

From @czender:

How do I restart zstash after it apparently timed-out after archiving a good portion of my simulation? It emitted

ERROR: Exception: Failed to get an auth_code from Globus Auth.

after I neglected to paste a URL for too long. If I repeat the original command ...

zstash create --hpss=globus://nersc/home/z/zender/globus/${caseid} . &> zcreate.txt

will it automatically pick-up from where it left off? Or should I issue a different command, e.g.,

zstash update --hpss=globus://nersc/home/z/zender/globus/${caseid} . &> zcreate.txt

The documentation could be clearer about the recommended use of create vs. update for partially completed archives.

Are there are any possible answers you came across?

No response

What machine were you running on?

Unclear

Environment

Unclear, presumably zstash v1.4.4

Minimal Complete Verifiable Example (MCVE)

No response

Relevant log output

No response

Anything else we need to know?

No response

[forsyth2]

My first thought was to reference #339, where Chris Golaz has outlined a good workflow. (Note that some steps will be rendered obsolete by the upcoming release of zstash v1.5.0, in E3SM Unified v1.12.0). But this isn't quite the same situation. Step 6 of #339 has the workaround of "Using Globus web interface, manually transfer zstash files that were not transferred due to token expiration." but here we haven't even gotten to that point. Rather, the Globus URL prompt timed out.

My intuition is to use zstash update rather than zstash create if at least something has been transferred. zstash should be able to pick up what hasn't been transferred.

@TonyB9000 Have you encountered this situation before, and if so, how have you handled it? Either the specific situation of the auth code prompt timing out or the general situation of zstash erroring out during create?

[TonyB9000]

@forsyth2 The only time I have used "zstash create" was when testing the ability to keep/not-keep cached tar-files. So I have never encountered this issue (and have never used "update" to my knowledge).

Curiously, this does intersect with my earlier comments regarding atomicity. When "fetching" remote archives, the ability to issue a separate command for each individual tar-file (and gain confirmation for each) largely obviates worries about incomplete transfers - easy to re-request those that failed. But this would not make sense for archive creation unless the "protocol" was to "create the remote index.db", and then issue a separate "zstash update" command for each tar-file (which would need to be crafted in advance, or the data "chunked" such that each update were well-defined at some granularity).

[czender]

I resubmitted the original command (zstash create ...). It is working, but I knew it would work. The real questiona are: Did it start over from scratch or would it have been faster to use zstash. update? Unknown. Documentation of this would be helpful.

[forsyth2]

@czender Can you provide more context on "timed-out after archiving a good portion of my simulation?" That auth code prompt happens pretty early on, i.e., before any files transfer. If it happened mid-transfer, then that suggests the tokens timed out and it re-prompted (i.e., my point above re: step 6 of #339). In any case, the database is the last thing to transfer, so if create didn't complete, then I don't think there's anything for update to work off of.

Some thoughts from Claude:

---

Why create was the safer choice:

By rerunning create, the user ensured:

• Clean slate with a fresh database
• All files properly archived
• No reliance on a potentially corrupted partial state

The real answer to "which would be faster?":

It depends on when the failure occurred:

• Failed during initial Globus auth (user's case): Minimal work was lost, so create and update would take similar time. create is safer.

• Failed after archiving many files successfully:

  • If the database was uploaded to HPSS before failure → update would be much faster
  • If the database was NOT uploaded to HPSS → update would fail or use stale local database → create is necessary