Stop producing *.xz and/or *.gz files?

[genehack]

(Per 1:1 conversation with @joverlee521…)

While reviewing S3 bucket contents as part of the cleanup effort, I noticed that we're generating multiple copies of the same file using different compression schemes (e.g., aligned.fasta.xz and aligned.fasta.zst, flagged_metadata.txt.gz and flagged_metadata.txt.zst, etc).

Is the ZST format mature enough that we can consolidate down to just generating those files? (Or, alternatively, if we think we're never going to be rid of gz and/or xz, can we just generate those? Multiple versions of things that differ only by the compression scheme feels like a waste of resources.)

[victorlin]

We should consider impacts to existing external usage before pulling the plug on xz/gz.

Unfortunately, we don't have any metrics in place for data.nextstrain.org, but I believe we can get a sense of at least the quantity of usage through an AWS usage report (I don't have access to billing info to confirm).

There are some external tools that that may be regularly pulling those files, such as the SARS-CoV-2 LAPIS instance. Without metrics, it'll be hard to track all of these down, but collectively the team should be aware of most usages of this kind and can contact maintainers to ensure they've switched to using our zst files.

We should also delete/move them completely rather than letting them go stale, so it'll be obvious to users that we've stopped supporting the files. We could have "brownout" periods to help identify any regular external usage.

[corneliusroemer]

I just noticed how much cpu/time we spend on compression so yes, it'd be good to pull the plug on those - it's unnecessary compute/storage.

Removing vs going stale both have advantages and disadvantages. Either is fine with me.

I wouldn't worry too much about external users, they can just switch from .xz/.gz to .zst.

[AngieHinrichs]

Removing vs going stale both have advantages and disadvantages

I'd like to put in a word for removing, because then downstream users at least know that support is discontinued. If the files just go stale, then it may take a long time for downstream users to realize that despite the lack of errors, they are not getting what they think they are getting.