Should we try to archive provenance data of any E3SM job on certain machines?

[ndkeen]

(Discussion could apply to any machine)
At NERSC, on pm-cpu/pm-gpu, we currently save timing/provenance data for any job (*) that is using the following allocations:

<SAVE_TIMING_DIR_PROJECTS>e3sm,m3411,m3412</SAVE_TIMING_DIR_PROJECTS>

But that's far from the whole picture of users running various versions of E3SM. We could continue to add projects to this setting to capture more, but what about just allowing for "any" case to save data?

We could maybe have option to save data from all NERSC projects except certain ones?
We could have concept of "trusted" projects that save all data as normal, and then "others" where we save a minimal set?

(*) just to be pedantic, currently "tests" don't save data and any case free to use xmlchange on a variable to turn it off.

[sarats]

Prior discussion: #5753