revisit provenance log functionality #2584
Description
storytime questions: what events would we want to track? where would we log them, and how would we preserve that log?
filing this because we recently removed our initial/only/unused attempt at this. per the description i left on the shared_configs PR that removed the provlog setting from the prod configs:
The functionality controlled by this setting was removed in #2524
and it was never actually used in prod, or for any significant length of time in stage or QA, per the setting comment about the logging being too voluminous. as the naming of the setting obliquely implies, it was meant to be a provenance logging mechanism. it predated (iirc) the dor-services-app event log. i would posit that we should lean more into sending events to the DSA event log, and preserving that, as a provenance solution, at this point.
Advantages of the event log for this purpose:
- pres cat already writes replication events to it, as well as replication audit related status codes, and some CV/M2C/C2M failure codes. we could consider writing other audit events that would've gone to the (now also defunct)
preservationAuditWF(see Decouple preservation from preservationAuditWF #2533). e.g. checksum validation failure, going back tookstatus after checksum validation passes on a remediated Moab, etc. - other applications also write to the event log, so it would keep from siloing the preservation related history/provenance, and put it in context with other SDR application activity.
- lets us be more intentional and targeted, and less noisy, in what we capture, than a log of every attempted database change to every object (which is what the old code did, afaict from a quick glance back at it).
Other considerations: typically, we would want robust long term preservation for a provenance log. at present, the DSA event log is regularly backed up in production (since it's in the DSA PostgreSQL DB, and regular prod PG backups are standard procedure for us). But backup != archiving/preservation.