bin/fixity_check_replicated_moabs

[jmartin-sul]

TODO to get this out of draft:

• there's a deprecation warning about the current download approach and the need to switch to transfer manager. see here for an example of that switch: Switch to using Aws transfer manager. #2526
• extract the meat of the functionality into a service class
• write unit tests for said service class
  • Q: is it worth adding localstack and writing a more integration-y test with that? or perhaps easier and more worthwhile as a test: somehow calling this from our integration test suite, e.g. by executing it via SSH as part of the replication check in EventHelpers.confirm_archive_zip_replication_events?

Why was this change made? 🤔

This is the start of a script for pulling down and fixity checking content we've archived to cloud buckets. It's a shaping up from what was essentially ruby console notes to myself for this sort of check, into something that is runnable as a script by anyone. The machinery here could also form a starting point for #1076.

It has been useful for spot checking GCP replication on prod (as we did with its predecessor against stage and QA). See #2426 and #2518.

Usage

Usage: bin/fixity_check_replicated_moabs.rb [options]
        --druid_list DRUID_LIST      comma-separated (no spaces) list of bare druids (no prefixes)
        --druid_list_file DRUID_LIST_FILE
                                     file with a list of provided druids, e.g. from integration tests, manual tests, your own queries, etc
        --fixity_check_base_location FIXITY_CHECK_BASE_LOCATION
                                     target directory for downloading cloud archived Moabs, where they will be inflated and fixity checked.  ensure sufficient free space.
        --single_part_druid_sample_count SINGLE_PART_DRUID_SAMPLE_COUNT
                                     number of < 10 GB Moabs to query for and retrieve (default: 0)
        --multipart_druid_sample_count MULTIPART_DRUID_SAMPLE_COUNT
                                     number of > 10 GB Moabs to query for and retrieve (default: 0)
        --endpoints_to_audit ENDPOINTS_TO_AUDIT
                                     list of cloud endpoints to audit (comma-separated, no spaces, names from config)
        --[no-]force_part_md5_comparison
                                     Even if the zip parts are not downloaded on this run, compare the previously downloaded MD5 results to what is in the DB
        --[no-]dry_run               Simulate download and fixity check for druid list (defaults to false)
        --[no-]quiet                 Do not output progress information (defaults to false)
    -h, --help                       Displays help.

How was this change tested? 🤨

it was used to fixity check the test objects listed in #2518. when an earlier pass at the script failed to properly re-assemble multi-part zips it had downloaded, the missing moab manifest files were detected (they were indeed missing from that first zip extraction attempt; but were present in the multi-part zip, retrieved when the script was updated to re-assemble the zip successfully, and led to the moabs in question passing fixity check once the zip re-assembly was fixed). this was a convenient natural experiment with a corrupted zip extraction that was resolved by fixing the extraction approach.

[jmartin-sul]

note that there is more indirection to go through if you want to retrieve from AWS endpoints, see #1076 (comment):

not all content is directly retrievable: while GCP lets you just retrieve from the bucket using a simple synchronous request, AWS endpoint policy is such that the content is shuffled off to cold storage, a request for its retrieval must be made, and then something must wait for that request to be fulfilled.

these docs may be out of date in light of #2526, but these are the related tabs i had open from the bit of digging i did on this last year:

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html
• https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#restore_object-instance_method

[aaron-collier]

@jmartin-sul I'm still working on verifying the files, but at a minimum, https://github.com/sul-dlss/preservation_catalog/wiki/Extracting-segmented-zipfiles#the-hard-way indicates that this method of zip -s 0 does NOT work and to individually cat the files in order. I think the fact that unzipped parts are incomplete confirms this.

So I think this needs to be updated to try to best identify the parts and individually cat them.

[aaron-collier]

And, for what it's worth, the combined seems too small here as well:

rw-r--r-- 1 pres pres  28G Jan 28 21:02 zy152mz9183.v0002.combined.zip
-rw-r--r-- 1 pres pres  10G Jan 28 17:51 zy152mz9183.v0002.z01
-rw-r--r-- 1 pres pres  10G Jan 28 17:55 zy152mz9183.v0002.z02
-rw-r--r-- 1 pres pres  10G Jan 28 17:59 zy152mz9183.v0002.z03
-rw-r--r-- 1 pres pres  10G Jan 28 18:03 zy152mz9183.v0002.z04
-rw-r--r-- 1 pres pres  10G Jan 28 18:07 zy152mz9183.v0002.z05
-rw-r--r-- 1 pres pres  10G Jan 28 18:11 zy152mz9183.v0002.z06
-rw-r--r-- 1 pres pres 3.0G Jan 28 18:13 zy152mz9183.v0002.zip

[aaron-collier]

cat-ing the files seems more accurate, maybe:

-rw-r--r-- 1 pres pres  28G Jan 28 21:02 zy152mz9183.v0002.combined.zip
-rw-r--r-- 1 pres pres  63G Feb  2 13:29 zy152mz9183.v0002.fixed.zip