add docs about --bag switch

Author: DocOtak

docs/running.md

@@ -58,6 +58,15 @@ It is safe to kill ({kbd}`control` + {kbd}`c`) and restart the QA process when t
 The python program, not the docker container, the container should clean itself up when python exits.
 :::
 
+#### Control how closely to follow BagIt manifest validation spec `--bag`
+The first release version of this software would only check what is in the manifest-md5.txt file, that was found to not be as robust as we wanted.
+Some breakouts were found to have files, but empty manifests, this software would treat this as an empty breakout and... crash.
+A stricter mode was implemented that can be controlled by the --bag switch value:
+
+* `strict`, any files in the `/data` directory and not in the manifest-md5.txt cause the manifest OK test to report failure.
+* `flex`, a reasonable set of file names are allowed to exist in `/data` and not in the manifest-md5.txt, see [](#r2r_ctd.breakout.FLEX_FILES_OK) for the list of filenames allowed.
+* `manifest` reverts to the original behavior where only paths in the manifest-md5.txt are checked and any extra files in `/data` are ignored.
+
 ## Breakout Structure
 When R2R receives data from a cruise it will be split up into separate collections called "breakouts".
 To be processed, the breakout is expected to be a directory with contents, not an archive such as a zip file.

src/r2r_ctd/breakout.py

@@ -20,6 +20,11 @@
 
 logger = getLogger(__name__)
 
+FLEX_FILES_OK = """.DS_Store
+thumbs.db"""
+"""Filenames that, when in --bag flex validation mode, will not cause a fail"""
+# kept as a string so it prints in the documentation nicely
+
 
 class BBox(NamedTuple):
     """namedtuple to represent a geo bounding box
@@ -178,9 +183,7 @@ def manifest_ok(self) -> bool:
 
         This is one of the checks that goes into the stoplight report.
         """
-        flex_files = {
-            ".DS_Store",
-        }
+        flex_files_ok = set(FLEX_FILES_OK.split("\n"))
         logger.info(f"Bag validation mode: {self.bag_strictness}")
         err_message = "Files are in payload directory and not in manifest, breakout is likely invalid or corrupted"
         for root, _, files in self.payload_path.walk():
@@ -191,7 +194,7 @@ def manifest_ok(self) -> bool:
                 return False
 
             if self.bag_strictness == "flex" and not all(
-                d.name in flex_files for d in diff
+                d.name in flex_files_ok for d in diff
             ):
                 logger.critical(err_message)
                 return False