feat(oiiotool): --eraseattrib:fromfile=1 (#4763)

Extended eraseattrib to be able to read regex patterns from a file,
one per line, with `--eraseattrib:fromfile=1 filename`.
Erases all metadata attribute fields that match.

Added a new test to `oiiotool-attribs` which has been verified
painstakenly to be working!
Also tried running the command directly from dist/bin and verified with
`oiiotool --info -v`.

More detail on the motivation:

It's common in a studio to have tools insert all sorts of metadata into files as they're rendered, etc. But a lot of that are things that SHOULD NOT leave the facility -- like security issues from internal pathnames or network info, or secrets like plot details revealed by sequence or character names. So generally you want to scrub the metadata from the final images before they are shipped to the client or to marketing or whatever.

oiiotool's --eraseattrib is handy for this, but it takes a single regex. But we were thinking that:

-  You may have several patterns to fully describe the metadata you want to be sure not to ship. This could be an unwieldy regex or a cluttered series of multiple --eraseattrib commands.
-  That list of patterns may change over time, and it would be error prone to have to change it in every script that calls oiiotool.
-  That list of patterns may need to be different per show.

We thought a solution to facilitate this might be a version of --eraseattrib that takes a filename that contains a list of regexes, one per line of the text file. So that master list of things to scrub could be maintained in a central place and not need to update every oiiotool-calling script in order to update it. Furthermore, you could have the command say something like

    $ oiiotool ready_to_ship.exr --eraseattrib:fromfile=1 /blah/$SHOW/nometadata.txt -o clean.exr

to make a single command that allows per-show customization.

---------

Signed-off-by: Lydia Zheng <[email protected]>

Author: lydia-zheng

src/doc/oiiotool.rst

@@ -2036,6 +2036,13 @@ current top image.
         Only included subimages will have the attribute changed. If subimages
         are not set, only the first subimage will be changed, or all subimages
         if the `-a` command line flag was used.
+      
+      `:fromfile=` *int*
+        When set to 1, the next argument will be interpreted as
+        the name of a file containing a list of patterns to erase, for example:
+        `--eraseattrib:fromfile=1 patterns.txt`,
+        The patterns will be case insensitive and one pattern per line of the file.
+        Default value is 0 (False).
 
     Examples::
 
@@ -2048,6 +2055,13 @@ current top image.
         # Remove all metadata
         oiiotool in.exr --eraseattrib:subimages=all ".*" -o no_metadata.exr
 
+        # Remove all attribute that match any regex in text file
+        oiiotool in.exr --eraseattrib:fromfile=1 no_gps_make.txt -o no_gps_make_metadata.exr
+
+        Example contents of file no_gps_make.txt:
+            Make
+            GPS:.*
+
 
 .. option:: --orientation <orient>
 
@@ -4770,7 +4784,3 @@ General commands that also work for deep images
     `NaN` or `Inf` values (hereafter referred to collectively as
     "nonfinite") are repaired.  The *strategy* may be either `black` or
     `error`.
-
-
-
-

src/oiiotool/oiiotool.cpp

@@ -1494,6 +1494,14 @@ set_oiio_attribute(Oiiotool& ot, cspan<const char*> argv)
         OIIO::attribute(p.name(), p.type(), p.data());
 }
 
+static std::vector<std::string>
+get_regex_list_from_file(string_view filename)
+{
+    // helper function that takes a text file and return all regex attribname entries as a list
+    std::string contents;
+    Filesystem::read_text_file(filename, contents);
+    return Strutil::splits(contents, "\n");
+}
 
 
 // Special OiiotoolOp whose purpose is to set attributes on the top image.
@@ -1503,8 +1511,16 @@ class OpAttribSetter final : public OiiotoolOp {
         : OiiotoolOp(ot, opname, argv, 1)
     {
         inplace(true);  // This action operates in-place
-        attribname = args(1);
-        value      = (nargs() > 2 ? args(2) : "");
+        auto options  = ot.extract_options(opname);
+        bool fromfile = options.get_int("fromfile", 0);
+        if (!fromfile) {
+            attribname = args(1);
+        } else {
+            // handle erase attribute using regex text file case
+            regex_file      = args(1);
+            attribname_list = get_regex_list_from_file(regex_file);
+        }
+        value = (nargs() > 2 ? args(2) : "");
     }
     bool setup() override
     {
@@ -1516,7 +1532,15 @@ class OpAttribSetter final : public OiiotoolOp {
         // Because this is an in-place operation, img[0] is the same as
         // img[1].
         if (value.empty()) {
-            img[0]->specmod().erase_attribute(attribname);
+            if (attribname_list.empty()) {
+                img[0]->specmod().erase_attribute(attribname);
+            } else {
+                for (string_view attr : attribname_list) {
+                    //std::cout << "list element:" << attr << std::endl;
+                    img[0]->specmod().erase_attribute(attr);
+                }
+            }
+
         } else {
             TypeDesc type(options()["type"].as_string());
             set_attribute_helper(img[0]->specmod(), attribname, value, type);
@@ -1527,6 +1551,8 @@ class OpAttribSetter final : public OiiotoolOp {
 private:
     string_view attribname;
     string_view value;
+    string_view regex_file;
+    std::vector<std::string> attribname_list;
 };
 
 
@@ -6663,7 +6689,7 @@ Oiiotool::getargs(int argc, char* argv[])
       .help("Sets string metadata attribute")
       .OTACTION(action_sattrib);
     ap.arg("--eraseattrib %s:REGEX")
-      .help("Erase attributes matching regex")
+      .help("Erase attributes matching regex (options: fromfile=1 filename-containing-patterns")
       .OTACTION(erase_attribute);
     ap.arg("--caption %s:TEXT")
       .help("Sets caption (ImageDescription metadata)")

testsuite/oiiotool-attribs/ref/out-jpeg9d.txt

@@ -1,3 +1,21 @@
+Reading nomakegps.jpg
+nomakegps.jpg        : 2048 x 1536, 3 channel, uint8 jpeg
+    SHA-1: A74C7DF2B01825DCB6881407AE77C11DC56AB741
+    channel list: R, G, B
+    Model: "T-Mobile G1"
+    Orientation: 1 (normal)
+    ResolutionUnit: "none"
+    XResolution: 72
+    YResolution: 72
+    Exif:ColorSpace: 1
+    Exif:ExifVersion: "0230"
+    Exif:FlashPixVersion: "0100"
+    Exif:PixelXDimension: 2048
+    Exif:PixelYDimension: 1536
+    Exif:WhiteBalance: 0 (auto)
+    Exif:YCbCrPositioning: 1
+    jpeg:subsampling: "4:2:0"
+    oiio:ColorSpace: "sRGB"
 Reading nomake.jpg
 nomake.jpg           : 2048 x 1536, 3 channel, uint8 jpeg
     SHA-1: A74C7DF2B01825DCB6881407AE77C11DC56AB741

testsuite/oiiotool-attribs/ref/out.txt

@@ -1,3 +1,21 @@
+Reading nomakegps.jpg
+nomakegps.jpg        : 2048 x 1536, 3 channel, uint8 jpeg
+    SHA-1: 2623446988E34395C6B0A4AA4FC75107C708BF18
+    channel list: R, G, B
+    Model: "T-Mobile G1"
+    Orientation: 1 (normal)
+    ResolutionUnit: "none"
+    XResolution: 72
+    YResolution: 72
+    Exif:ColorSpace: 1
+    Exif:ExifVersion: "0230"
+    Exif:FlashPixVersion: "0100"
+    Exif:PixelXDimension: 2048
+    Exif:PixelYDimension: 1536
+    Exif:WhiteBalance: 0 (auto)
+    Exif:YCbCrPositioning: 1
+    jpeg:subsampling: "4:2:0"
+    oiio:ColorSpace: "sRGB"
 Reading nomake.jpg
 nomake.jpg           : 2048 x 1536, 3 channel, uint8 jpeg
     SHA-1: 2623446988E34395C6B0A4AA4FC75107C708BF18

testsuite/oiiotool-attribs/run.py

@@ -7,6 +7,12 @@
 # Test oiiotool's ability to add and delete attributes
 
 
+# Test --eraseattrib:fromfile=1 to erase from a list of regex matching attributes
+
+command += oiiotool (OIIO_TESTSUITE_IMAGEDIR+"/tahoe-gps.jpg --eraseattrib:fromfile=1 src/regex_list.txt -o nomakegps.jpg")
+command += info_command ("nomakegps.jpg", safematch=True)
+
+
 # Test --eraseattrib ability to erase one specific attribute
 # The result should not have "Make"
 command += oiiotool (OIIO_TESTSUITE_IMAGEDIR+"/tahoe-gps.jpg --eraseattrib Make -o nomake.jpg")

testsuite/oiiotool-attribs/src/regex_list.txt

@@ -0,0 +1,2 @@
+Make
+GPS:.*