[#48] Add migration feature to migrate output products forward from MMTC 1.5.1 onward

MMTC output products changed significantly since 1.5.1.  Specifically, the Run History File and Time History File both had changes that would take some time and verification for our users to do manually.

Therefore, as part of MMTC 1.6.0 maintenance, we're added a simple migration capability to check the version of the built-in output products and migrat them.  Output plugins are not yet included in this functionality.

For MMTC commands that are sensitive to the structure of the built-in output products, the output product versions are now checked before proceeding.  If they are found not to be up to date, MMTC will halt and ask users to manually run the 'migrate' command.

When users run `mmtc migrate`, MMTC will copy all existing output files into a zip archive, and then perform the mgration.  After migration is complete, MMTC will exit, at which time the user (or automation) can run MMTC as normal.

Closes #48.

Author: rmseng

docs/MMTC_Users_Guide.adoc

@@ -421,13 +421,15 @@ Note that the MMTC RPM currently does not have special handling for configuratio
 
 If a user named “mmtc” does not already exist, the RPM installation will also create the mmtc user and group. All files installed by the RPM are world-readable; however, only the mmtc user and members of the mmtc group can modify configuration files and add or create files (such as new log files or additional telemetry source plugins) in the conf, log, and lib/plugins directories.
 
-After installing the RPM, do the following steps as needed:
+After installing the RPM, perform the following steps as needed:
 
 * Add users to the mmtc group.
 * To use a telemetry source other than AMPCS, add an appropriate telemetry source plugin to the lib/plugins folder.
 * Update the configuration files, in particular TimeCorrelationConfigProperties.xml, in the conf folder to suit the mission.
 * Place a seed SCLK kernel file in the output folder.
 
+After upgrading an MMTC installation, run `bin/mmtc migrate` in order to migrate output product structures forward to their new versions, where applicable.  A backup archive in ZIP format will be created, should the prior state of the files be needed.
+
 MMTC can then be invoked from the command line. Change the working directory to the MMTC installation directory (`/opt/local/mmtc` by default), and then run:
 
 [source, bash]
@@ -1444,6 +1446,10 @@ Defaults to 9 digits if key isn't included in config or if the value provided ca
 |STR
 |A comma-separated list of names that uniquely identify plugin-provided (custom) output product types.  If this parameter is populated, additional per-custom-product parameters are also required.  Please reference <<_configuring_custom_output_products>> for details.
 
+|product.archive.directory
+|OPTIONAL
+|STR
+|The path to a directory where zip archives of output products will be made before MMTC executes product migration or rollback commands.  If left blank, defaults to a new directory named 'output-archives', created within the same directory as the Run History File location.
 
 |===
 
@@ -1968,7 +1974,7 @@ Please note that a single plugin jar may include more than one _OutputProductDef
 
 == Output Product Rollback
 MMTC 1.5.0 and later provides a simple system for users to revert all of MMTC's output products back to an earlier state. Rollback makes use of the Run History File to determine which output products and
-records should and shouldn't be present after rolling back to the state MMTC's products were in after a prior run. Rollback is irreversible and should only be used when backup copies have been made of all relevant files.
+records should and shouldn't be present after rolling back to the state MMTC's products were in after a prior run. Rollback may be complicated to reverse; please use caution.  A backup archive in ZIP format will be created before output products are adjusted, which can be used to help restore the pre-rollback state if necessary.
 
 === Executing Rollback
 MMTC enters the rollback interface any time "rollback" is provided as the very first argument when invoking MMTC regardless of subsequent flags or arguments, i.e. `bin/mmtc rollback`. Users will immediately be provided with
@@ -1980,12 +1986,12 @@ output files that will be deleted (SCLK kernels, SCLKSCET files, and/or Uplink C
 After listing the effects of rollback, MMTC will ask users for Y/N confirmation to proceed. This marks the "point of no return" after which files will potentially be permanently deleted, so users are again reminded to make backups
 of *all* output products before initiating a rollback.
 
-After positive confirmation is received (only a "y" or "Y" is considered confirmation), MMTC will delete or truncate any output files necessary to return its products to the state they had after the specified prior run.
+After positive confirmation is received (only a "y" or "Y" is considered confirmation), MMTC will create a backup archive, and will then delete or truncate any output files necessary to return its products to the state they had after the specified prior run.
 MMTC will print out the names of all files modified (or potentially files that failed to be modified) and MMTC will exit with status code 0 if rollback was successful.
 
 Prior runs that were rolled back are not deleted from the Run History File but are flagged as previously involved in rollback and will be ignored by any future rollbacks.
 
-Insufficient system permissions to remove relevant files are the most likely source of rollback failures, so it is recommended that users verify that they have the requisite permissions to modify MMTC output products prior to initiating rollback.
+Insufficient system permissions to remove relevant files can be a source of rollback failures, so it is recommended that users verify that they have the requisite permissions to modify MMTC output products prior to initiating rollback.
 
 == Telemetry Caching
 

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/app/BuildInfo.java

@@ -28,6 +28,10 @@ public BuildInfo() {
         commit = buildProperties.getProperty(COMMIT_KEY, UNKNOWN_VALUE);
     }
 
+    public String getNumericalVersion() {
+        return version.replace("-SNAPSHOT", "");
+    }
+
     public String toString() {
         return String.format("Version: %s\nBuild date: %s\nCommit: %s", version, buildDate, commit);
     }

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/app/MmtcCli.java

@@ -1,5 +1,6 @@
 package edu.jhuapl.sd.sig.mmtc.app;
 
+import edu.jhuapl.sd.sig.mmtc.products.util.BuiltInOutputProductMigrationManager;
 import edu.jhuapl.sd.sig.mmtc.rollback.TimeCorrelationRollback;
 import edu.jhuapl.sd.sig.mmtc.sandbox.MmtcSandboxCreator;
 import edu.jhuapl.sd.sig.mmtc.tlm.persistence.cache.TelemetryCacheUserOperations;
@@ -24,8 +25,9 @@ public enum ApplicationCommand {
         CORRELATION,
         ROLLBACK,
         CREATE_SANDBOX,
+        MIGRATE,
         PRECACHE,
-        CACHE_STATS,
+        CACHE_STATS
     }
 
     private static class ApplicationInvocation {
@@ -46,7 +48,7 @@ private static ApplicationInvocation determineApplicationCommand(String... cliAr
 
         if (Arrays.asList("-h", "--help").contains(cliArgs[0])) {
             final String helpMessage =
-                    "usage: mmtc [correlation|rollback|create-sandbox|precache|cache-stats] [options] <additional arguments>\n" +
+                    "usage: mmtc [correlation|rollback|create-sandbox|migrate|precache|cache-stats] [options] <additional arguments>\n" +
                     " -h,--help      Print this message.\n" +
                     " -v,--version   Print the MMTC version.\n" +
                     "\n" +
@@ -56,6 +58,7 @@ private static ApplicationInvocation determineApplicationCommand(String... cliAr
                     "- rollback: roll back (undo) one or many correlations\n" +
                     "- create-sandbox: create a copy of this MMTC installation to run locally,\n" +
                     "without affecting this installation\n" +
+                    "- migrate: migrate MMTC's output products from a prior version of MMTC\n" +
                     "- precache: query the configured telemetry source to proactively retrieve\n" +
                     "and store time correlation telemetry into a local cache\n" +
                     "- cache-stats: log statistics about the locally-cached telemetry\n" +
@@ -71,6 +74,8 @@ private static ApplicationInvocation determineApplicationCommand(String... cliAr
             return new ApplicationInvocation(ApplicationCommand.CREATE_SANDBOX, removeFirstElement(cliArgs));
         } else if (cliArgs[0].equalsIgnoreCase("correlation")) {
             return new ApplicationInvocation(ApplicationCommand.CORRELATION, removeFirstElement(cliArgs));
+        } else if (cliArgs[0].equalsIgnoreCase("migrate")) {
+            return new ApplicationInvocation(ApplicationCommand.MIGRATE, removeFirstElement(cliArgs));
         } else if (cliArgs[0].equalsIgnoreCase("precache")) {
             return new ApplicationInvocation(ApplicationCommand.PRECACHE, removeFirstElement(cliArgs));
         } else if (cliArgs[0].equalsIgnoreCase("cache-stats")) {
@@ -126,6 +131,15 @@ public static void main(String[] args) {
                 }
                 break;
             }
+            case MIGRATE: {
+                try {
+                    new BuiltInOutputProductMigrationManager(appInvoc.args).migrate();
+                } catch (Exception e) {
+                    logger.fatal("Output product migration failed.", e);
+                    System.exit(1);
+                }
+                break;
+            }
             case PRECACHE: {
                 try {
                     TelemetryCacheUserOperations.precache(appInvoc.args);
@@ -145,7 +159,8 @@ public static void main(String[] args) {
                 break;
             }
             default: {
-                throw new RuntimeException("No such command: " + appInvoc.command);
+                logger.fatal("Unrecognized command: " + appInvoc.command);
+                System.exit(1);
             }
         }
     }

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/app/TimeCorrelationApp.java

@@ -15,6 +15,7 @@
 import edu.jhuapl.sd.sig.mmtc.products.definition.SclkKernelProductDefinition;
 import edu.jhuapl.sd.sig.mmtc.products.definition.util.ProductWriteResult;
 import edu.jhuapl.sd.sig.mmtc.products.model.*;
+import edu.jhuapl.sd.sig.mmtc.products.util.BuiltInOutputProductMigrationManager;
 import edu.jhuapl.sd.sig.mmtc.tlm.FrameSample;
 import edu.jhuapl.sd.sig.mmtc.tlm.TelemetrySource;
 import edu.jhuapl.sd.sig.mmtc.tlm.selection.SamplingTelemetrySelectionStrategy;
@@ -72,6 +73,8 @@ public TimeCorrelationApp(String... args) throws Exception {
     private void init() throws Exception {
         config.validate();
 
+        new BuiltInOutputProductMigrationManager(config).assertExistingProductsDoNotRequireMigration();
+
         logger.debug("Loading SPICE library");
         TimeConvert.loadSpiceLib();
         TimeConvert.loadSpiceKernels(config.getKernelsToLoad());
@@ -207,11 +210,14 @@ private void recordRunHistoryFilePreRunValues() throws MmtcException {
         ctx.runId.set(newRunId);
 
         // Run info
-        newRunHistoryFileRecord.setValue(RunHistoryFile.RUN_TIME,              ctx.appRunTime.toString());
-        newRunHistoryFileRecord.setValue(RunHistoryFile.RUN_ID,                String.format("%05d",newRunId));
-        newRunHistoryFileRecord.setValue(RunHistoryFile.ROLLEDBACK,            "false");
-        newRunHistoryFileRecord.setValue(RunHistoryFile.RUN_USER,              System.getProperty("user.name"));
-        newRunHistoryFileRecord.setValue(RunHistoryFile.CLI_ARGS,              String.join(" ", config.getCliArgs()));
+        final String mmtcVersion = new BuildInfo().getNumericalVersion();
+        newRunHistoryFileRecord.setValue(RunHistoryFile.RUN_TIME,                               ctx.appRunTime.toString());
+        newRunHistoryFileRecord.setValue(RunHistoryFile.RUN_ID,                                 String.format("%05d",newRunId));
+        newRunHistoryFileRecord.setValue(RunHistoryFile.MMTC_VERSION,                           mmtcVersion);
+        newRunHistoryFileRecord.setValue(RunHistoryFile.MMTC_BUILT_IN_OUTPUT_PRODUCT_VERSION,   mmtcVersion);
+        newRunHistoryFileRecord.setValue(RunHistoryFile.ROLLEDBACK,                             "false");
+        newRunHistoryFileRecord.setValue(RunHistoryFile.RUN_USER,                               System.getProperty("user.name"));
+        newRunHistoryFileRecord.setValue(RunHistoryFile.CLI_ARGS,                               String.join(" ", config.getCliArgs()));
 
         // Output products
         for (OutputProductDefinition<?> prodDef : config.getAllOutputProductDefs()) {

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/cfg/MigrationConfig.java

@@ -0,0 +1,27 @@
+package edu.jhuapl.sd.sig.mmtc.cfg;
+
+import edu.jhuapl.sd.sig.mmtc.app.MmtcException;
+import org.apache.commons.cli.*;
+
+public class MigrationConfig extends MmtcConfig {
+    public MigrationConfig(String... args) throws Exception {
+        super();
+
+        final Options opts = new Options();
+        opts.addOption("h", "help", false, "Print this message.");
+
+        final CommandLineParser parser = new DefaultParser();
+        final CommandLine cmdLine = parser.parse(opts, args);
+
+        if (cmdLine.hasOption("h") || cmdLine.hasOption("help")) {
+            final HelpFormatter help = new HelpFormatter();
+            final String helpFooter = "\nInvoke the MMTC migration feature on the console.  Takes no CLI arguments.";
+            help.printHelp("mmtc migrate", "", opts, helpFooter);
+            System.exit(0);
+        }
+
+        if (cmdLine.getArgList().size() != 0) {
+            throw new MmtcException("Error parsing command line arguments.");
+        }
+    }
+}

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/cfg/MmtcConfig.java

@@ -917,6 +917,17 @@ public int getOscTempFractionDigits() throws MmtcException {
         return numDigits;
     }
 
+    /**
+     * Gets the path at which to write zip archives containing backups of MMTC output products.
+     * @return the path as described above
+     */
+    public Path getProductArchiveLocation() {
+        if (containsKey("product.archive.directory")) {
+            return Paths.get(getString("product.archive.directory"));
+        } else {
+            return getRunHistoryFilePath().getParent().resolve("output-archives");
+        }
+    }
 
     public enum SclkScetFileLeapSecondSclkRate {
         ONE,

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/products/definition/OutputProductDefinition.java

@@ -5,16 +5,18 @@
 import edu.jhuapl.sd.sig.mmtc.cfg.RollbackConfig;
 import edu.jhuapl.sd.sig.mmtc.correlation.TimeCorrelationContext;
 import edu.jhuapl.sd.sig.mmtc.products.definition.util.ProductWriteResult;
+import edu.jhuapl.sd.sig.mmtc.products.definition.util.ResolvedProductDirPrefixSuffix;
 import edu.jhuapl.sd.sig.mmtc.products.definition.util.ResolvedProductLocation;
+import edu.jhuapl.sd.sig.mmtc.products.definition.util.ResolvedProductPath;
 import edu.jhuapl.sd.sig.mmtc.products.model.TextProductException;
 import edu.jhuapl.sd.sig.mmtc.rollback.TimeCorrelationRollback;
 import edu.jhuapl.sd.sig.mmtc.util.Settable;
 import edu.jhuapl.sd.sig.mmtc.util.TimeConvertException;
 
 import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Path;
-import java.util.Map;
-import java.util.Optional;
+import java.util.*;
 
 /**
  * Defines common fields and methods common across all OutputProductDefinition implementations.
@@ -106,4 +108,26 @@ public final boolean isBuiltIn() {
      */
     public abstract Map<String, String> getSandboxConfigUpdates(MmtcConfig originalConfig, Path newProductOutputDir);
 
+    /**
+     * Resolve all filepaths related to this product that currently exist on disk
+     *
+     * @param config the loaded MMTC configuration
+     * @return the resolved product filepaths
+     * @throws MmtcException if any problem is encountered while determining the resolved product paths
+     */
+    public Set<Path> resolveAllExistingPaths(MmtcConfig config) throws MmtcException, IOException {
+        T resolvedLocation = resolveLocation(config);
+        if (resolvedLocation instanceof ResolvedProductPath) {
+            Path singlePath = ((ResolvedProductPath) resolvedLocation).pathToProduct;
+            if (Files.exists(singlePath)) {
+                return new HashSet<>(Arrays.asList(singlePath));
+            } else {
+                return Collections.emptySet();
+            }
+        } else if (resolvedLocation instanceof ResolvedProductDirPrefixSuffix) {
+            return new HashSet<>(((ResolvedProductDirPrefixSuffix) resolvedLocation).findAllMatching());
+        } else {
+            throw new IllegalStateException("Unrecognized resolved location: " + resolvedLocation);
+        }
+    }
 }

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/products/definition/util/ResolvedProductDirPrefixSuffix.java

@@ -1,6 +1,10 @@
 package edu.jhuapl.sd.sig.mmtc.products.definition.util;
 
+import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Path;
+import java.util.List;
+import java.util.stream.Collectors;
 
 public class ResolvedProductDirPrefixSuffix implements ResolvedProductLocation {
     public final Path containingDirectory;
@@ -12,4 +16,11 @@ public ResolvedProductDirPrefixSuffix(Path containingDirectory, String filenameP
         this.filenamePrefix = filenamePrefix;
         this.filenameSuffix = filenameSuffix;
     }
+
+    public List<Path> findAllMatching() throws IOException {
+        return Files.list(containingDirectory)
+                .filter(p -> p.getFileName().toString().startsWith(filenamePrefix))
+                .filter(p -> p.getFileName().toString().endsWith(filenameSuffix))
+                .collect(Collectors.toList());
+    }
 }

mmtc-core/src/main/java/edu/jhuapl/sd/sig/mmtc/products/model/RunHistoryFile.java

@@ -22,6 +22,8 @@ public class RunHistoryFile extends AbstractTimeCorrelationTable {
 
     public static final String RUN_TIME = "Run Time";
     public static final String RUN_ID = "Run ID";
+    public static final String MMTC_VERSION = "MMTC Version";
+    public static final String MMTC_BUILT_IN_OUTPUT_PRODUCT_VERSION = "Built-In Output Product Version";
     public static final String ROLLEDBACK = "Rolled Back?";
     public static final String RUN_USER = "Run User";
     public static final String CLI_ARGS = "MMTC Invocation Args Used";
@@ -90,6 +92,8 @@ public RunHistoryFile(Path path, List<OutputProductDefinition<?>> allOutputProdD
             final List<String> headers = new ArrayList<>(Arrays.asList(
                     RUN_TIME,
                     RUN_ID,
+                    MMTC_VERSION,
+                    MMTC_BUILT_IN_OUTPUT_PRODUCT_VERSION,
                     ROLLEDBACK,
                     RUN_USER,
                     CLI_ARGS,