Merge branch 'kpaulisse-hiera-5' into kpaulisse-mega-merge

Author: kpaulisse

doc/configuration-hiera.md

@@ -1,6 +1,28 @@
 # Configuring octocatalog-diff to use Hiera
 
-If you are using Hiera with Puppet, then you must already have a [`hiera.yaml`](https://docs.puppet.com/puppet/latest/reference/config_file_hiera.html) file to configure it. These instructions will guide you through pointing octocatalog-diff at that configuration file.
+## Hiera 5
+
+Hiera 5 is included with Puppet 4.9 and higher.
+
+If there is a `hiera.yaml` file in the base directory of the environment that is in hiera 5 format, and you are running Puppet 4.9 or higher, then that file will be recognized by Puppet (and therefore, by octocatalog-diff). There is no special configuration for octocatalog-diff needed to make this work. Similarly, there is no command line option or setting to changed this behavior, because there is no corresponding option to change Puppet's behavior.
+
+If you are running Puppet 4.8 or lower, then the `hiera.yaml` file in the base directory of the environment will be ignored (unless you use `--hiera-config` to specify it as your global configuration file).
+
+If you have no global hiera configuration and you wish to rely on a `hiera.yaml` file in the base directory of your environment, make sure that you are *not* using any of the following command line options or [configuration settings](/doc/configuration.md):
+
+- `--hiera-path` or `settings[:hiera_path]`
+- `--hiera-path-strip` or `settings[:hiera_path_strip]`
+- `--hiera-config` or `settings[:hiera_config]`
+
+There is more information about Hiera 5 in Puppet's documentation:
+
+- [Enable the environment layer for existing Hiera data](https://puppet.com/docs/puppet/5.3/hiera_migrate_environments.html)
+
+## Hiera global configuration
+
+If you are using Hiera 5 with a global configuration, or you are using Hiera 3 or before, then you must already have a [`hiera.yaml`](https://docs.puppet.com/puppet/latest/reference/config_file_hiera.html) file to configure it. These instructions will guide you through pointing octocatalog-diff at that configuration file.
+
+octocatalog-diff will automatically determine the version of your Hiera configuration file and treat it accordingly. (Hiera 5 configuration files are identified as such by a `version: 5` line in the file itself.)
 
 Before you start, please understand how octocatalog-diff compiles a catalog:
 
@@ -10,9 +32,9 @@ Before you start, please understand how octocatalog-diff compiles a catalog:
 - It compiles the catalog, based on the temporary directory, for environment=production
 - It removes the temporary directory
 
-## Configuring the location of hiera.yaml
+### Configuring the location of global hiera.yaml
 
-The command line option `--hiera-config PATH` allows you to set the path to hiera.yaml.
+The command line option `--hiera-config PATH` allows you to set the path to the global hiera.yaml.
 
 You may specify this as either an absolute or a relative path.
 
@@ -24,12 +46,9 @@ You may specify this as either an absolute or a relative path.
     bin/octocatalog-diff --hiera-config hiera.yaml ...
     ```
 
-  The path is relative to a checkout of your Puppet repository. With the setting above, it will use the file named `hiera.yaml` that is at the top level
-  of your Puppet checkout.
+  The path is relative to a checkout of your Puppet repository. With the setting above, it will use the file named `hiera.yaml` that is at the top level of your Puppet checkout.
 
-  Perhaps your hiera.yaml file is in a subdirectory of your Puppet checkout. In that case, just use the relative directory path. Be sure not to add a leading `/` though,
-  because you don't want octocatalog-diff to treat it as an absolute path. In the following example, suppose you have a top level directory called `config` and your
-  `hiera.yaml` file is contained within it. You could then use:
+  Perhaps your hiera.yaml file is in a subdirectory of your Puppet checkout. In that case, just use the relative directory path. Be sure not to add a leading `/` though, because you don't want octocatalog-diff to treat it as an absolute path. In the following example, suppose you have a top level directory called `config` and your `hiera.yaml` file is contained within it. You could then use:
 
     ```
     bin/octocatalog-diff --hiera-config config/hiera.yaml ...
@@ -65,7 +84,7 @@ You may specify this as either an absolute or a relative path.
 
   We strongly recommend that you version-control your hiera.yaml file within your Puppet repository, and use the relative path option described above.
 
-## Configuring the directory in your repository in which hiera data files are found
+### Configuring the directory in your repository in which hiera data files are found
 
 The command line option `--hiera-path PATH` allows you to set the directory path, relative to the checkout of your Puppet repository, of your Hiera YAML/JSON data files.
 
@@ -88,7 +107,7 @@ If you are specifying the Hiera data path in the [configuration file](/doc/confi
 
 octocatalog-diff will fail if you specify a path that is not a directory.
 
-## Configuring the prefix path to strip
+### Configuring the prefix path to strip
 
 This is a different, and potentially more complex, alternative to `--hiera-path` / `settings[:hiera_path]` described in the prior section. Unless you have a very good reason, you should prefer to use the instructions above.
 

doc/optionsref.md

@@ -43,11 +43,18 @@ Usage: octocatalog-diff [command line options]
         --master-cache-branch BRANCH Branch to cache
         --safe-to-delete-cached-master-dir PATH
                                      OK to delete cached master directory at this path
-        --hiera-config PATH          Relative path to hiera YAML file
+        --hiera-config STRING        Full or relative path to global Hiera configuration file globally
+        --to-hiera-config STRING     Full or relative path to global Hiera configuration file for the to branch
+        --from-hiera-config STRING   Full or relative path to global Hiera configuration file for the from branch
         --no-hiera-config            Disable hiera config file installation
-        --hiera-path PATH            Path to hiera data directory, relative to top directory of repository
+        --hiera-path STRING          Path to hiera data directory, relative to top directory of repository globally
+        --to-hiera-path STRING       Path to hiera data directory, relative to top directory of repository for the to branch
+        --from-hiera-path STRING     Path to hiera data directory, relative to top directory of repository for the from branch
         --no-hiera-path              Do not use any default hiera path settings
-        --hiera-path-strip PATH      Path prefix to strip when munging hiera.yaml
+        --hiera-path-strip STRING    Path prefix to strip when munging hiera.yaml globally
+        --to-hiera-path-strip STRING Path prefix to strip when munging hiera.yaml for the to branch
+        --from-hiera-path-strip STRING
+                                     Path prefix to strip when munging hiera.yaml for the from branch
         --no-hiera-path-strip        Do not use any default hiera path strip settings
         --ignore-attr "attr1,attr2,..."
                                      Attributes to ignore
@@ -81,10 +88,10 @@ Usage: octocatalog-diff [command line options]
         --from-puppet-binary STRING  Full path to puppet binary for the from branch
         --facts-terminus STRING      Facts terminus: one of yaml, facter
         --puppetdb-ssl-ca FILENAME   CA certificate that signed the PuppetDB certificate
-        --puppetdb-ssl-client-password PASSWORD
-                                     Password for SSL client key to connect to PuppetDB
         --puppetdb-ssl-client-cert FILENAME
                                      SSL client certificate to connect to PuppetDB
+        --puppetdb-ssl-client-password PASSWORD
+                                     Password for SSL client key to connect to PuppetDB
         --puppetdb-ssl-client-key FILENAME
                                      SSL client key to connect to PuppetDB
         --puppetdb-ssl-client-password-file FILENAME
@@ -643,6 +650,43 @@ by permitting a data type specification as well. (<a href="../lib/octocatalog-di
     </td>
   </tr>
 
+  <tr>
+    <td valign=top>
+      <pre><code>--from-hiera-config STRING</code></pre>
+    </td>
+    <td valign=top>
+      Full or relative path to global Hiera configuration file for the from branch
+    </td>
+    <td valign=top>
+      Specify a relative path to the Hiera yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_config.rb">hiera_config.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--from-hiera-path STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path to hiera data directory, relative to top directory of repository for the from branch
+    </td>
+    <td valign=top>
+      Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the
+Puppet control repo template, the value of this should be 'hieradata', which is the default. (<a href="../lib/octocatalog-diff/cli/options/hiera_path.rb">hiera_path.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--from-hiera-path-strip STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path prefix to strip when munging hiera.yaml for the from branch
+    </td>
+    <td valign=top>
+      Specify the path to strip off the datadir to munge hiera.yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_path_strip.rb">hiera_path_strip.rb</a>)
+    </td>
+  </tr>
+
   <tr>
     <td valign=top>
       <pre><code>--from-puppet-binary STRING</code></pre>
@@ -773,10 +817,10 @@ This timeout is specified in seconds. (<a href="../lib/octocatalog-diff/cli/opti
 
   <tr>
     <td valign=top>
-      <pre><code>--hiera-config PATH</code></pre>
+      <pre><code>--hiera-config STRING</code></pre>
     </td>
     <td valign=top>
-      Relative path to hiera YAML file
+      Full or relative path to global Hiera configuration file globally
     </td>
     <td valign=top>
       Specify a relative path to the Hiera yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_config.rb">hiera_config.rb</a>)
@@ -785,10 +829,10 @@ This timeout is specified in seconds. (<a href="../lib/octocatalog-diff/cli/opti
 
   <tr>
     <td valign=top>
-      <pre><code>--hiera-path PATH</code></pre>
+      <pre><code>--hiera-path STRING</code></pre>
     </td>
     <td valign=top>
-      Path to hiera data directory, relative to top directory of repository
+      Path to hiera data directory, relative to top directory of repository globally
     </td>
     <td valign=top>
       Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the
@@ -798,10 +842,10 @@ Puppet control repo template, the value of this should be 'hieradata', which is
 
   <tr>
     <td valign=top>
-      <pre><code>--hiera-path-strip PATH</code></pre>
+      <pre><code>--hiera-path-strip STRING</code></pre>
     </td>
     <td valign=top>
-      Path prefix to strip when munging hiera.yaml
+      Path prefix to strip when munging hiera.yaml globally
     </td>
     <td valign=top>
       Specify the path to strip off the datadir to munge hiera.yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_path_strip.rb">hiera_path_strip.rb</a>)
@@ -1541,6 +1585,43 @@ by permitting a data type specification as well. (<a href="../lib/octocatalog-di
     </td>
   </tr>
 
+  <tr>
+    <td valign=top>
+      <pre><code>--to-hiera-config STRING</code></pre>
+    </td>
+    <td valign=top>
+      Full or relative path to global Hiera configuration file for the to branch
+    </td>
+    <td valign=top>
+      Specify a relative path to the Hiera yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_config.rb">hiera_config.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--to-hiera-path STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path to hiera data directory, relative to top directory of repository for the to branch
+    </td>
+    <td valign=top>
+      Specify the path to the Hiera data directory (relative to the top level Puppet checkout). For Puppet Enterprise and the
+Puppet control repo template, the value of this should be 'hieradata', which is the default. (<a href="../lib/octocatalog-diff/cli/options/hiera_path.rb">hiera_path.rb</a>)
+    </td>
+  </tr>
+
+  <tr>
+    <td valign=top>
+      <pre><code>--to-hiera-path-strip STRING</code></pre>
+    </td>
+    <td valign=top>
+      Path prefix to strip when munging hiera.yaml for the to branch
+    </td>
+    <td valign=top>
+      Specify the path to strip off the datadir to munge hiera.yaml file (<a href="../lib/octocatalog-diff/cli/options/hiera_path_strip.rb">hiera_path_strip.rb</a>)
+    </td>
+  </tr>
+
   <tr>
     <td valign=top>
       <pre><code>--to-puppet-binary STRING</code></pre>

lib/octocatalog-diff/catalog-util/builddir.rb

@@ -236,25 +236,11 @@ def install_hiera_config(logger, options)
 
         # Munge datadir in hiera config file
         obj = YAML.load_file(file_src)
-        ([obj[:backends]].flatten || %w(yaml json)).each do |key|
-          next unless obj.key?(key.to_sym)
-          if options[:hiera_path_strip].is_a?(String)
-            next if obj[key.to_sym][:datadir].nil?
-            rexp1 = Regexp.new('^' + options[:hiera_path_strip])
-            obj[key.to_sym][:datadir].sub!(rexp1, @tempdir)
-          elsif options[:hiera_path].is_a?(String)
-            obj[key.to_sym][:datadir] = File.join(@tempdir, 'environments', environment, options[:hiera_path])
-          end
-          rexp2 = Regexp.new('%{(::)?environment}')
-          obj[key.to_sym][:datadir].sub!(rexp2, environment)
-
-          # Make sure the dirctory exists. If not, log a warning. This is *probably* a setup error, but we don't
-          # want it to be fatal in case (for example) someone is doing an octocatalog-diff to verify moving this
-          # directory around or even setting up Hiera for the very first time.
-          unless File.directory?(obj[key.to_sym][:datadir])
-            message = "WARNING: Hiera datadir for #{key} doesn't seem to exist at #{obj[key.to_sym][:datadir]}"
-            logger.warn message
-          end
+        version = obj['version'] || obj[:version] || 3
+        if version.to_i == 5
+          update_hiera_config_v5(logger, options, obj)
+        else
+          update_hiera_config_v3(logger, options, obj)
         end
 
         # Write properly formatted hiera config file into temporary directory
@@ -291,6 +277,77 @@ def install_ssl(logger, options)
 
       private
 
+      # Jump-off for hiera v3 (or earlier)
+      # @param logger [Logger] Logger object
+      # @param options [Hash] Options hash
+      # @param obj [Hash] Parsed hiera.yaml file
+      def update_hiera_config_v3(logger, options, obj)
+        ([obj[:backends]].flatten || %w(yaml json)).each do |key|
+          next unless obj.key?(key.to_sym)
+          obj[key.to_sym][:datadir] = hiera_munge(options, obj[key.to_sym][:datadir])
+
+          # Make sure the directory exists. If not, log a warning. This is *probably* a setup error, but we don't
+          # want it to be fatal in case (for example) someone is doing an octocatalog-diff to verify moving this
+          # directory around or even setting up Hiera for the very first time.
+          unless File.directory?(obj[key.to_sym][:datadir])
+            message = "WARNING: Hiera datadir for #{key} doesn't seem to exist at #{obj[key.to_sym][:datadir]}"
+            logger.warn message
+          end
+        end
+      end
+
+      # Jump-off for hiera v5
+      # @param logger [Logger] Logger object
+      # @param options [Hash] Options hash
+      # @param obj [Hash] Parsed hiera.yaml file
+      def update_hiera_config_v5(_logger, options, obj)
+        defaults_key = obj.key?(:defaults) ? :defaults : 'defaults'
+        hierarchy_key = obj.key?(:hierarchy) ? :hierarchy : 'hierarchy'
+
+        # Fix defaults:datadir
+        if obj[defaults_key].is_a?(Hash)
+          [:datadir, 'datadir'].each do |key|
+            next unless obj[defaults_key].key?(key)
+            obj[defaults_key][key] = hiera_munge(options, obj[defaults_key][key])
+          end
+        end
+
+        # Fix hierarchy:datadir
+        if obj[hierarchy_key].is_a?(Array)
+          obj[hierarchy_key].each do |level|
+            [:datadir, 'datadir'].each do |key|
+              next unless level.key?(key)
+              if options[:hiera_path_strip].is_a?(String)
+                level[key] = hiera_munge(options, level[key])
+              elsif options[:hiera_path].is_a?(String)
+                message = [
+                  "Hierarchy item #{level.inspect} has a datadir.",
+                  '--hiera-path is not supported in this situation.',
+                  'Please use --hiera-path-strip.'
+                ].join(' ')
+                raise ArgumentError, message
+              end
+            end
+          end
+        end
+      end
+
+      # Hiera munge - shared method to apply :hiera_path_strip and :hiera_path
+      def hiera_munge(options, current_value)
+        return if current_value.nil?
+
+        if options[:hiera_path_strip].is_a?(String)
+          rexp1 = Regexp.new('^' + options[:hiera_path_strip])
+          current_value.sub!(rexp1, @tempdir)
+        elsif options[:hiera_path].is_a?(String)
+          current_value = File.join(@tempdir, 'environments', environment, options[:hiera_path])
+        end
+        rexp2 = Regexp.new('%{(::)?environment}')
+        current_value.sub!(rexp2, environment)
+
+        current_value
+      end
+
       # Install SSL certificate authority certificate
       # @param logger [Logger] Logger object
       # @param options [Hash] Options hash

lib/octocatalog-diff/cli.rb

@@ -7,6 +7,7 @@
 require_relative 'cli/printer'
 require_relative 'errors'
 require_relative 'util/catalogs'
+require_relative 'util/util'
 require_relative 'version'
 
 require 'logger'
@@ -53,7 +54,7 @@ class Cli
     # @return [Integer] Exit code: 0=no diffs, 1=something went wrong, 2=worked but there are diffs
     def self.cli(argv = ARGV, logger = Logger.new(STDERR), opts = {})
       # Save a copy of argv to print out later in debugging
-      argv_save = argv.dup
+      argv_save = OctocatalogDiff::Util::Util.deep_dup(argv)
 
       # Are there additional ARGV to munge, e.g. that have been supplied in the options from a
       # configuration file?
@@ -70,8 +71,14 @@ def self.cli(argv = ARGV, logger = Logger.new(STDERR), opts = {})
       # Note: do NOT use 'options[k] ||= v' here because if the value of options[k] is boolean(false)
       # it will then be overridden. Whereas the intent is to define values only for those keys that don't exist.
       opts.each { |k, v| options[k] = v unless options.key?(k) }
-      veto_options = %w(enc header hiera_config include_tags)
+      veto_options = %w(enc header include_tags)
       veto_options.each { |x| options.delete(x.to_sym) if options["no_#{x}".to_sym] }
+      if options[:no_hiera_config]
+        vetoes = %w[hiera_config to_hiera_config from_hiera_config]
+        vetoes.each do |key|
+          options.delete(key.to_sym)
+        end
+      end
       options[:ignore].concat opts.fetch(:additional_ignores, [])
 
       # Incorporate default options where needed.
@@ -125,7 +132,7 @@ def self.cli(argv = ARGV, logger = Logger.new(STDERR), opts = {})
     # @param argv [Array] Command line arguments (MUST be specified)
     # @return [Hash] Options
     def self.parse_opts(argv)
-      options = { ignore: DEFAULT_IGNORES.dup }
+      options = { ignore: OctocatalogDiff::Util::Util.deep_dup(DEFAULT_IGNORES) }
       Options.parse_options(argv, options)
     end