Merge pull request #3 from github/kpaulisse-document-indexing

Document indexing and ENCs, and add reindex action

Author: kpaulisse

.github/PULL_REQUEST_TEMPLATE.md

@@ -19,7 +19,7 @@ This pull request [introduces/changes/removes] [functionality/feature].
 ## Checklist
 
 - [ ] Make sure that all of the tests pass, and fix any that don't. Just run `bundle exec rake` in your checkout directory, or review the CI job triggered whenever you push to a pull request.
-- [ ] Make sure that there is 100% test coverage (the CI job will test for this). You can ignore untestable sections of code with `# :nocov` comments. If you need help getting to 100% coverage please ask; however, don't just submit code with no tests.
+- [ ] Make sure that there is 100% test coverage (the CI job will test for this). You can ignore untestable sections of code with `# :nocov:` comments. If you need help getting to 100% coverage please ask; however, don't just submit code with no tests.
 - [ ] If you have added any new gem dependencies, make sure those gems are licensed under the MIT or Apache 2.0 license. We cannot add any dependencies on gems licensed under GPL.
 - [ ] If you have added any new gem dependencies, make sure you've checked in a copy of the `.gem` file into the [vendor/cache](/vendor/cache) directory.
 

.version

@@ -1 +1 @@
-0.5.0
+0.5.1

Gemfile.lock

@@ -1,8 +1,8 @@
 PATH
   remote: .
   specs:
-    octofacts (0.5.0)
-    octofacts-updater (0.5.0)
+    octofacts (0.5.1)
+    octofacts-updater (0.5.1)
       diffy (>= 3.1.0)
       net-ssh (>= 2.9)
       octocatalog-diff (>= 1.4.1)

doc/octofacts-updater.md

@@ -99,6 +99,55 @@ Note that `facter` may need to run as root to gather all of the facts for the sy
 
 Also, if you are using Puppet 4 or later, but are relying on "legacy" facts that were used in Puppet 3, you may need to add `--show-legacy` to the `facter` command line.
 
+## Indexing facts
+
+`octofacts` supports an index of facts, allowing you to select fixtures dynamically based on parameters, rather than by specifying the name of a specific file in your tests. When properly set up, the use of an index in rspec-puppet tests will look like this example:
+
+```
+let(:facts) { Octofacts.from_index(app: "my_app", role: "my_role") }
+```
+
+To set up the index, you need to configure three settings:
+
+- `file`: The absolute or relative path to the index file.
+- `node_path`: The absolute or relative path to the directory where fixtures for nodes are stored.
+- `indexed_facts`: An array of facts that you want to index. For the example above, you would need to index the `app` and `role` facts.
+
+In your configuration, this might look like:
+
+```title=octofacts-updater.yaml
+index:
+  file: ../spec/fixtures/facts/octofacts-index.yaml
+  node_path: ../spec/fixtures/facts/octofacts
+  indexed_facts:
+    - app
+    - role
+```
+
+Once you have configured the index, you can use `octofacts-updater` to build the index from the node fixtures you already have.
+
+```
+touch spec/fixtures/facts/octofacts-index.yaml
+octofacts-updater --config octofacts-updater.yaml --action reindex
+```
+
+:information_source: If a file or directory path starts with `.` or `..`, the path is treated as relative to the configuration file itself. This allows you to specify locations within a Puppet code repository, without regard to where on the system `octofacts-updater` is actually installed. Of course, you can also use absolute paths (starting with `/`).
+
+## Using an External Node Classifier (ENC)
+
+If your Puppet setup uses an external node classifier (ENC), it may supply settings in its `parameters` hash that are treated as top level variables by Puppet code (like facts), and can be accessed in Puppet manifests. For example, if your ENC sets a top level parameter `app: application_name` then in your Puppet code, `$::app` will equal "application_name". Furthermore, `%{::app}` will be interpolated in a Hiera configuration file.
+
+To configure `octofacts-updater` to run your ENC when updating facts for a node, add the path to your ENC in the configuration file.
+
+```title=octofacts-updater.yaml
+enc:
+  path: /usr/local/sbin/enc.sh
+```
+
+Note that per conventions, all ENCs must take exactly one parameter (the hostname) and return output in YAML format. You do not need to indicate the hostname (via `%%NODE%%` or hard-coding) when configuring the ENC, because this is assumed.
+
+If the ENC returns `parameters` at the top level, these are merged in to the facts gathered for the node. In the case that the node and the ENC both contain the same key, the key from the ENC will be used.
+
 ## Anonymizing and rewriting facts
 
 To avoid committing sensitive information into source control, and to prevent rspec-puppet tests from inadvertently contacting actual systems, `octofacts-updater` supports anonymizing and rewriting facts. For example, you might remove or scramble SSH keys, delete or hard-code facts like system uptime that change upon each run, or change IP addresses.

examples/config/quickstart.yaml

@@ -31,10 +31,20 @@
 # without explicitly naming a node, list the facts here that you would like to have indexed.
 # You can also specify the path to the index file. Be sure to adjust it for your site.
 index:
-  file: /var/lib/jenkins/workspace/puppet/spec/fixtures/facts/octofacts-index.yaml
+  file: ../spec/fixtures/facts/octofacts-index.yaml
+  node_path: ../spec/fixtures/facts/octofacts
   indexed_facts:
+    - app
+    - role
     - virtual
 
+# If you are using an external node classifier (ENC) you can import its results into octofacts
+# by having the octofacts-updater run the ENC as the node's facts are obtained. Anything in the
+# ENC's "parameters" hash will be configured as a fact and override the facts from the data source.
+# Provide the path to your ENC script here.
+# enc:
+#   path: ./config/enc.sh
+
 # This section cleans up and anonymizes facts. We have added a number of facts to this list
 # that contain sensitive information or change frequently. You should add any facts from your
 # site as needed.

lib/octofacts_updater/cli.rb

@@ -16,8 +16,8 @@ def initialize(argv)
         end
 
         opts.on("-c", "--config <config_file>", String, "Path to configuration file") do |f|
-          raise "Invalid configuration file" unless File.file?(f)
-          @opts[:config] = f
+          @opts[:config] = File.expand_path(f)
+          raise "Invalid configuration file #{@opts[:config].inspect}" unless File.file?(@opts[:config])
         end
 
         opts.on("-H", "--hostname <hostname>", String, "FQDN of the host whose facts are to be gathered") do |h|
@@ -67,11 +67,12 @@ def initialize(argv)
     end
 
     def usage
-      puts "Usage: octofacts-updater --action <action> [--config-file /path/to/config.yaml] [other options]"
+      puts "Usage: octofacts-updater --action <action> [--config /path/to/config.yaml] [other options]"
       puts ""
       puts "Available actions:"
-      puts "  bulk:  Update fixtures and index in bulk"
-      puts "  facts: Obtain facts for one node (requires --hostname <hostname>)"
+      puts "  bulk:    Update fixtures and index in bulk"
+      puts "  facts:   Obtain facts for one node (requires --hostname <hostname>)"
+      puts "  reindex: Build a new index from the existing fact fixtures"
       puts ""
     end
 
@@ -106,6 +107,7 @@ def run
 
       return handle_action_bulk if opts[:action] == "bulk"
       return handle_action_facts if opts[:action] == "facts"
+      return handle_action_bulk if opts[:action] == "reindex"
 
       usage
       exit 255
@@ -126,19 +128,39 @@ def substitute_relative_paths!(object_in, basedir)
       end
     end
 
-    def handle_action_bulk
-      facts_to_index = @config.fetch("index", {})["indexed_facts"]
-      unless facts_to_index.is_a?(Array)
-        raise ArgumentError, "Must declare index:indexed_facts in configuration to use bulk update"
-      end
+    def nodes_for_bulk
+      if opts[:action] == "reindex"
+        @opts[:quick] = true
+
+        path = if opts[:path]
+          File.expand_path(opts[:path])
+        elsif @config.fetch("index", {})["node_path"]
+          File.expand_path(@config.fetch("index", {})["node_path"], File.dirname(opts[:config]))
+        else
+          raise ArgumentError, "Must set --path, or define index:node_path to a valid directory in configuration"
+        end
+
+        unless File.directory?(path)
+          raise Errno::ENOENT, "--path must be a directory (#{path.inspect} is not)"
+        end
 
-      nodes = if opts[:host_list]
+        Dir.glob("#{path}/*.yaml").map { |f| File.basename(f, ".yaml") }
+      elsif opts[:host_list]
         opts[:host_list]
       elsif opts[:hostname]
         [opts[:hostname]]
       else
         OctofactsUpdater::FactIndex.load_file(index_file).nodes(true)
       end
+    end
+
+    def handle_action_bulk
+      facts_to_index = @config.fetch("index", {})["indexed_facts"]
+      unless facts_to_index.is_a?(Array)
+        raise ArgumentError, "Must declare index:indexed_facts in configuration to use bulk update"
+      end
+
+      nodes = nodes_for_bulk
       if nodes.empty?
         raise ArgumentError, "Cannot run bulk update with no nodes to check"
       end