Merge pull request #161 from github/kpaulisse-puppetdb-ssl-basicauth

Add puppetdb basic auth, update change log and docs

Author: kpaulisse

.version

@@ -1 +1 @@
-1.5.0
+1.5.1

doc/CHANGELOG.md

@@ -7,6 +7,15 @@
 <th>Description / Changes</th>
 </tr>
 </thead><tbody>
+
+<tr valign=top>
+<td>1.5.1</td>
+<td>2017-11-16</td>
+<td>
+<li><a href="https://github.com/github/octocatalog-diff/pull/159">#159</a>: (Enhancement) Add support for puppetdb behind basic auth</li>
+</td>
+</tr>
+
 <tr valign=top>
 <td>1.5.0</td>
 <td>2017-10-18</td>

doc/configuration-puppetdb.md

@@ -8,13 +8,19 @@ octocatalog-diff can interact with PuppetDB in the following ways:
 
 For this to work, you will need to configure or provide information about your PuppetDB server to octocatalog-diff. You can provide this information via a [configuration file](/doc/configuration.md), via environment variables, or via command line parameters.
 
-## Required information
+# Required information
 
 - **Version of PuppetDB**: octocatalog-diff supports PuppetDB's query API v4, which requires that you be running PuppetDB 2.3 or higher.
 
 - **URL to PuppetDB**: This is the URL with the host name and port number to reach your PuppetDB instance. If you have already set up your Puppet master to communicate with PuppetDB, you can see the URL by reviewing `/etc/puppetlabs/puppet/puppetdb.conf` (on Puppet Server) or `/etc/puppet/puppetdb.conf` (on Puppet Master 3.x). The URL (or URLs) to your PuppetDB installation are visible in the `server_urls` configuration setting.
 
-- **SSL Authentication Information**: Whether your PuppetDB instance requires clients to authenticate via SSL certificates. Unless you have made a special effort to configure your PuppetDB instance not to require client certificates, it is likely that client certificate authentication is required.
+  To use basic authentication, place the username and password in the URL, e.g.:
+
+  ```
+  https://username:[email protected]:8081
+  ```
+
+- **SSL Authentication Information**: Whether your PuppetDB instance requires clients to authenticate via SSL certificates. Unless you have made a special effort to configure your PuppetDB instance not to require client certificates, it is likely that client certificate authentication is required. Please see the separate section below concerning SSL certificates.
 
 NOTE: In certain situations, you may need to define or alter the `certificate-whitelist` setting in your PuppetDB configuration to whitelist the certificate used by octocatalog-diff. Please see [Configuring PuppetDB](https://docs.puppet.com/puppetdb/latest/configure.html#certificate-whitelist) in the Puppet documentation for additional information.
 
@@ -25,9 +31,9 @@ The following settings can be used in a [configuration file](/doc/configuration.
 | Setting | Description |
 | --- | --- |
 | `settings[:puppetdb_url]` | PuppetDB URL settings. If this is a string, it will set a single PuppetDB URL. If it is an array, it will set multiple URLs, which will be tried in a random order until one responds. |
-| `settings[:puppetdb_ssl_ca]` | Path to the certificate of the CA that signed PuppetDB's certificate. This file is typically found in `/etc/puppetlabs/puppetdb/ssl/ca.pem` on your PuppetDB server. This file should contain only the public certificate, so it is safe to distribute to developer workstations or CI environments. |
-| `settings[:puppetdb_ssl_client_cert]` | TEXT of the certificate of the client SSL keypair. You should generate a keypair specifically for this client (or if you are running this on a machine managed by Puppet, you may be able to use the keypair for the client machine). You should **NOT** copy the certificate from your PuppetDB server itself. Note: This variable needs to be set to the TEXT of the certificate, and not the file path. This means you will likely want to use `File.read(...)` if you are configuring this to be read from a file. |
-| `settings[:puppetdb_ssl_client_key]` | TEXT of the private key of the client SSL keypair. You should generate a keypair specifically for this client (or if you are running this on a machine managed by Puppet, you may be able to use the keypair for the client machine). You should **NOT** copy the private key from your PuppetDB server itself.  Note: This variable needs to be set to the TEXT of the key, and not the file path. This means you will likely want to use `File.read(...)` if you are configuring this to be read from a file. |
+| `settings[:puppetdb_ssl_ca]` | Path to the certificate of the CA that signed PuppetDB's certificate. This file should contain only the public certificate, so it is safe to distribute to developer workstations or CI environments. |
+| `settings[:puppetdb_ssl_client_cert]` | TEXT of the certificate of the client SSL keypair used to authenticate to PuppetDB. Note: This variable is not set to a file path, which means you will likely want to use `File.read(...)` if you are configuring this to be read from a file. |
+| `settings[:puppetdb_ssl_client_key]` | TEXT of the private key of the client SSL keypair used to authenticate to PuppetDB. Note: This variable is not set to a file path, which means you will likely want to use means you will likely want to use `File.read(...)` if you are configuring this to be read from a file. |
 | `settings[:puppetdb_ssl_client_pem]` | Concatenation of the text of `puppetdb_ssl_client_key` and `puppetdb_ssl_client_cert` as previously described. This is a good alternative if your certificate chain is complex and it's easier just to put everything in a single place. Note: this option is second in precedence; if `settings[:puppetdb_ssl_client_cert]` and `settings[:puppetdb_ssl_client_key]` are both set, this will be ignored. |
 | `settings[:puppetdb_ssl_client_password]` | Plain text string containing the password to unlock the private key. For keys generated by the Puppet Master CA, this is not required and should be left undefined. |
 
@@ -38,9 +44,9 @@ The following arguments can be used on the command line.
 | Setting | Description |
 | --- | --- |
 | --puppetdb-url https://puppetdb.example.net:8081 | PuppetDB URL. The argument should match the `server_urls` configuration setting as described previously. Please note that only one URL is supported via the command line method, so if you have multiple `server_urls` URLs specified, you can only choose one. To use multiple URLs for failover purposes, please configure via configuration files. |
-| --puppetdb-ssl-ca FILENAME | Path to the certificate of the CA that signed PuppetDB's certificate. This file is typically found in `/etc/puppetlabs/puppetdb/ssl/ca.pem` on your PuppetDB server. This file should contain only the public certificate, so it is safe to distribute to developer workstations or CI environments. |
-| --puppetdb-ssl-client-cert FILENAME | Path to the certificate of the client SSL keypair. You should generate a keypair specifically for this client (or if you are running this on a machine managed by Puppet, you may be able to use the keypair for the client machine). You should **NOT** copy the certificate from your PuppetDB server itself. |
-| --puppetdb-ssl-client-key FILENAME | Path to the private key of the client SSL keypair. You should generate a keypair specifically for this client (or if you are running this on a machine managed by Puppet, you may be able to use the keypair for the client machine). You should **NOT** copy the private key from your PuppetDB server itself. |
+| --puppetdb-ssl-ca FILENAME | Path to the certificate of the CA that signed PuppetDB's certificate. This file should contain only the public certificate, so it is safe to distribute to developer workstations or CI environments. |
+| --puppetdb-ssl-client-cert FILENAME | Path to the certificate of the client SSL keypair. |
+| --puppetdb-ssl-client-key FILENAME | Path to the private key of the client SSL keypair. |
 | --puppetdb-ssl-client-password PASSWORD_STRING | Plain text string containing the password to unlock the private key. For keys generated by the Puppet Master CA, this is not required. |
 
 ## Supplying necessary information via the environment
@@ -50,3 +56,11 @@ The following arguments can be used on the command line.
 Set the environment variable `PUPPETDB_URL` to match the `server_urls` configuration setting as described previously. Please note that only one URL is supported via the environment variable method, so if you have multiple `server_urls` URLs specified, you can only choose one. To use multiple URLs for failover purposes, please configure via configuration files.
 
 Environment variable support is not currently available for SSL client authentication settings.
+
+# Notes about SSL certificates
+
+SSL support is enabled via any of the `--puppetdb-ssl-...` command line options or `puppetdb_ssl_...` configuration settings as described above. Please note the following concerning these SSL certificates.
+
+- The CA certificate should be the public certificate of the CA that signed your PuppetDB server's certificate. This file can be found in `/etc/puppetlabs/puppetdb/ssl/ca.pem` on a PuppetDB server. Since this is a public certificate, it is safe (and recommended) to distribute this file to any clients that may connect to this PuppetDB instance.
+
+- The client keypair (key, certificate, and optionally password) should be generated individually for each client. You should NOT copy SSL keypairs from your PuppetDB server (or anywhere else) to your clients. If you are using `octocatalog-diff` on a system that is managed by Puppet, you may wish to use the same SSL credentials that the system uses to authenticate to Puppet. With recent versions of the Puppet agent, those certificates are found in `/etc/puppetlabs/puppet/ssl`.

rake/common.rb

@@ -1,4 +1,10 @@
 # Constants
 BASEDIR = File.expand_path('..', File.dirname(__FILE__)).freeze
 PUPPET_BINARY = File.join(BASEDIR, 'bin', 'puppet').freeze
-TEST_COMMAND = "parallel_rspec --suffix '_spec.rb$'".freeze
+TEST_COMMAND = begin
+  if ENV['TRAVIS']
+    "rspec --pattern '*_spec.rb'"
+  else
+    "parallel_rspec --suffix '_spec.rb$'"
+  end
+end.freeze

rake/spec.rb

@@ -46,7 +46,9 @@ def write_config(filename)
           end
         end
         write_config(logfile)
-        abort unless system("bundle exec #{TEST_COMMAND} --runtime-log .parallel_runtime_rspec.log #{paths} 2>/dev/null")
+        runtime_log = TEST_COMMAND =~ /parallel/ ? '--runtime-log .parallel_runtime_rspec.log' : ''
+        cmd = "bundle exec #{TEST_COMMAND} #{runtime_log} #{paths} 2>/dev/null"
+        abort unless system(cmd)
         f1 = File.open(File.expand_path('../.parallel_runtime_integration.log', File.dirname(__FILE__)), 'w')
         f2 = File.open(File.expand_path('../.parallel_runtime_tests.log', File.dirname(__FILE__)), 'w')
         File.read(logfile).split(/\n/).each do |line|
@@ -68,7 +70,8 @@ def write_config(filename)
       abort('Puppet binary missing. Please run script/bootstrap!') unless File.file?(PUPPET_BINARY)
       begin
         write_config('.parallel_runtime_integration.log')
-        cmd = "bundle exec #{TEST_COMMAND} --runtime-log .parallel_runtime_integration.log spec/octocatalog-diff/integration"
+        runtime_log = TEST_COMMAND =~ /parallel/ ? '--runtime-log .parallel_runtime_integration.log' : ''
+        cmd = "bundle exec #{TEST_COMMAND} #{runtime_log} spec/octocatalog-diff/integration 2>/dev/null"
         abort unless system(cmd)
       ensure
         FileUtils.rm PARALLEL_CONFIG if File.file?(PARALLEL_CONFIG)
@@ -79,7 +82,8 @@ def write_config(filename)
       abort('Puppet binary missing. Please run script/bootstrap!') unless File.file?(PUPPET_BINARY)
       begin
         write_config('.parallel_runtime_tests.log')
-        cmd = "bundle exec #{TEST_COMMAND} --runtime-log .parallel_runtime_tests.log spec/octocatalog-diff/tests"
+        runtime_log = TEST_COMMAND =~ /parallel/ ? '--runtime-log .parallel_runtime_tests.log' : ''
+        cmd = "bundle exec #{TEST_COMMAND} #{runtime_log} spec/octocatalog-diff/tests 2>/dev/null"
         abort unless system(cmd)
       ensure
         FileUtils.rm PARALLEL_CONFIG if File.file?(PARALLEL_CONFIG)

spec/octocatalog-diff/integration/examples_spec.rb

@@ -40,7 +40,11 @@
 
   context 'executing' do
     before(:all) do
-      @stdout, @stderr, @exitcode = Open3.capture3(@script)
+      # Add retries just in case something goes wrong in CI
+      3.times do
+        @stdout, @stderr, @exitcode = Open3.capture3(@script)
+        break if @exitcode.exitstatus.zero?
+      end
     end
 
     it 'should run without error' do
@@ -74,7 +78,11 @@
 
   context 'executing' do
     before(:all) do
-      @stdout, @stderr, @exitcode = Open3.capture3(@script)
+      # Add retries just in case something goes wrong in CI
+      3.times do
+        @stdout, @stderr, @exitcode = Open3.capture3(@script)
+        break if @exitcode.exitstatus.zero?
+      end
     end
 
     it 'should run without error' do
@@ -106,7 +114,11 @@
 
   context 'executing' do
     before(:all) do
-      @stdout, @stderr, @exitcode = Open3.capture3(@script)
+      # Add retries just in case something goes wrong in CI
+      3.times do
+        @stdout, @stderr, @exitcode = Open3.capture3(@script)
+        break if @exitcode.exitstatus.zero?
+      end
     end
 
     it 'should run without error' do