(GH-94) Support for service principal authentication (#93)

Added ability to utilize service_principal_credentials for both hiera lookup function and the puppet 4 function.

Author: kev-in-shu

README.md

@@ -11,13 +11,15 @@
 
 1. [Description](#description)
 1. [Setup](#setup)
+1. [Managed Service Identity (MSI) vs Service Principal Credentials](#managed-service-identity-msi-vs-service-principal-credentials)
 1. [How it works](#how-it-works)
     * [Puppet Function](#puppet-function)
     * [Hiera Backend](#hiera-backend)
 1. [How it's secure by default](#how-its-secure-by-default)
 1. [Usage](#usage)
     * [Embedding a secret in a file](#embedding-a-secret-in-a-file)
     * [Retrieving a specific version of a secret](#retrieving-a-specific-version-of-a-secret)
+    * [Retrieving a certificate](#retrieving-a-certificate)
 1. [Reference - An under-the-hood peek at what the module is doing and how](#reference)
 1. [Development - Guide for contributing to the module](#development)
 
@@ -32,19 +34,29 @@ The module requires the following:
 
 * Puppet Agent 6.0.0 or later.
 * Azure Subscription with one or more vaults already created and loaded with secrets.
-* Puppet Server running on a machine with Managed Service Identity ( MSI ) and assigned the appropriate permissions
+* One of the following authentication strategies
+  * Managed Service Identity ( MSI )
+    * Puppet Server running on a machine with Managed Service Identity ( MSI ) and assigned the appropriate permissions
   to pull secrets from the vault. To learn more or get help with this please visit https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/tutorial-windows-vm-access-nonaad
+  * Service Principal
+    * Following the required steps to setup a Service Principal.  To learn more or get help with this please visit https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app
+
+# Managed Service Identity (MSI) vs Service Principal Credentials
+
+This module provides 2 ways for users to authenticate with azure key vault and pull secrets. These 2 options are Managed Service Identity ( MSI ) and Service Principal Credentials.  We highly recommend you utilize Managed Service Identity over service principal credentials whenever possible.  This is because you do not have to manage and secure a file on our machines that contain credentials!  In some cases, Managed Service Identity ( MSI ) might not be an option for you.  One example of this is  if your Puppet server and some of your puppet agents are not hosted in Azure.  In that case, you can create a Service Principal in Azure Active Directory, assign the appropriate permissions to this Service Principal, and both the function and Hiera Backend provided in this module can authenticate to Azure Keyvault using the credentials of this Service Principal.
 
 ## How it works
 
 ### Puppet Function
 
-This module contains a Puppet 4 function that allows you to securely retrieve secrets from Azure Key Vault.  In order to get started simply call the function in your manifests passing in the required parameters:
+This module contains a Puppet 4 function that allows you to securely retrieve secrets from Azure Key Vault.  In order to get started simply call the function in your manifests passing in the required parameters.
+
+#### Using Managed Service Identity ( MSI )
 
 ```puppet
 $important_secret = azure_key_vault::secret('production-vault', 'important-secret', {
-  metadata_api_version => '2018-04-02',
   vault_api_version    => '2016-10-01',
+  metadata_api_version => '2018-04-02',
 })
 ```
 
@@ -57,10 +69,30 @@ In the above example the api_versions hash is important.  It is pinning both of
 * Instance Metadata Service Versions ( https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service )
 * Vault Versions ( TBD )
 
+
+#### Using Service Principal Credentials
+
+
+```puppet
+$important_secret = azure_key_vault::secret('production-vault', 'important-secret', {
+  vault_api_version    => '2016-10-01',
+  service_principal_credentials => {
+    tenant_id     => '00000000-0000-1234-1234-000000000000',
+    client_id     => '00000000-0000-1234-1234-000000000000',
+    client_secret => lookup('azure_client_secret'),
+  }
+})
+```
+
+This example show how to utilize service principal credentials if you for some reason are unable to use Managed Service Identity ( MSI ) at your organization.  The client_secret must be of type "Sensitive".  Please ensure you configure hiera to return the value wrapped in this type as this is what the secret function expects to ensure there is possibilty of leaking the client_secret.
+
 ### Hiera Backend
 
 This module contains a Hiera 5 backend that allows you to securely retrieve secrets from Azure key vault and use them in hiera.
 
+
+#### Using Managed Service Identity ( MSI )
+
 Add a new entry to the `hierarchy` hash in `hiera.yaml` providing the following required lookup options:
 
 ```yaml
@@ -77,6 +109,33 @@ Add a new entry to the `hierarchy` hash in `hiera.yaml` providing the following
         - '^password.*'
 ```
 
+#### Using Service Principal Credentials
+
+To utilize service principal credentials in hiera simply replace `metadata_api_version` with `service_principal_credentials` and ensure it points to a valid yaml file that contains the service principal credentials you would like to use.
+
+```yaml
+- name: 'Azure Key Vault Secrets'
+    lookup_key: azure_key_vault::lookup
+    options:
+      vault_name: production-vault
+      vault_api_version: '2016-10-01'
+      service_principal_credentials: '/etc/puppetlabs/puppet/azure_key_vault_credentials.yaml'
+      key_replacement_token: '-'
+      confine_to_keys:
+        - '^azure_.*'
+        - '^.*_password$'
+        - '^password.*'
+
+```
+
+Below is the format of the file that is expected to contain your service principal credentials.
+
+```yaml
+tenant_id: '00000000-0000-1234-1234-000000000000'
+client_id: '00000000-0000-1234-1234-000000000000'
+client_secret: some-secret
+```
+
 To retrieve a secret in puppet code you can use the `lookup` function:
 
 ```puppet
@@ -110,6 +169,8 @@ Alternatively a custom trusted fact can be included [in the certificate request]
         - '^password.*'
 ```
 
+**NOTE: While the above examples show manual lookups happening, it's recommended and considered a best practice to utilize Hiera's automatic parameter lookup (APL) within your puppet code**
+
 ### What is confine_to_keys?
 
 By design, hiera will traverse the configured heiarchy for a given key until one is found.  This means that there can be a potentially large number of web requests against azure key vault. In order to improve performance and prevent hitting the Azure KeyVault rate limits ( ex: currently there is a maximum of 2,000 lookups every 10 seconds allowed against a key vault), the confine_to_keys allows you to provide an array of regexs that help avoid making a remote call.

lib/puppet/functions/azure_key_vault/lookup.rb

@@ -3,7 +3,14 @@
 Puppet::Functions.create_function(:'azure_key_vault::lookup') do
   dispatch :lookup_key do
     param 'Variant[String, Numeric]', :secret_name
-    param 'Struct[{vault_name => String, vault_api_version => String, metadata_api_version => String, confine_to_keys => Array[String], Optional[key_replacement_token] => String}]', :options
+    param 'Struct[{
+      vault_name => String,
+      vault_api_version => String,
+      Optional[metadata_api_version] => String,
+      confine_to_keys => Array[String],
+      Optional[key_replacement_token] => String,
+      Optional[service_principal_credentials] => String
+    }]', :options
     param 'Puppet::LookupContext', :context
     return_type 'Variant[Sensitive, Undef]'
   end
@@ -35,7 +42,21 @@ def lookup_key(secret_name, options, context)
     return Puppet::Pops::Types::PSensitiveType::Sensitive.new(context.cached_value(normalized_secret_name)) if context.cache_has_key(normalized_secret_name)
     access_token = context.cached_value('access_token')
     if access_token.nil?
-      access_token = TragicCode::Azure.get_access_token(options['metadata_api_version'])
+      metadata_api_version = options['metadata_api_version']
+      service_principal_credentials = options['service_principal_credentials']
+      if metadata_api_version && service_principal_credentials
+        raise ArgumentError, 'metadata_api_version and service_principal_credentials cannot be used together'
+      end
+      if !metadata_api_version && !service_principal_credentials
+        raise ArgumentError, 'must configure at least one of metadata_api_version or service_principal_credentials'
+      end
+
+      if service_principal_credentials
+        credentials = YAML.load_file(service_principal_credentials)
+        access_token = TragicCode::Azure.get_access_token_service_principal(credentials)
+      else
+        access_token = TragicCode::Azure.get_access_token(metadata_api_version)
+      end
       context.cache('access_token', access_token)
     end
     begin

lib/puppet/functions/azure_key_vault/secret.rb

@@ -4,34 +4,61 @@
 Puppet::Functions.create_function(:'azure_key_vault::secret', Puppet::Functions::InternalFunction) do
   # @param vault_name Name of the vault in your Azure subscription.
   # @param secret_name Name of the secret to be retrieved.
-  # @param api_versions_hash A Hash of the exact versions of the metadata_api_version and vault_api_version to use.
+  # @param api_endpoint_hash A Hash with API endpoint and authentication information
   # @param secret_version The version of the secret you want to retrieve.  This parameter is optional and if not passed the default behavior is to retrieve the latest version.
   # @return [Sensitive[String]] Returns the secret as a String wrapped with the Sensitive data type.
   dispatch :secret do
     cache_param
     required_param 'String', :vault_name
     required_param 'String', :secret_name
-    required_param 'Hash',   :api_versions_hash
+    param 'Struct[{
+      vault_api_version => String,
+      Optional[metadata_api_version] => String,
+      Optional[service_principal_credentials] => Struct[{
+        tenant_id => String,
+        client_id => String,
+        client_secret => String
+      }]
+    }]', :api_endpoint_hash
     optional_param 'String', :secret_version
     return_type 'Sensitive[String]'
   end
 
-  def secret(cache, vault_name, secret_name, api_versions_hash, secret_version = '')
+  def secret(cache, vault_name, secret_name, api_endpoint_hash, secret_version = '')
     Puppet.debug("vault_name: #{vault_name}")
     Puppet.debug("secret_name: #{secret_name}")
     Puppet.debug("secret_version: #{secret_version}")
-    Puppet.debug("metadata_api_version: #{api_versions_hash['metadata_api_version']}")
-    Puppet.debug("vault_api_version: #{api_versions_hash['vault_api_version']}")
+    Puppet.debug("metadata_api_version: #{api_endpoint_hash['metadata_api_version']}")
+    Puppet.debug("vault_api_version: #{api_endpoint_hash['vault_api_version']}")
+    if api_endpoint_hash['service_principal_credentials']
+      partial_credentials = api_endpoint_hash['service_principal_credentials'].slice('tenant_id', 'client_id')
+      Puppet.debug("service_principal_credentials: #{partial_credentials}")
+    end
     cache_hash = cache.retrieve(self)
-    unless cache_hash.key?(:access_token)
+    access_token_id = :"access_token_#{vault_name}"
+    unless cache_hash.key?(access_token_id)
       Puppet.debug("retrieving access token since it's not in the cache")
-      cache_hash[:access_token] = TragicCode::Azure.get_access_token(api_versions_hash['metadata_api_version'])
+      metadata_api_version = api_endpoint_hash['metadata_api_version']
+      service_principal_credentials = api_endpoint_hash['service_principal_credentials']
+      if metadata_api_version && service_principal_credentials
+        raise ArgumentError, 'metadata_api_version and service_principal_credentials cannot be used together'
+      end
+
+      if service_principal_credentials
+        access_token = TragicCode::Azure.get_access_token_service_principal(service_principal_credentials)
+      elsif metadata_api_version
+        access_token = TragicCode::Azure.get_access_token(metadata_api_version)
+      else
+        raise ArgumentError, 'hash must contain at least one of metadata_api_version or service_principal_credentials'
+      end
+      cache_hash[access_token_id] = access_token
     end
+
     secret_value = TragicCode::Azure.get_secret(
       vault_name,
       secret_name,
-      api_versions_hash['vault_api_version'],
-      cache_hash[:access_token],
+      api_endpoint_hash['vault_api_version'],
+      cache_hash[access_token_id],
       secret_version,
     )
 

lib/puppet_x/tragiccode/azure.rb

@@ -19,6 +19,22 @@ def self.get_access_token(api_version)
       JSON.parse(res.body)['access_token']
     end
 
+    def self.get_access_token_service_principal(credentials)
+      uri = URI("https://login.microsoftonline.com/#{credentials.fetch('tenant_id')}/oauth2/v2.0/token")
+      data = {
+        'grant_type': 'client_credentials',
+        'client_id': credentials.fetch('client_id'),
+        'client_secret': credentials.fetch('client_secret'),
+        'scope': 'https://vault.azure.net/.default'
+      }
+      req = Net::HTTP::Post.new(uri.request_uri)
+      res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
+        http.request(req, URI.encode_www_form(data))
+      end
+      raise res.body unless res.is_a?(Net::HTTPSuccess)
+      JSON.parse(res.body)['access_token']
+    end
+
     def self.get_secret(vault_name, secret_name, vault_api_version, access_token, secret_version)
       version_parameter = secret_version.empty? ? secret_version : "/#{secret_version}"
       uri = URI("https://#{vault_name}.vault.azure.net/secrets/#{secret_name}#{version_parameter}?api-version=#{vault_api_version}")

spec/functions/azure_key_vault_lookup_spec.rb

@@ -9,6 +9,7 @@
       'confine_to_keys' => ['^.*sensitive_azure.*'],
     }
   end
+
   let(:lookup_context) do
     environment = instance_double('environment')
     allow(environment).to receive(:name).and_return('production')
@@ -27,6 +28,7 @@
   it 'accepts 3 required arguments' do
     is_expected.to run.with_params.and_raise_error(ArgumentError, %r{expects 3 arguments}i)
   end
+
   it 'validates the :options hash' do
     is_expected.to run.with_params(
       'profile::windows::sqlserver::sensitive_azure_sql_user_password', { 'key1' => 'value1' }, lookup_context
@@ -40,6 +42,7 @@
 
     expect(subject.execute('profile::windows::sqlserver::sensitive_azure_sql_user_password', options, lookup_context).unwrap).to eq 'value'
   end
+
   # rubocop:enable RSpec/NamedSubject
 
   # rubocop:disable RSpec/NamedSubject
@@ -65,7 +68,7 @@
   end
 
   # rubocop:disable RSpec/NamedSubject
-  it 'uses - as the default key_replacement_token' do
+  it "uses '-' as the default key_replacement_token" do
     secret_name = 'profile::windows::sqlserver::sensitive_azure_sql_user_password'
     access_token_value = 'access_value'
     secret_value = 'secret_value'
@@ -83,6 +86,20 @@
     ).and_raise_error(ArgumentError, %r{'confine_to_keys' expects an Array value}i)
   end
 
+  it "errors when using both 'metadata_api_version' and 'service_principal_credentials'" do
+    is_expected.to run.with_params(
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', options.merge({ 'service_principal_credentials' => 'path' }), lookup_context
+    ).and_raise_error(ArgumentError, %r{metadata_api_version and service_principal_credentials cannot be used together}i)
+  end
+
+  it "errors when missing both 'metadata_api_version' and 'service_principal_credentials'" do
+    bad_options = options
+    bad_options.delete('metadata_api_version')
+    is_expected.to run.with_params(
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', bad_options, lookup_context
+    ).and_raise_error(ArgumentError, %r{must configure at least one of metadata_api_version or service_principal_credentials}i)
+  end
+
   it 'errors when passing invalid regexes' do
     is_expected.to run.with_params(
       'profile::windows::sqlserver::sensitive_azure_sql_user_password', options.merge({ 'confine_to_keys' => ['['] }), lookup_context

spec/functions/azure_key_vault_secret_spec.rb

@@ -14,10 +14,37 @@
   let(:access_token) { 'random-access-token' }
   let(:secret_version) { 'a7f7es9a7d' }
 
+  PuppetSensitive = Puppet::Pops::Types::PSensitiveType::Sensitive
+
   it { is_expected.not_to eq(nil) }
 
-  context 'when passed the wrong number of arguments' do
-    it { is_expected.to run.with_params.and_raise_error(ArgumentError, %r{expects between 3 and 4 arguments}i) }
+  context 'when passed the wrong arguments' do
+    it 'errors when wrong number of arguments' do
+      is_expected.to run.with_params.and_raise_error(ArgumentError, %r{expects between 3 and 4 arguments}i)
+    end
+
+    it "errors when using both 'metadata_api_version' and 'service_principal_credentials'" do
+      bad_hash = {
+        'metadata_api_version' => 'test',
+        'vault_api_version' => 'test',
+        'service_principal_credentials' => {
+          'tenant_id' => 'test_tenant',
+          'client_id' => 'test_client',
+          'client_secret' => 'test_secret'
+        }
+      }
+      is_expected.to run.with_params(
+        vault_name, secret_name, bad_hash
+      ).and_raise_error(%r{metadata_api_version and service_principal_credentials cannot be used together})
+    end
+
+    it "errors when missing both 'metadata_api_version' and 'service_principal_credentials'" do
+      bad_hash = api_versions_hash
+      bad_hash.delete('metadata_api_version')
+      is_expected.to run.with_params(
+        vault_name, secret_name, bad_hash
+      ).and_raise_error(%r{hash must contain at least one of metadata_api_version or service_principal_credentials})
+    end
   end
 
   context 'when getting the latest version of a secret' do
@@ -74,4 +101,34 @@
     subject.execute(vault_name, secret_name, api_versions_hash)
   end
   # rubocop:enable RSpec/NamedSubject
+
+  context 'authenticating with service principal' do
+    let(:api_versions_hash) do
+      {
+        'vault_api_version' => 'test_version',
+        'service_principal_credentials' => {
+          'tenant_id' => 'test_tenant',
+          'client_id' => 'test_client',
+          'client_secret' => 'test_secret'
+        }
+      }
+    end
+
+    it 'returns the secret' do
+      expect(TragicCode::Azure).to receive(:get_access_token_service_principal).with(api_versions_hash['service_principal_credentials']).and_return(access_token)
+      expect(TragicCode::Azure).to receive(:get_secret).with(vault_name, secret_name, api_versions_hash['vault_api_version'], access_token, '').and_return(secret_value)
+
+      is_expected.to run.with_params(vault_name, secret_name, api_versions_hash).and_return(PuppetSensitive.new(secret_value))
+    end
+
+    # rubocop:disable RSpec/NamedSubject
+    it 'retrieves access_token from cache' do
+      expect(TragicCode::Azure).to receive(:get_access_token_service_principal).and_return(access_token).once
+      expect(TragicCode::Azure).to receive(:get_secret).and_return(secret_value).twice
+
+      subject.execute(vault_name, secret_name, api_versions_hash)
+      subject.execute(vault_name, secret_name, api_versions_hash)
+    end
+    # rubocop:enable RSpec/NamedSubject
+  end
 end