(GH-69) Add code and tests for new requires confine_to_keys options (#75)

Adding confine_to_keys option just like modules are doing in order to reduce the number of unnecessary calls to the secrets vault.  This helps resolve the rate limiting issue and slowness people are experiencing

Author: TraGicCode

README.md

@@ -57,7 +57,7 @@ In the above example the api_versions hash is important.  It is pinning both of
 
 This module contains a Hiera 5 backend that allows you to securely retrieve secrets from Azure key vault and use them in hiera.
 
-Add a new entry to the `hierarchy` hash in `hiera.yaml` referencing the vault name and API versions:
+Add a new entry to the `hierarchy` hash in `hiera.yaml` providing the following required lookup options:
 
 ```yaml
 - name: 'Azure Key Vault Secrets'
@@ -67,6 +67,10 @@ Add a new entry to the `hierarchy` hash in `hiera.yaml` referencing the vault na
       vault_api_version: '2016-10-01'
       metadata_api_version: '2018-04-02'
       key_replacement_token: '-'
+      confine_to_keys:
+        - '^azure_.*'
+        - '^.*_password$'
+        - '^password.*'
 ```
 
 To retrieve a secret in puppet code you can use the `lookup` function:
@@ -96,9 +100,35 @@ Alternatively a custom trusted fact can be included [in the certificate request]
       vault_api_version: '2016-10-01'
       metadata_api_version: '2018-04-02'
       key_replacement_token: '-'
+      confine_to_keys:
+        - '^azure_.*'
+        - '^.*_password$'
+        - '^password.*'
 ```
 
-### A note on secret keys
+### What is confine_to_keys?
+
+By design, hiera will traverse the configured heiarchy for a given key until one is found.  This means that there can be a potentially large number of web requests against azure key vault. In order to improve performance and prevent hitting the Azure KeyVault rate limits ( ex: currently there is a maximum of 2,000 lookups every 10 seconds allowed against a key vault), the confine_to_keys allows you to provide an array of regexs that help avoid making a remote call.
+
+As an example, if you defined your confine_to_keys as shown below, hiera will only make a web request to get the secret in azure key vault when the key being lookedup matches atleast one of the provided regular expressions in the confine_to_keys array.
+
+```yaml
+- name: 'Azure Key Vault Secrets from trusted fact'
+    lookup_key: azure_key_vault::lookup
+    options:
+      vault_name: "%{trusted.extensions.pp_environment}"
+      vault_api_version: '2016-10-01'
+      metadata_api_version: '2018-04-02'
+      key_replacement_token: '-'
+      confine_to_keys:
+        - '^azure_.*'
+        - '^.*_password$'
+        - '^password.*'
+```
+
+**NOTE: The confine_to_keys is very important to make you sure get right.  As a best practice, come up with some conventions to avoid having a large number of regexs you have to add/update/remove and also test.  The above example provides a great starting point.**
+
+### What is key_replacement_token?
 
 KeyVault secret names can only contain the characters `0-9`, `a-z`, `A-Z`, and `-`.
 

lib/puppet/functions/azure_key_vault/lookup.rb

@@ -3,13 +3,26 @@
 Puppet::Functions.create_function(:'azure_key_vault::lookup') do
   dispatch :lookup_key do
     param 'Variant[String, Numeric]', :secret_name
-    param 'Struct[{vault_name => String, vault_api_version => String, metadata_api_version => String, Optional[key_replacement_token] => String}]', :options
+    param 'Struct[{vault_name => String, vault_api_version => String, metadata_api_version => String, confine_to_keys => Array[Regexp], Optional[key_replacement_token] => String}]', :options
     param 'Puppet::LookupContext', :context
   end
 
   def lookup_key(secret_name, options, context)
     # This is a reserved key name in hiera
     return context.not_found if secret_name == 'lookup_options'
+
+    confine_keys = options['confine_to_keys']
+    if confine_keys
+      raise ArgumentError, 'confine_to_keys must be an array' unless confine_keys.is_a?(Array)
+
+      regex_key_match = Regexp.union(confine_keys)
+
+      unless secret_name[regex_key_match] == secret_name
+        context.explain { "Skipping azure_key_vault backend because secret_name '#{secret_name}' does not match confine_to_keys" }
+        context.not_found
+      end
+    end
+
     normalized_secret_name = TragicCode::Azure.normalize_object_name(secret_name, options['key_replacement_token'] || '-')
     context.explain { "  Using normalized KeyVault secret key for lookup: #{normalized_secret_name}" }
     return context.cached_value(normalized_secret_name) if context.cache_has_key(normalized_secret_name)

spec/functions/azure_key_vault_lookup_spec.rb

@@ -6,6 +6,7 @@
       'vault_name' => 'vault_name',
       'vault_api_version' => 'vault_api_version',
       'metadata_api_version' => 'metadata_api_version',
+      'confine_to_keys' => [%r{^.*sensitive_azure.*}],
     }
   end
   let(:lookup_context) do
@@ -28,14 +29,14 @@
   end
   it 'validates the :options hash' do
     is_expected.to run.with_params(
-      'secret_name', { 'key1' => 'value1' }, lookup_context
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', { 'key1' => 'value1' }, lookup_context
     ).and_raise_error(ArgumentError)
   end
   it 'uses the cache' do
-    expect(lookup_context).to receive(:cache_has_key).with('secret-name').and_return(true)
-    expect(lookup_context).to receive(:cached_value).with('secret-name').and_return('value')
+    expect(lookup_context).to receive(:cache_has_key).with('profile--windows--sqlserver--sensitive-azure-sql-user-password').and_return(true)
+    expect(lookup_context).to receive(:cached_value).with('profile--windows--sqlserver--sensitive-azure-sql-user-password').and_return('value')
     is_expected.to run.with_params(
-      'secret_name', options, lookup_context
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', options, lookup_context
     ).and_return('value')
   end
   it 'caches the access token after a cache miss' do
@@ -48,7 +49,7 @@
     expect(TragicCode::Azure).to receive(:get_secret).and_return(secret_value)
     expect(lookup_context).to receive(:cache).and_return(secret_value).ordered
     is_expected.to run.with_params(
-      'secret_name', options, lookup_context
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', options, lookup_context
     ).and_return(secret_value)
   end
 
@@ -60,14 +61,49 @@
   end
 
   it 'uses - as the default key_replacement_token' do
-    secret_name = 'profile::windows::sqlserver::sensitive_sql_user_password'
+    secret_name = 'profile::windows::sqlserver::sensitive_azure_sql_user_password'
     access_token_value = 'access_value'
     secret_value = 'secret_value'
     expect(TragicCode::Azure).to receive(:normalize_object_name).with(secret_name, '-')
     expect(TragicCode::Azure).to receive(:get_access_token).and_return(access_token_value)
     expect(TragicCode::Azure).to receive(:get_secret).and_return(secret_value)
     is_expected.to run.with_params(
-      'profile::windows::sqlserver::sensitive_sql_user_password', options, lookup_context
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', options, lookup_context
     ).and_return(secret_value)
   end
+
+  it 'errors when confine_to_keys is no array' do
+    is_expected.to run.with_params(
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', options.merge({ 'confine_to_keys' => '^vault.*$' }), lookup_context
+    ).and_raise_error(ArgumentError, %r{'confine_to_keys' expects an Array value}i)
+  end
+
+  it 'errors when passing invalid regexes' do
+    is_expected.to run.with_params(
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', options.merge({ 'confine_to_keys' => ['['] }), lookup_context
+    ).and_raise_error(ArgumentError, %r{'confine_to_keys' index 0 expects a Regexp value}i)
+  end
+
+  it 'returns the key if regex matches confine_to_keys' do
+    access_token_value = 'access_value'
+    secret_value = 'secret_value'
+    expect(TragicCode::Azure).to receive(:get_access_token).and_return(access_token_value)
+    expect(TragicCode::Azure).to receive(:get_secret).and_return(secret_value)
+    is_expected.to run.with_params(
+      'profile::windows::sqlserver::sensitive_azure_sql_user_password', options.merge({ 'confine_to_keys' => [%r{^.*sensitive_azure.*}] }), lookup_context
+    ).and_return(secret_value)
+  end
+
+  it 'does not return the key if regex does not match confine_to_keys' do
+    access_token_value = 'access_value'
+    secret_value = 'secret_value'
+
+    expect(lookup_context).to receive(:not_found)
+    expect(TragicCode::Azure).to receive(:get_access_token).and_return(access_token_value)
+    expect(TragicCode::Azure).to receive(:get_secret).and_return(secret_value)
+
+    is_expected.to run.with_params(
+      'profile::windows::sqlserver::sensitive_sql_user_password', options.merge({ 'confine_to_keys' => [%r{^sensitive_azure.*$}] }), lookup_context
+    )
+  end
 end