Adds external configuration support for Luna Security Module (#879)

Daniel Mikusa · web-flow · commit f66db372fa55 · 2021-05-26T10:35:39.000-04:00
- Extracts external configuration code into a utility
- Modifies AppDynamics to use the utility code
- Modifies Luna Security Provider to use the utility code
- Updates documentation for AppDynamics and Luna to explaini how external configuration works
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -15,8 +15,8 @@ GEM
     public_suffix (4.0.4)
     rainbow (3.0.0)
     rake (13.0.1)
-    rexml (3.2.5)
     redcarpet (3.5.1)
+    rexml (3.2.5)
     rspec (3.9.0)
       rspec-core (~> 3.9.0)
       rspec-expectations (~> 3.9.0)
@@ -69,4 +69,4 @@ DEPENDENCIES
   yard
 
 BUNDLED WITH
-   2.1.4
+   2.2.16
diff --git a/docs/framework-app_dynamics_agent.md b/docs/framework-app_dynamics_agent.md
@@ -44,7 +44,51 @@ The framework can be configured by modifying the [`config/app_dynamics_agent.yml
 | `version` | The version of AppDynamics to use. Candidate versions can be found in [this listing][].
 
 ### Additional Resources
-The framework can also be configured by overlaying a set of resources on the default distribution. To do this, add files to the `resources/app_dynamics_agent` directory in the buildpack fork. For example, to override the default `app-agent-config.xml` add your custom file to `resources/app_dynamics_agent/<version>/conf/app-agent-config.xml`.
+The framework can also be configured by overlaying a set of resources on the default distribution.  To do this follow one of the options below.
+
+Configuration files are created in this order:
+
+1. Default AppDynamics configuration
+2. Buildpack default configuration is taken from `resources/app_dynamics_agent/default`
+3. External Configuration if configured
+4. Local Configuration if configured
+5. Buildpack Fork if it exists
+
+#### Buildpack Fork
+Add files to the `resources/app_dynamics_agent` directory in the buildpack fork.  For example, to override the default `app-agent-config.xml` add your custom file to `resources/app_dynamics_agent/<version>/conf/app-agent-config.xml`.
+
+#### External Configuration
+Set `APPD_CONF_HTTP_URL` to an HTTP or HTTPS URL which points to the directory where your configuration files exist. You may also include a user and password in the URL, like `https://user:pass@example.com`.
+
+The Java buildpack will take the URL to the directory provided and attempt to download the following files from that directory:
+
+- `logging/log4j2.xml` 
+- `logging/log4j.xml`
+- `app-agent-config.xml` 
+- `controller-info.xml`
+- `service-endpoint.xml` 
+- `transactions.xml` 
+- `custom-interceptors.xml`
+- `custom-activity-correlation.xml`
+
+Any file successfully downloaded will be copied to the configuration directory. The buildpack does not fail if files are missing.
+
+#### Local Configuration
+Set `APPD_CONF_DIR` to a relative path which points to the directory in your application files where your custom configuration exists.
+
+The Java buildpack will take the `app_root` + `APPD_CONF_DIR` directory and attempt to copy the followinig files from that directory:
+
+- `logging/log4j2.xml`
+- `logging/log4j.xml`
+- `app-agent-config.xml`
+- `controller-info.xml`
+- `service-endpoint.xml`
+- `transactions.xml`
+- `custom-interceptors.xml`
+- `custom-activity-correlation.xml`
+
+Any files that exist will be copied to the configuration directory. The buildpack does not fail if files are missing.
+
 
 [`config/app_dynamics_agent.yml`]: ../config/app_dynamics_agent.yml
 [AppDynamics Java Agent Configuration Properties]: https://docs.appdynamics.com/display/PRO42/Java+Agent+Configuration+Properties
diff --git a/docs/framework-luna_security_provider.md b/docs/framework-luna_security_provider.md
@@ -98,7 +98,27 @@ The framework can be configured by modifying the [`config/luna_security_provider
 | `version` | Version of the Luna Security Provider to use.
 
 ### Additional Resources
-The framework can also be configured by overlaying a set of resources on the default distribution.  To do this, add files to the `resources/luna_security_provider` directory in the buildpack fork.
+The framework can also be configured by overlaying a set of resources on the default distribution.  To do this follow one of the options below.
+
+Configuration files are created in this order:
+
+1. Default configuration
+2. Buildpack fork
+3. Buildpack generated configuration if the bound service has both a `servers` and `groups` key
+4. External configuration if configured
+
+#### Buildpack Fork
+Add files to the `resources/luna_security_provider` directory in the buildpack fork.  For example, to override the default `Chrystoki.conf` add your custom file to `resources/luna_security_provider/Chrystoki.conf`.
+
+#### External Configuration
+Set `LUNA_CONF_HTTP_URL` to an HTTP or HTTPS URL which points to the directory where your configuration files exist. You may also include a user and password in the URL, like `https://user:pass@example.com`.
+
+The Java buildpack will take the URL to the directory provided and attempt to download the following files from that directory:
+
+- `Chrystoki.conf`
+- `server-certificates.pem`
+
+Any file successfully downloaded will be copied to the configuration directory. The buildpack does not fail if files are missing.
 
 [`config/luna_security_provider.yml`]: ../config/luna_security_provider.yml
 [Luna Security Service]: http://www.safenet-inc.com/data-encryption/hardware-security-modules-hsms/
diff --git a/lib/java_buildpack/framework/app_dynamics_agent.rb b/lib/java_buildpack/framework/app_dynamics_agent.rb
@@ -19,12 +19,22 @@
 require 'shellwords'
 require 'java_buildpack/component/versioned_dependency_component'
 require 'java_buildpack/framework'
+require 'java_buildpack/util/external_config'
 
 module JavaBuildpack
   module Framework
 
     # Encapsulates the functionality for enabling zero-touch AppDynamics support.
     class AppDynamicsAgent < JavaBuildpack::Component::VersionedDependencyComponent
+      include JavaBuildpack::Util::ExternalConfig
+
+      # Full list of configuration files that can be downloaded remotely
+      CONFIG_FILES = %w[logging/log4j2.xml logging/log4j.xml app-agent-config.xml controller-info.xml
+                        service-endpoint.xml transactions.xml custom-interceptors.xml
+                        custom-activity-correlation.xml].freeze
+
+      # Prefix to be used with external configuration environment variable
+      CONFIG_PREFIX = 'APPD'
 
       def initialize(context)
         super(context)
@@ -40,7 +50,7 @@ def compile
         default_conf_dir = resources_dir + @droplet.component_id + 'defaults'
 
         copy_appd_default_configuration(default_conf_dir)
-        override_default_config_remote
+        override_default_config_remote(&method(:save_cfg_file))
         override_default_config_local
         @droplet.copy_resources
       end
@@ -71,13 +81,9 @@ def supports?
 
       private
 
-      CONFIG_FILES = %w[logging/log4j2.xml logging/log4j.xml app-agent-config.xml controller-info.xml
-                        service-endpoint.xml transactions.xml custom-interceptors.xml
-                        custom-activity-correlation.xml].freeze
-
       FILTER = /app[-]?dynamics/.freeze
 
-      private_constant :CONFIG_FILES, :FILTER
+      private_constant :FILTER
 
       def application_name(java_opts, credentials)
         name = Shellwords.escape(@application.details['application_name'])
@@ -150,67 +156,6 @@ def copy_appd_default_configuration(default_conf_dir)
         end
       end
 
-      # Check if configuration file exists on the server before download
-      # @param [ResourceURI] uri URI of the remote configuration server
-      # @param [ConfigFileName] conf_file Name of the configuration file
-      # @return [Boolean] returns true if files exists on path specified by APPD_CONF_HTTP_URL, false otherwise
-      def check_if_resource_exists(resource_uri, conf_file)
-        # check if resource exists on remote server
-        begin
-          opts = { use_ssl: true } if resource_uri.scheme == 'https'
-          response = Net::HTTP.start(resource_uri.host, resource_uri.port, opts) do |http|
-            req = Net::HTTP::Head.new(resource_uri)
-            if resource_uri.user != '' || resource_uri.password != ''
-              req.basic_auth(resource_uri.user, resource_uri.password)
-            end
-            http.request(req)
-          end
-        rescue StandardError => e
-          @logger.error { "Request failure: #{e.message}" }
-          return false
-        end
-
-        case response
-        when Net::HTTPSuccess
-          true
-        when Net::HTTPRedirection
-          location = response['location']
-          @logger.info { "redirected to #{location}" }
-          check_if_resource_exists(location, conf_file)
-        else
-          @logger.info { "Could not retrieve #{resource_uri}.  Code: #{response.code} Message: #{response.message}" }
-          false
-        end
-      end
-
-      # Check for configuration files on a remote server. If found, copy to conf dir under each ver* dir
-      # @return [Void]
-      def override_default_config_remote
-        return unless @application.environment['APPD_CONF_HTTP_URL']
-
-        JavaBuildpack::Util::Cache::InternetAvailability.instance.available(
-          true, 'The AppDynamics remote configuration download location is always accessible'
-        ) do
-          agent_root = @application.environment['APPD_CONF_HTTP_URL'].chomp('/') + '/java/'
-          @logger.info { "Downloading override configuration files from #{agent_root}" }
-          CONFIG_FILES.each do |conf_file|
-            uri = URI(agent_root + conf_file)
-
-            # `download()` uses retries with exponential backoff which is expensive
-            # for situations like 404 File not Found. Also, `download()` doesn't expose
-            # an api to disable retries, which makes this check necessary to prevent
-            # long install times.
-            next unless check_if_resource_exists(uri, conf_file)
-
-            download(false, uri.to_s) do |file|
-              Dir.glob(@droplet.sandbox + 'ver*') do |target_directory|
-                FileUtils.cp_r file, target_directory + '/conf/' + conf_file
-              end
-            end
-          end
-        end
-      end
-
       # Check for configuration files locally. If found, copy to conf dir under each ver* dir
       # @return [Void]
       def override_default_config_local
@@ -226,9 +171,13 @@ def override_default_config_local
 
           next unless File.file?(conf_file_path)
 
-          Dir.glob(@droplet.sandbox + 'ver*') do |target_directory|
-            FileUtils.cp_r conf_file_path, target_directory + '/conf/' + conf_file
-          end
+          save_cfg_file(conf_file_path, conf_file)
+        end
+      end
+
+      def save_cfg_file(file, conf_file)
+        Dir.glob(@droplet.sandbox + 'ver*') do |target_directory|
+          FileUtils.cp_r file, target_directory + '/conf/' + conf_file
         end
       end
     end
diff --git a/lib/java_buildpack/framework/luna_security_provider.rb b/lib/java_buildpack/framework/luna_security_provider.rb
@@ -18,6 +18,7 @@
 require 'fileutils'
 require 'java_buildpack/component/versioned_dependency_component'
 require 'java_buildpack/framework'
+require 'java_buildpack/util/external_config'
 require 'java_buildpack/util/qualify_path'
 
 module JavaBuildpack
@@ -26,6 +27,18 @@ module Framework
     # Encapsulates the functionality for enabling zero-touch Safenet Luna HSM Java Security Provider support.
     class LunaSecurityProvider < JavaBuildpack::Component::VersionedDependencyComponent
       include JavaBuildpack::Util
+      include JavaBuildpack::Util::ExternalConfig
+
+      # Full list of configuration files that can be downloaded remotely
+      CONFIG_FILES = %w[Chrystoki.conf server-certificates.pem].freeze
+
+      # Prefix to be used with external configuration environment variable
+      CONFIG_PREFIX = 'LUNA'
+
+      def initialize(context)
+        super(context)
+        @logger = JavaBuildpack::Logging::LoggerFactory.instance.get_logger LunaSecurityProvider
+      end
 
       # (see JavaBuildpack::Component::BaseComponent#compile)
       def compile
@@ -36,10 +49,11 @@ def compile
         @droplet.security_providers << 'com.safenetinc.luna.provider.LunaProvider'
         @droplet.root_libraries << luna_provider_jar if @droplet.java_home.java_9_or_later?
 
-        credentials = @application.services.find_service(FILTER, 'client', 'servers', 'groups')['credentials']
-        write_client credentials['client']
-        write_servers credentials['servers']
-        write_configuration credentials['servers'], credentials['groups']
+        write_credentials
+
+        override_default_config_remote do |file, conf_file|
+          FileUtils.cp_r file, @droplet.sandbox + conf_file
+        end
       end
 
       # (see JavaBuildpack::Component::BaseComponent#release)
@@ -57,11 +71,27 @@ def release
 
       # (see JavaBuildpack::Component::VersionedDependencyComponent#supports?)
       def supports?
-        @application.services.one_service? FILTER, 'client', 'servers', 'groups'
+        @application.services.one_service?(FILTER, 'client', 'servers', 'groups') ||
+          @application.services.one_service?(FILTER, 'client', 'servers') ||
+          @application.services.one_service?(FILTER, 'client')
       end
 
       private
 
+      def write_credentials
+        service = @application.services.find_service(FILTER, 'client', 'servers', 'groups') ||
+          @application.services.find_service(FILTER, 'client', 'servers') ||
+          @application.services.find_service(FILTER, 'client')
+        credentials = service['credentials']
+
+        write_client credentials['client'] if credentials.key? 'client'
+        write_servers credentials['servers'] if credentials.key? 'servers'
+
+        return unless credentials.key?('servers') && credentials.key?('groups')
+
+        write_configuration credentials['servers'], credentials['groups']
+      end
+
       FILTER = /luna/.freeze
 
       private_constant :FILTER
@@ -254,6 +284,11 @@ def write_servers(servers)
         end
       end
 
+      # Overrides method from ExternalConfig module & provides root URL for where external configuration will be located
+      def external_config_root
+        @application.environment["#{CONFIG_PREFIX}_CONF_HTTP_URL"].chomp('/') + '/'
+      end
+
     end
   end
 end
diff --git a/lib/java_buildpack/util/external_config.rb b/lib/java_buildpack/util/external_config.rb
diff --git a/spec/java_buildpack/framework/luna_security_provider_spec.rb b/spec/java_buildpack/framework/luna_security_provider_spec.rb
