Merge branch '436-java-memory-assistant' into 3.x

Author: nebhale

README.md

@@ -81,6 +81,7 @@ To learn how to configure various properties of the buildpack, follow the "Confi
 	* [Dynatrace SaaS/Managed OneAgent](docs/framework-dynatrace_one_agent.md) ([Configuration](docs/framework-dynatrace_one_agent.md#configuration))
 	* [Google Stackdriver Debugger](docs/framework-google_stackdriver_debugger.md) ([Configuration](docs/framework-google_stackdriver_debugger.md#configuration))
 	* [Introscope Agent](docs/framework-introscope_agent.md) ([Configuration](docs/framework-introscope_agent.md#configuration))
+	* [Java Memory Assistant](docs/framework-java_memory_assistant.md) ([Configuration](docs/framework-java_memory_assistant.md#configuration))
 	* [Java Options](docs/framework-java_opts.md) ([Configuration](docs/framework-java_opts.md#configuration))
 	* [JRebel Agent](docs/framework-jrebel_agent.md) ([Configuration](docs/framework-jrebel_agent.md#configuration))
 	* [JMX](docs/framework-jmx.md) ([Configuration](docs/framework-jmx.md#configuration))

config/components.yml

@@ -48,6 +48,7 @@ frameworks:
   - "JavaBuildpack::Framework::DynatraceOneAgent"
   - "JavaBuildpack::Framework::GoogleStackdriverDebugger"
 # - "JavaBuildpack::Framework::IntroscopeAgent"
+  - "JavaBuildpack::Framework::JavaMemoryAssistant"
   - "JavaBuildpack::Framework::Jmx"
   - "JavaBuildpack::Framework::JrebelAgent"
   - "JavaBuildpack::Framework::LunaSecurityProvider"

config/java_memory_assistant.yml

@@ -0,0 +1,36 @@
+# Cloud Foundry Java Buildpack
+# Copyright 2013-2017 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+---
+enabled: false
+agent:
+  version: 0.+
+  repository_root: https://raw.githubusercontent.com/SAP/java-memory-assistant/repository
+  heap_dump_folder:
+  check_interval: 5s
+  max_frequency: 1/1m
+  log_level:
+  thresholds:
+    heap:
+    code_cache:
+    metaspace:
+    perm_gen:
+    compressed_class:
+    eden:
+    survivor:
+    old_gen: ">600MB"
+clean_up:
+  version: 0.+
+  repository_root: https://raw.githubusercontent.com/SAP/java-memory-assistant-tools/repository-cu
+  max_dump_count: 1

docs/framework-java_memory_assistant.md

@@ -0,0 +1,117 @@
+# Java Memory Assistant Framework
+The Java Memory Assistant is a Java agent (as in `-javaagent`) that creats heap dumps of your application automatically based on preconfigured conditions of memory usage.
+The heap dumps created by the Java Memory Assistant can be analyzed using Java memory profilers that support the `.hprof` format (i.e., virtually all profilers).
+
+<table>
+  <tr>
+    <td><strong>Detection Criterion</strong></td><td><code>enabled</code> set in the <code>config/java_memory_assistant.yml</code></td>
+  </tr>
+  <tr>
+    <td><strong>Tags</strong></td><td><tt>java-memory-assistant=&lt;version&gt;</tt></td>
+  </tr>
+</table>
+Tags are printed to standard output by the buildpack detect script.
+
+## Configuration
+For general information on configuring the buildpack, including how to specify configuration values through environment variables, refer to [Configuration and Extension][].
+
+The framework can be configured by modifying the [`config/java_memory_assistant.yml`][] file in the buildpack fork.
+
+| Name | Description
+| ---- | -----------
+| `enabled` | Whether to enable the Java Memory Assistant framework. By default the agent is turned off.
+| `agent.heap_dump_folder` | The folder on the container's filesystem where heap dumps are created. Default value: `$PWD`
+| `agent.thresholds.<memory_area>` | This configuration allows to define thresholds for every memory area of the JVM. Thresholds can be defined in absolute percentages, e.g., `75%` creates a heap dump at 75% of the selected memory area. It is also possible to specify relative increases and decreases of memory usage: for example, `+5%/2m` will triggera heap dumpo if the particular memory area has increased by `5%` or more over the last two minutes. See below to check which memory areas are supported. Since version `0.3.0`, thresholds can also be specified in terms of absolute values, e.g., `>400MB` (more than 400 MB) or `<=30KB` (30 KB or less); supported memory size units are `KB`, `MB` and `GB`.
+| `agent.check_interval` | The interval between checks. Examples: `1s` (once a second), `3m` (every three minutes), `1h` (once every hour). Default: `5s` (check every five seconds).
+| `agent.max_frequency` | Maximum amount of heap dumps that the Java Memory Assistant is allowed to create in a given amount of time. Examples: `1/30s` (no more than one heap dump every thirty seconds), `2/3m` (up to two heap dumps every three minutes), `1/2h` (one heap dump every two hours). The time interval is checked every time one heap dump *should* be created (based on the specified thresholds), and compared with the timestamps of the previously created heap dumps to make sure that the maximum frequency is not exceeded. Default: `1/1m` (one heap dump per minute). |
+| `agent.log_level` | The log level used by the Java Memory Assistant. Supported values are the same as the Java buildpack's: `DEBUG`, `WARN`, `INFO`, `ERROR` and `FATAL` (the latter is equivalent to `ERROR`). If the `agent.log_level` is not specified, the Java buildpack's log level will be used. |
+| `clean_up.max_dump_count` | Maximum amount of heap dumps that can be stored in the filesystem of the container; when the creation of a new heap dump would cause the threshold to be surpassed, the oldest heap dumps are removed from the file system. Default value: `1` |
+
+### Heap Dump Names
+
+The heap dump filenames will be generated according to the following name pattern:
+
+`<INSTANCE-INDEX>-%ts:yyyyMMdd'T'mmssSSSZ%-<INSTANCE-ID[0,8]>.hprof`
+
+The timestamp pattern `%ts:yyyyMMdd'T'mmssSSSZ%` is equivalent to the `%FT%T%z` pattern of [strftime](http://www.cplusplus.com/reference/ctime/strftime/) for [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601).
+
+### Supported Memory Areas
+
+| Memory Area            | Property Name    |
+|------------------------|------------------|
+| Heap                   | `heap`             |
+| Code Cache             | `code_cache`       |
+| Metaspace              | `metaspace`        |
+| Compressed Class Space | `compressed_class` |
+| Eden                   | `eden`             |
+| Survivor               | `survivor`         |
+| Old Generation         | `old_gen`          |
+
+The default values can be found in the [`config/java_memory_assistant.yml`][] file.
+
+### Examples
+
+Enable the Java Memory Assistant with its default settings:
+
+```yaml
+JBP_CONFIG_JAVA_MEMORY_ASSISTANT: '{enabled : true}'
+```
+
+Create heap dumps when the old generation memory pool exceeds 800 MB:
+
+```yaml
+JBP_CONFIG_JAVA_MEMORY_ASSISTANT: '{enabled : true, agent: { thresholds : { old_gen : ">800MB" } } }'
+```
+
+Create heap dumps when the old generation grows by more than 20% in two minutes:
+
+```yaml
+JBP_CONFIG_JAVA_MEMORY_ASSISTANT: '{enabled : true, agent : { thresholds : { old_gen : +20%/2m } } }'
+```
+
+### What are the right thresholds for your application?
+
+Well, it depends.
+The way applications behave in terms of memory management is a direct result of how they are implemented.
+This is much more then case when the applications are under heavy load.
+Thus, there is no "silver bullet" configuration that will serve all applications equally well, and Java Memory Assistant configurations should result from profiling the application under load and then encode the expected memory usage patterns (plus a margin upwards) to detect anomalies.
+
+Nevertheless, a memory area that tends to be particularly interesting to monitor is the so called "old generation" (`old_gen`).
+When instantiated, bjects in the Java heap are allocated in an area called `eden`.
+As garbage collections occur, objects that are not reclaimed become first "survivors" (and belong to the namesake `survivor` memory area) and then eventually become `old_gen`.
+In other words, `old_gen` objects are those that survived multiple garbage collections.
+In contrast, `eden` and `survivor` objects are collectively called "young generation".
+
+Application-wide singletons and pooled objects (threads, connections) are examples of "legitimate" `old_gen` candidates.
+But memory leaks, by their very nature or surviving multiple garbage collections, end up in `old_gen` too.
+Under load that is not too high for the application (and you should find out what it is with load tests and avoid it via rate limiting, e.g., using [route services](https://docs.cloudfoundry.org/services/route-services.html) in front of your application), Java code that allows the JVM to perform efficient memory management tends to have a rather consistent baseline of `old_gen` objects, with most objects being reclaimed as they are still young generation.
+That is, when the `old_gen` grows large with respect to the overall heap, this often signifies some sort of memory leak or, at the very least, suboptimal memory management.
+Notable exceptions to this rule of thumb are applications that use large local caches.
+
+### Making sure heap dumps can be created
+
+The Java Virtual Machine must create heap dumps on a file.
+Unless you are using a `volume service`, it pretty much means that, even if you are uploading the heap dump somewhere else, the heap dump must first land on the ephemeral disk of the container.
+Ephemeral disks have quotas and, if all the space is taken by heap dumps (even incomplete ones!), horrible things are bound to happen to your app.
+
+The maximum size of a heap dump depends on the maximum size of the heap of the Java Virtual Machine.
+Consider increasing the disk quota of your warden/garden container via the `cf scale <app-name> -k [new size]` using as `new size` to the outcome of the following calculation:
+
+`[max heap size] * [max heap dump count] + 200MB`
+
+The aditional `200MB` is a rule-of-thumb, generous over-approximation of the amount of disk the buildpack and the application therein needs to run.
+If your application requires more filesystem than just a few tens of megabytes, you must increase the additional portion of the disk amount calculation accordingly.
+
+### Where to best store heap dumps?
+
+Heap dumps are created by the Java Virtual Machine on a file on the filesystem mounted by the garden container.
+Normally, the filesystem of a container is ephemeral.
+That is, if your app crashes or it is shut down, the filesystem of its container is gone with it and so are your heap dumps.
+
+To prevent heap dumps from "going down" with the container, you should consider storing them on a `volume service`.
+
+#### Container-mounted volumes
+
+If you are using a filesystem service that mounts persistent volumes to the container, it is enough to name one of the volume services `heap-dump` or tag one volume with `heap-dump`, and the path specified as the `heap_dump_folder` configuration will be resolved against `<mount-point>/<space_name>-<space_id[0,8]>/<application_name>-<application_id[0-8]>`.
+
+[`config/java_memory_assistant.yml`]: ../config/java_memory_assistant.yml

lib/java_buildpack/framework/java_memory_assistant.rb

@@ -0,0 +1,49 @@
+# Cloud Foundry Java Buildpack
+# Copyright 2013-2017 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'java_buildpack/component/modular_component'
+require 'java_buildpack/framework'
+require 'java_buildpack/framework/java_memory_assistant/agent'
+require 'java_buildpack/framework/java_memory_assistant/clean_up'
+require 'java_buildpack/framework/java_memory_assistant/heap_dump_folder'
+
+module JavaBuildpack
+  module Framework
+
+    # Encapsulates the integraton of the JavaMemoryAssistant.
+    class JavaMemoryAssistant < JavaBuildpack::Component::ModularComponent
+
+      protected
+
+      # (see JavaBuildpack::Component::ModularComponent#command)
+      def command; end
+
+      # (see JavaBuildpack::Component::ModularComponent#sub_components)
+      def sub_components(context)
+        [
+          JavaMemoryAssistantAgent.new(sub_configuration_context(context, 'agent')),
+          JavaMemoryAssistantHeapDumpFolder.new(sub_configuration_context(context, 'agent')),
+          JavaMemoryAssistantCleanUp.new(sub_configuration_context(context, 'clean_up'))
+        ]
+      end
+
+      # (see JavaBuildpack::Component::ModularComponent#supports?)
+      def supports?
+        @configuration['enabled']
+      end
+
+    end
+  end
+end

lib/java_buildpack/framework/java_memory_assistant/agent.rb

@@ -0,0 +1,99 @@
+# Cloud Foundry Java Buildpack
+# Copyright 2013-2017 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'fileutils'
+require 'java_buildpack/component'
+require 'java_buildpack/component/versioned_dependency_component'
+require 'java_buildpack/component/droplet'
+require 'java_buildpack/component/environment_variables'
+require 'java_buildpack/framework'
+
+module JavaBuildpack
+  module Framework
+
+    # Encapsulates the integraton of the JavaMemoryAssistant to inject the agent in the JVM.
+    class JavaMemoryAssistantAgent < JavaBuildpack::Component::VersionedDependencyComponent
+
+      # (see JavaBuildpack::Component::BaseComponent#compile)
+      def compile
+        download_jar
+        @droplet.copy_resources
+      end
+
+      # (see JavaBuildpack::Component::BaseComponent#release)
+      def release
+        @droplet.java_opts
+                .add_javaagent(@droplet.sandbox + jar_name)
+                .add_system_property('jma.enabled', 'true')
+                .add_system_property('jma.heap_dump_name', %("#{name_pattern}"))
+                .add_system_property 'jma.log_level', normalized_log_level
+
+        add_system_prop_if_config_present 'check_interval', 'jma.check_interval'
+        add_system_prop_if_config_present 'max_frequency', 'jma.max_frequency'
+
+        return unless @configuration.key?('thresholds')
+
+        @configuration['thresholds'].each do |key, value|
+          @droplet.java_opts.add_system_property "jma.thresholds.#{key}", value.to_s
+        end
+      end
+
+      protected
+
+      # (see JavaBuildpack::Component::VersionedDependencyComponent#jar_name)
+      def jar_name
+        "java-memory-assistant-#{@version}.jar"
+      end
+
+      def supports?
+        true
+      end
+
+      private
+
+      LOG_LEVEL_MAPPING = {
+        'DEBUG' => 'DEBUG',
+        'WARN' => 'WARNING',
+        'INFO' => 'INFO',
+        'ERROR' => 'ERROR',
+        'FATAL' => 'ERROR'
+      }.freeze
+
+      private_constant :LOG_LEVEL_MAPPING
+
+      def add_system_prop_if_config_present(config_entry, system_property_name)
+        return unless @configuration.key?(config_entry)
+        @droplet.java_opts.add_system_property(system_property_name, @configuration[config_entry])
+      end
+
+      def log_level
+        @configuration['log_level'] || ENV['JBP_LOG_LEVEL'] || 'ERROR'
+      end
+
+      def normalized_log_level
+        normalized_log_level = LOG_LEVEL_MAPPING[log_level.upcase]
+        raise "Invalid value of the 'log_level' property: '#{log_level}'" unless normalized_log_level
+        normalized_log_level
+      end
+
+      def name_pattern
+        # Double escaping quotes of doom. Nothing less would work.
+        %q(%env:CF_INSTANCE_INDEX%-%ts:yyyy-MM-dd'"'"'T'"'"'mm'"'"':'"'"'ss'"'"':'"'"'SSSZ%-) \
+          '%env:CF_INSTANCE_GUID[,8]%.hprof'
+      end
+
+    end
+  end
+end

lib/java_buildpack/framework/java_memory_assistant/clean_up.rb

@@ -0,0 +1,54 @@
+# Cloud Foundry Java Buildpack
+# Copyright 2013-2017 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'java_buildpack/component/versioned_dependency_component'
+require 'java_buildpack/component/droplet'
+require 'java_buildpack/framework'
+
+module JavaBuildpack
+  module Framework
+
+    # Encapsulates the integraton of the JavaMemoryAssistant to set up clean up of dumps.
+    class JavaMemoryAssistantCleanUp < JavaBuildpack::Component::VersionedDependencyComponent
+
+      # (see JavaBuildpack::Component::BaseComponent#compile)
+      def compile
+        return unless supports?
+
+        download_zip false
+      end
+
+      # (see JavaBuildpack::Component::BaseComponent#release)
+      def release
+        return unless supports?
+
+        @droplet.environment_variables
+                .add_environment_variable 'JMA_MAX_DUMP_COUNT', @configuration['max_dump_count'].to_s
+
+        @droplet.java_opts
+                .add_system_property('jma.command.interpreter', '')
+                .add_system_property('jma.execute.before', @droplet.sandbox + 'cleanup')
+      end
+
+      protected
+
+      # (see JavaBuildpack::Component::VersionedDependencyComponent#supports?)
+      def supports?
+        @configuration['max_dump_count'].to_i > 0
+      end
+
+    end
+  end
+end