Gradle builder for Java (#69)

Author: dagnir

.appveyor.yml

@@ -4,6 +4,7 @@ image: Visual Studio 2017
 environment:
   GOPATH: c:\gopath
   GOVERSION: 1.11
+  GRADLE_OPTS: -Dorg.gradle.daemon=false
 
   matrix:
 
@@ -36,6 +37,9 @@ install:
 - "go version"
 - "go env"
 
+# setup Gradle
+- "choco install gradle"
+
 test_script:
 - "%PYTHON%\\python.exe -m pytest --cov aws_lambda_builders --cov-report term-missing tests/unit tests/functional"
 - "%PYTHON%\\python.exe -m pytest tests/integration"

aws_lambda_builders/workflows/__init__.py

@@ -7,3 +7,4 @@
 import aws_lambda_builders.workflows.ruby_bundler
 import aws_lambda_builders.workflows.go_dep
 import aws_lambda_builders.workflows.go_modules
+import aws_lambda_builders.workflows.java_gradle

aws_lambda_builders/workflows/java_gradle/DESIGN.md

@@ -0,0 +1,195 @@
+# Java - Gradle Lambda Builder
+
+## Scope
+
+This package enables the creation of a Lambda deployment package for Java
+projects managed using the Gradle build tool.
+
+For Java projects, the most popular way to create a distribution package for
+Java based Lambdas is to create an "uber" or "fat" JAR. This is a single JAR
+file that contains both the customers' classes and resources, as well as all the
+classes and resources extracted from their dependency JAR's. However, this can
+cause files that have the same path in two different JAR's to collide within the
+uber JAR.
+
+Another solution is to create a distribution ZIP containing the customer's
+classes and resources and include their dependency JARs under a `lib` directory.
+This keeps the customers' classes and resources separate from their
+dependencies' to avoid any file collisions. However, this incurs some overhead
+as the ZIP must be unpacked before the code can run.
+
+To avoid the problem of colliding files, we will choose the second option and
+create distribution ZIP.
+
+## Challenges
+
+Java bytecode can only run on the same or newer version of the JVM for which
+it was compiled for. For example Java 8 bytecode can run a JVM that is at
+least version 8, but bytecode targetting Java 9 cannot run on a Java 8 VM.
+This is further complicated by the fact that a newer JDK can generate code to
+be run on an older VM if configured using the `targetCompatibility` and
+`sourceCompatibility` properties of the Java plugin. Therefore, it is not
+sufficient to check the version of the local JDK, nor is it possible to check
+the value set for `targetCompatibility` or `sourceCompatibility` since it can
+be local to the compile/build task. At best, we can check if the local
+version of the JDK is newer than Java 8 and emit a warning that the built
+artifact may not run in Lambda.
+
+Gradle projects are configured using `build.gradle` build scripts. These are
+executable files authored in either Groovy or since 5.0, Kotlin, and using the
+Gradle DSL. This presents a similar problem to `setup.py` in the Python world in
+that arbitrary logic can be executed during build time that could affect both
+how the customer's artifact is built, and which dependencies are chosen.
+
+An interesting challenge is dealing with single build and multi build projects.
+Consider the following different projects structures:
+
+**Project A**
+```
+ProjectA
+├── build.gradle
+├── gradlew
+├── src
+└── template.yaml
+```
+
+**Project B**
+```
+ProjectB
+├── common
+│   └── build.gradle
+├── lambda1
+│   └── build.gradle
+├── lambda2
+│   └── build.gradle
+├── build.gradle
+├── gradlew
+├── settings.gradle
+└── template.yaml
+```
+
+Here `ProjectA` is a a single lambda function, and `ProjectB` is a multi-build
+project where sub directories `lambda1` and `lambda2` are each a lambda
+function. In addition, suppose that `ProjectB/lambda1` has a dependency on its
+sibling project `ProjectB/common`.
+
+Building Project A is relatively simple since we just need to issue `gradlew
+build` and place the built ZIP within the artifact directory.
+
+Building `ProjectB/lambda1` is very similar from the point of view of the
+workflow since it still issues the same command (`gradlew build`), but it
+requires that Gradle is able to find its way back up to the parent `ProjectB` so
+that it can also build `ProjectB/common` which can be a challenge when mounting
+within a container.
+
+## Implementation
+
+### Build Workflow
+
+We leverage Gradle to do all the heavy lifting for executing the
+`build.gradle` script which will resolve and download the dependencies and
+build the project. To create the distribution ZIP, we use the help of a
+Gradle init script to insert a post-build action to do this.
+
+#### Step 1: Copy custom init file to temporary location
+
+There is no standard task in Gradle to create a distribution ZIP (or uber JAR).
+We add this functionality through the use of a Gradle init script. The script
+will be responsible for adding a post-build action that creates the distribution
+ZIP.
+
+It will do something similar to:
+
+```sh
+cp /path/to/lambda-build-init.gradle /$SCRATCH_DIR/
+```
+
+where the contents of `lambda-build-init.gradle` contains the code for defining
+the post-build action:
+
+```gradle
+gradle.project.afterProject { p ->
+  // Set the give project's buildDir to one under SCRATCH_DIR
+}
+
+// Include the project classes and resources in the root, and the dependencies
+// under lib
+gradle.taskGraph.afterTask { t ->
+    if (t.name != 'build') {
+        return
+    }
+
+    // Step 1: Find the directory under scratch_dir where the artifact for
+    // t.project is located
+    // Step 2: Open ZIP file in $buildDir/distributions/lambda_build
+    // Step 3: Copy project class files and resources to ZIP root
+    // Step 3: Copy libs in configurations.runtimeClasspath into 'lib'
+    // subdirectory in ZIP
+}
+```
+
+#### Step 2: Resolve Gradle executable to use
+
+[The recommended
+way](https://docs.gradle.org/current/userguide/gradle_wrapper.html)  way to
+author and distribute a Gradle project is to include a `gradlew` or Gradle
+Wrapper file within the root of the project. This essentially locks in the
+version of Gradle for the project and uses an executable that is independent of
+any local installations. This helps ensure that builds are always consistent
+over different environments.
+
+The `gradlew` script, if it is included, will be located at the root of the
+project. We will rely on the invoker of the workflow to supply the path to the
+`gradlew` script.
+
+We give precedence to this `gradlew` file, and if isn't found, we use the
+`gradle` executable found on the `PATH` using the [path resolver][path resolver].
+
+#### Step 3: Check Java version and emit warning
+
+Check whether the local JDK version is <= Java 8, and if it is not, emit a
+warning that the built artifact may not run in Lambda unless a) the project is
+properly configured (i.e. using `targetCompatibility`) or b) the project is
+built within a Lambda-compatibile environment like `lambci`.
+
+We use the Gradle executable from Step 2 for this to ensure that we check the
+actual JVM version Gradle is using in case it has been configured to use a
+different one than can be found on the PATH.
+
+#### Step 4: Build and package
+
+```sh
+$GRADLE_EXECUTABLE --project-cache-dir $SCRATCH_DIR/gradle-cache \
+    -Dsoftware.amazon.aws.lambdabuilders.scratch-dir=$SCRATCH_DIR \
+    --init-script $SCRATCH_DIR/lambda-build-init.gradle build
+```
+
+Since by default, Gradle stores its build-related metadata in a `.gradle`
+directory under the source directory, we specify an alternative directory under
+`scratch_dir` to avoid writing anything under `source_dir`. This is simply a
+`gradle-cache` directory under `scratch_dir`.
+
+Next, we also pass the location of the `scratch_dir` as a Java system
+property so that it's availabe to our init script. This allows it to correctly
+map the build directory for each sub-project within `scratch_dir`. Again, this
+ensures that we are not writing anything under the source directory.
+
+One important detail here is that the init script may create *multiple*
+subdirectories under `scratch_dir`, one for each project involved in building
+the lambda located at `source_dir`. Going back to the `ProjectB` example, if
+we're building `lambda1`, this also has the effect of building `common` because
+it's a declared dependency in its `build.gradle`. So, within `scratch_dir` will
+be a sub directory for each project that gets built as a result of building
+`source_dir`; in this case there will be one for each of `lambda1` and `common`.
+The init file uses some way of mapping the source root of each project involved
+to a unique directory under `scratch_dir`, like a hashing function.
+
+#### Step 5: Copy to artifact directory
+
+The workflow implementation is aware of the mapping scheme used to map a
+`source_dir` to the correct directory under `scratch_dir` (described in step 4),
+so it knows where to find the built Lambda artifact when copying it to
+`artifacts_dir`. They will be located in
+`$SCRATCH_DIR/<mapping for source_dir>/build/distributions/lambda-build`.
+
+[path resolver]: https://github.com/awslabs/aws-lambda-builders/pull/55

aws_lambda_builders/workflows/java_gradle/__init__.py

@@ -0,0 +1,5 @@
+"""
+Builds Java Lambda functions using the Gradle build tool
+"""
+
+from .workflow import JavaGradleWorkflow

aws_lambda_builders/workflows/java_gradle/actions.py

@@ -0,0 +1,84 @@
+"""
+Actions for the Java Gradle Workflow
+"""
+
+import os
+from aws_lambda_builders.actions import ActionFailedError, BaseAction, Purpose
+from .gradle import GradleExecutionError
+
+
+class JavaGradleBuildAction(BaseAction):
+    NAME = "GradleBuild"
+    DESCRIPTION = "Building the project using Gradle"
+    PURPOSE = Purpose.COMPILE_SOURCE
+
+    INIT_SCRIPT = 'lambda-build-init.gradle'
+    SCRATCH_DIR_PROPERTY = 'software.amazon.aws.lambdabuilders.scratch-dir'
+    GRADLE_CACHE_DIR_NAME = 'gradle-cache'
+
+    def __init__(self,
+                 source_dir,
+                 build_file,
+                 subprocess_gradle,
+                 scratch_dir,
+                 os_utils):
+        self.source_dir = source_dir
+        self.build_file = build_file
+        self.scratch_dir = scratch_dir
+        self.subprocess_gradle = subprocess_gradle
+        self.os_utils = os_utils
+        self.cache_dir = os.path.join(self.scratch_dir, self.GRADLE_CACHE_DIR_NAME)
+
+    def execute(self):
+        init_script_file = self._copy_init_script()
+        self._build_project(init_script_file)
+
+    @property
+    def gradle_cache_dir(self):
+        return self.cache_dir
+
+    def _copy_init_script(self):
+        try:
+            src = os.path.join(os.path.dirname(__file__), 'resources', self.INIT_SCRIPT)
+            dst = os.path.join(self.scratch_dir, self.INIT_SCRIPT)
+            return self.os_utils.copy(src, dst)
+        except Exception as ex:
+            raise ActionFailedError(str(ex))
+
+    def _build_project(self, init_script_file):
+        try:
+            if not self.os_utils.exists(self.scratch_dir):
+                self.os_utils.makedirs(self.scratch_dir)
+            self.subprocess_gradle.build(self.source_dir, self.build_file, self.gradle_cache_dir,
+                                         init_script_file,
+                                         {self.SCRATCH_DIR_PROPERTY: os.path.abspath(self.scratch_dir)})
+        except GradleExecutionError as ex:
+            raise ActionFailedError(str(ex))
+
+
+class JavaGradleCopyArtifactsAction(BaseAction):
+    NAME = "CopyArtifacts"
+    DESCRIPTION = "Copying the built artifacts"
+    PURPOSE = Purpose.COPY_SOURCE
+
+    def __init__(self,
+                 source_dir,
+                 artifacts_dir,
+                 build_dir,
+                 os_utils):
+        self.source_dir = source_dir
+        self.artifacts_dir = artifacts_dir
+        self.build_dir = build_dir
+        self.os_utils = os_utils
+
+    def execute(self):
+        self._copy_artifacts()
+
+    def _copy_artifacts(self):
+        lambda_build_output = os.path.join(self.build_dir, 'build', 'distributions', 'lambda-build')
+        try:
+            if not self.os_utils.exists(self.artifacts_dir):
+                self.os_utils.makedirs(self.artifacts_dir)
+            self.os_utils.copytree(lambda_build_output, self.artifacts_dir)
+        except Exception as ex:
+            raise ActionFailedError(str(ex))

aws_lambda_builders/workflows/java_gradle/gradle.py

@@ -0,0 +1,53 @@
+"""
+Wrapper around calls to Gradle through a subprocess.
+"""
+
+import logging
+import subprocess
+
+LOG = logging.getLogger(__name__)
+
+
+class GradleExecutionError(Exception):
+    MESSAGE = "Gradle Failed: {message}"
+
+    def __init__(self, **kwargs):
+        Exception.__init__(self, self.MESSAGE.format(**kwargs))
+
+
+class BuildFileNotFoundError(GradleExecutionError):
+    def __init__(self, build_file_path):
+        super(BuildFileNotFoundError, self).__init__(
+            message='Gradle build file not found: %s' % build_file_path)
+
+
+class SubprocessGradle(object):
+
+    def __init__(self, gradle_binary, os_utils=None):
+        if gradle_binary is None:
+            raise ValueError("Must provide Gradle BinaryPath")
+        self.gradle_binary = gradle_binary
+        if os_utils is None:
+            raise ValueError("Must provide OSUtils")
+        self.os_utils = os_utils
+
+    def build(self, source_dir, build_file, cache_dir=None, init_script_path=None, properties=None):
+        if not self.os_utils.exists(build_file):
+            raise BuildFileNotFoundError(build_file)
+
+        args = ['build', '--build-file', build_file]
+        if cache_dir is not None:
+            args.extend(['--project-cache-dir', cache_dir])
+        if properties is not None:
+            args.extend(['-D%s=%s' % (n, v) for n, v in properties.items()])
+        if init_script_path is not None:
+            args.extend(['--init-script', init_script_path])
+        ret_code, _, stderr = self._run(args, source_dir)
+        if ret_code != 0:
+            raise GradleExecutionError(message=stderr.decode('utf8').strip())
+
+    def _run(self, args, cwd=None):
+        p = self.os_utils.popen([self.gradle_binary.binary_path] + args, cwd=cwd, stdout=subprocess.PIPE,
+                                stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate()
+        return p.returncode, stdout, stderr

aws_lambda_builders/workflows/java_gradle/gradle_resolver.py

@@ -0,0 +1,31 @@
+"""
+Gradle executable resolution
+"""
+
+from .utils import OSUtils
+
+
+class GradleResolver(object):
+
+    def __init__(self, executable_search_paths=None, os_utils=None):
+        self.binary = 'gradle'
+        self.executables = [self.binary]
+        self.executable_search_paths = executable_search_paths
+        self.os_utils = os_utils if os_utils else OSUtils()
+
+    @property
+    def exec_paths(self):
+        # Prefer gradlew/gradlew.bat
+        paths = self.os_utils.which(self.wrapper_name, executable_search_paths=self.executable_search_paths)
+        if not paths:
+            # fallback to the gradle binary
+            paths = self.os_utils.which('gradle', executable_search_paths=self.executable_search_paths)
+
+        if not paths:
+            raise ValueError("No Gradle executable found!")
+
+        return paths
+
+    @property
+    def wrapper_name(self):
+        return 'gradlew.bat' if self.os_utils.is_windows() else 'gradlew'