Merge pull request #25 from lesserwhirls/java

Various Improvements

Author: lesserwhirls

README.md

@@ -36,7 +36,7 @@ packer build --only=<type> thredds-test-env.pkr.hcl
 ~~~
 
 `<type>` will one or more (separated by commas) of the following:
-* `docker.docker-jenkins`: Provision a Docker container and generate and tag a local Docker image (`docker.unidata.ucar.edu/thredds-test-environment:latest`).
+* `docker.docker-jenkins`: Provision a Docker container and generate and tag a local Docker image (`docker.unidata.ucar.edu/thredds-test-environment:20.04`).
 * `docker.docker-github-action`: Provision a Docker container for use with GitHub Actions and tag a local docker image (`ghcr.io/unidata/thredds-test-action:v3`).
 * `docker.docker-export`: Provision a Docker container and generate a local Docker image as a file (`image.tar`).
 
@@ -46,24 +46,24 @@ Typically, we would run the following to update the Jenkins and Github Action Do
 packer build --only=docker.docker-jenkins,docker.docker-github-action thredds-test-env.pkr.hcl
 ~~~
 
-The Docker image builds takes about 1 hour to create.
+The Docker image builds takes about 30 minutes to create on MacOS.
 Packer will run the builders in parallel, so the total time to create the `thredds-test-environment` images is around an hour.
 
 If using `docker-jenkins`, then once the image is built you can test out the environment by using:
 
 ~~~bash
-docker run -i -t --rm docker.unidata.ucar.edu/thredds-test-environment:latest
+docker run -i -t --rm docker.unidata.ucar.edu/thredds-test-environment:24.04
 ~~~
 
-These images are not produced using the multiplatform feature of Docker.
-If you are on a arm64 based mac, you will need to run:
+These images are built using the multiplatform feature of Docker, and only `linux/amd64` images are produced.
+If you are on a `arm64` based mac, you will need to run:
 
 ~~~bash
-DOCKER_DEFAULT_PLATFORM=linux/amd64 docker run -i -t --rm docker.unidata.ucar.edu/thredds-test-environment:latest
+DOCKER_DEFAULT_PLATFORM=linux/amd64 docker run -i -t --rm docker.unidata.ucar.edu/thredds-test-environment:24.04
 ~~~
 
 Note that images are not pushed as part of this build process.
-Pushes can be done via the normal docker mechanisms, e.g. `docker image push docker.unidata.ucar.edu/thredds-test-environment:latest` and `docker image push ghcr.io/unidata/thredds-test-action:v3`.
+Pushes can be done via the normal docker mechanisms, e.g. `docker image push docker.unidata.ucar.edu/thredds-test-environment:24.04` and `docker image push ghcr.io/unidata/thredds-test-action:v3`.
 
 ## Project layout
 
@@ -92,12 +92,11 @@ The `ansible/` directory is laid out as follows:
 We use the following roles when provisioning our images:
 
 * `cleanup`: General cleanup related tasks, such as remove the temporary build directory and running `ldconfig`
+* `corretto`: Obtain and install LTS versions of Corretto.
 * `general-packages`: Install general packages needed for the build environment using the OS package manager.
-* `gradle-builds-cache-bootstrap`: Pull in and build netCDF-Java to populate the gradle cache for user ubuntu.
 * `init`: Initialize the build environment by ensuring the temporary ansible build directory exists.
 * `libnetcdf-and-deps`: Configure, build, and install `zlib`, `HDF5`, and `netCDF-C`.
 * `maven`: Obtain and install the Apache Maven software project management and comprehension tool. 
-* `miniconda`: Obtain and install the Anaconda miniconda python distribution.
 * `security`:
   * Add the `ubuntu` user.
   * Add a default `maven-settings.xml` file configured to publish to the Unidata artifacts server.
@@ -107,8 +106,6 @@ We use the following roles when provisioning our images:
 * `test-data-mount-prep`: Prepare the environment to mount the `thredds-test-data` datasets when available (currently used on Jenkins worker nodes).
 * `zulu`: Obtain and install LTS versions of Zulu.
 
-We also use a role from [Ansible Galaxy](https://galaxy.ansible.com/) to setup a Ruby environment ([geerlingguy.ruby](https://galaxy.ansible.com/geerlingguy/ruby)).
-
 ## THREDDS Test Environment Highlights
 
 ### netCDF-C
@@ -118,10 +115,6 @@ We also use a role from [Ansible Galaxy](https://galaxy.ansible.com/) to setup a
    * zlib version: `1.2.11`
    * hdf5 version: `1.12.1`
 
-### miniconda
- * location: `/usr/thredds-test-environment/miniconda3`
- * version: `Miniconda3-latest-Linux-x86_64`
-
 ### maven:
  * location: `/usr/thredds-test-environment/mvn`
  * version: `3.6.3`
@@ -139,12 +132,14 @@ We also use a role from [Ansible Galaxy](https://galaxy.ansible.com/) to setup a
    * 17 (`/usr/thredds-test-environment/zulu17`)
    * 21 (`/usr/thredds-test-environment/zulu21`)
 
-### Ruby
-  * ruby (via [geerlingguy.ruby](https://galaxy.ansible.com/geerlingguy/ruby) from [Ansible Galaxy](https://galaxy.ansible.com/))
+ * Corretto (latest version available from docs.aws.amazon.com/corretto/)
+   * 8 (`/usr/thredds-test-environment/corretto8`)
+   * 11 (`/usr/thredds-test-environment/corretto11`)
+   * 17 (`/usr/thredds-test-environment/corretto17`)
+   * 21 (`/usr/thredds-test-environment/corretto21`)
 
 ### Bash functions:
- * `select-java <vendor> <version>` (where version is 8, 11, 17, or 21, and vendor is `temurin` or `zulu`)
- * `activate-conda`
+ * `select-java <vendor> <version>` (where version is 8, 11, 17, or 21, and vendor is `temurin`, `zulu`, or `corretto`)
  * `get_pw <key>`
 
 ### Latest version available via the OS Package Manager
@@ -160,26 +155,26 @@ We also use a role from [Ansible Galaxy](https://galaxy.ansible.com/) to setup a
 ### Docker Image
 
 ~~~
-    docker.docker-jenkins: Friday 22 November 2024  21:12:32 +0000 (0:00:02.871)       0:20:20.107 *******
+    docker.docker-jenkins: Tuesday 25 February 2025  15:28:53 +0000 (0:00:02.993)       0:25:00.302 ******
     docker.docker-jenkins: ===============================================================================
-    docker.docker-jenkins: Wait for the HDF5 async test task to complete. ------------------------ 362.21s
-    docker.docker-jenkins: libnetcdf-and-deps : Install hdf5. ------------------------------------ 272.84s
-    docker.docker-jenkins: libnetcdf-and-deps : Configure netCDF-c. ------------------------------ 128.28s
-    docker.docker-jenkins: libnetcdf-and-deps : Configure hdf5. ---------------------------------- 114.44s
-    docker.docker-jenkins: zulu : Fetch latest Zulu Java builds. ---------------------------------- 54.52s
-    docker.docker-jenkins: temurin : Fetch latest Temurin Java builds. ---------------------------- 46.20s
-    docker.docker-jenkins: zulu : Unpack Zulu Java Installations. --------------------------------- 35.03s
-    docker.docker-jenkins: libnetcdf-and-deps : Install netCDF-c. --------------------------------- 28.84s
-    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 28.32s
-    docker.docker-jenkins: temurin : Unpack Temurin Java Installations. --------------------------- 26.71s
-    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 24.31s
-    docker.docker-jenkins: geerlingguy.ruby : Install ruby and other required dependencies. ------- 12.96s
-    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 12.77s
-    docker.docker-jenkins: miniconda : Download and unpack miniconda. ------------------------------ 9.31s
-    docker.docker-jenkins: libnetcdf-and-deps : Install zlib. -------------------------------------- 5.46s
-    docker.docker-jenkins: libnetcdf-and-deps : Configure zlib. ------------------------------------ 4.13s
-    docker.docker-jenkins: security : Update SSH configuration to be more secure. ------------------ 3.78s
-    docker.docker-jenkins: libnetcdf-and-deps : Download and unpack hdf5. -------------------------- 3.59s
-    docker.docker-jenkins: zulu : Read versions of installed Zulu. --------------------------------- 3.34s
-    docker.docker-jenkins: cleanup : Remove packages that are not needed in final environment. ----- 3.07s
+    docker.docker-jenkins: Wait for the HDF5 async test task to complete. ------------------------ 362.87s
+    docker.docker-jenkins: libnetcdf-and-deps : Install hdf5. ------------------------------------ 328.13s
+    docker.docker-jenkins: libnetcdf-and-deps : Configure netCDF-c. ------------------------------ 180.21s
+    docker.docker-jenkins: libnetcdf-and-deps : Configure hdf5. ---------------------------------- 135.15s
+    docker.docker-jenkins: temurin : Fetch latest Temurin Java builds. ---------------------------- 83.71s
+    docker.docker-jenkins: zulu : Fetch latest Zulu Java builds. ---------------------------------- 77.12s
+    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 41.00s
+    docker.docker-jenkins: corretto : Fetch latest Corretto Java builds. -------------------------- 40.41s
+    docker.docker-jenkins: zulu : Unpack Zulu Java Installations. --------------------------------- 37.98s
+    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 36.87s
+    docker.docker-jenkins: libnetcdf-and-deps : Install netCDF-c. --------------------------------- 34.90s
+    docker.docker-jenkins: corretto : Unpack Corretto Java Installations. ------------------------- 28.92s
+    docker.docker-jenkins: temurin : Unpack Temurin Java Installations. --------------------------- 28.57s
+    docker.docker-jenkins: general-packages : Install os managed tools. --------------------------- 15.22s
+    docker.docker-jenkins: libnetcdf-and-deps : Install zlib. -------------------------------------- 6.27s
+    docker.docker-jenkins: security : Update SSH configuration to be more secure. ------------------ 5.02s
+    docker.docker-jenkins: libnetcdf-and-deps : Configure zlib. ------------------------------------ 4.52s
+    docker.docker-jenkins: zulu : Read versions of installed Zulu. --------------------------------- 3.74s
+    docker.docker-jenkins: libnetcdf-and-deps : Download and unpack hdf5. -------------------------- 3.65s
+    docker.docker-jenkins: cleanup : Remove packages that are not needed in final environment. ----- 3.54s
 ~~~

packer/provisioners/ansible/group_vars/all.yml

@@ -3,10 +3,6 @@
 install_dir: /usr/thredds-test-environment
 tmp_dir: /tmp/thredds-ansible-tmp
 
-# Shared between gradle-builds-cache-bootstrap and cleanup roles.
-tmp_repo_dir: "{{ tmp_dir }}/git_repositories"
-gradle_user_home: "/home/{{ thredds_test_user }}/.gradle"
-
 # The variable thredds_test_user is defined in the packer configuration
 # located at packer/thredds-test-env.json.
 
@@ -29,18 +25,3 @@ hdf5_async_check_retries: 80
 netcdf_c_async_timeout: 1800 # seconds (30 minutes)
 netcdf_c_async_check_interval: 120 # seconds
 netcdf_c_async_check_retries: 30
-
-miniconda_async_timeout: 1800 # seconds (30 minutes)
-miniconda_async_check_interval: 60 # seconds
-miniconda_async_check_retries: 30
-
-git_clone_async_timeout: 600 # seconds (10 minutes)
-git_clone_async_check_interval: 20 # seconds
-git_clone_async_check_retries: 30
-
-bootstrap_gradle_cache_async_timeout: 1800 # seconds (30 minutes)
-bootstrap_gradle_cache_async_check_interval: 60 # seconds
-bootstrap_gradle_cache_async_check_retries: 30
-
-# geerlingguy.ruby role variables
-ruby_install_bundler: false

packer/provisioners/ansible/requirements.yml

@@ -1,21 +1,4 @@
 ---
-- name: "Cleanup temporary git repository directory."
-  file:
-    path: "{{ tmp_repo_dir }}"
-    state: absent
-
-- name: Check if gradle cache directory exists.
-  stat:
-    path: "{{ gradle_user_home }}"
-  register: gradle_user_home_check
-
-- name: "Change owner of gradle cache directory to be {{ thredds_test_user }}."
-  when: gradle_user_home_check.stat.exists == true
-  file:
-    path: "{{ gradle_user_home }}"
-    owner: "{{ thredds_test_user }}"
-    recurse: yes
-
 - name: Prepare to add thredds-test-environment native libraries to run-time linker
   copy:
     dest: "/etc/ld.so.conf.d/tte.conf"

packer/provisioners/ansible/roles/cleanup/tasks/main.yml

@@ -0,0 +1,18 @@
+---
+- name: Fetch latest Corretto Java builds.
+  uri:
+    url: "https://corretto.aws/downloads/latest/amazon-corretto-{{ item }}-x64-linux-jdk.tar.gz"
+    method: GET
+    follow_redirects: safe
+    dest: "/tmp/{{ base_install_name }}{{ item }}.tar.gz"
+    # Expect 200 locally (i.e. Docker builder), 304 (for some versions) on AWS EC2.
+    status_code: [200, 304]
+  register: uri_result
+  retries: "{{ network_call_retries }}"
+  delay: "{{ network_call_time_between_retries }}"
+  until: ("status" in uri_result) and (uri_result.status == 200 or uri_result.status == 304)
+  loop: "{{ java_versions }}"
+
+- debug:
+    var: uri_result
+    verbosity: 2

packer/provisioners/ansible/roles/corretto/tasks/fetch_remote.yml

@@ -0,0 +1,36 @@
+---
+- name: Create installation directories, if they do not exist.
+  file:
+    path: "{{ install_dir }}/{{ base_install_name }}{{ item }}"
+    state: directory
+  loop: "{{ java_versions }}"
+  loop_control:
+    label: "Create Directory for Corretto {{ item }} Installation"
+
+- import_tasks: fetch_remote.yml
+  tags: [ fetch-remote ]
+
+- name: Unpack Corretto Java Installations.
+  unarchive:
+    src: "/tmp/{{ base_install_name }}{{ item }}.tar.gz"
+    dest: "{{ install_dir }}/{{ base_install_name }}{{ item }}"
+    extra_opts: [--strip-components=1]
+    creates: "{{ install_dir }}/{{ base_install_name }}{{ item }}/bin"
+    remote_src: yes
+  loop: "{{ java_versions }}"
+  loop_control:
+    label: "Unpack Corretto {{ item }}"
+
+- name: Read versions of installed Corretto.
+  shell: "{{ install_dir }}/{{ base_install_name }}{{ item }}/bin/java -version"
+  register: shell_output
+  loop: "{{ java_versions }}"
+  loop_control:
+    label: "Read Corretto {{ item }} Version Information"
+
+- name: Print versions of installed Corretto Java instances.
+  debug:
+    msg="{{ item.stderr_lines[1] }}"
+  loop: "{{ shell_output.results }}"
+  loop_control:
+    label: "Corretto {{ item.item }} Version:"

packer/provisioners/ansible/roles/corretto/tasks/main.yml

@@ -0,0 +1,8 @@
+---
+base_install_name: corretto
+
+java_versions:
+  - 8
+  - 11
+  - 17
+  - 21