[CTM-142] New terra base image with sudo (#511)

* [CTM-142] New terra base image with sudo

* move to old file structure

* add to publishing steps

* tweaks

* Update comment

* make version 1.0.0

Author: lucymcnatt

.github/workflows/publish.yml

@@ -7,6 +7,7 @@ on:
         required: true
         type: choice
         options:
+          - terra-base
           - terra-jupyter-aou
           - terra-jupyter-base
           - terra-jupyter-bioconductor

README.md

@@ -5,7 +5,7 @@ This repo provides docker images for running jupyter notebook in [Terra](https:/
 Make sure to go through the [contributing guide](https://github.com/DataBiosphere/terra-docker/blob/master/CONTRIBUTING.md#contributing) as you make changes to this repo.
 
 # Terra Base Images
-[terra-jupyter-base](terra-jupyter-base/README.md)
+[terra-base](terra-base/README.md)
 
 [terra-jupyter-python](terra-jupyter-python/README.md)
 
@@ -20,15 +20,18 @@ Make sure to go through the [contributing guide](https://github.com/DataBiospher
 # How to create your own Custom image to use with notebooks on Terra
 Custom docker images need to use a Terra base image (see above) in order to work with the service that runs notebooks on Terra.
 * You can use any of the base images above
-* Here is an example of how to build off of a base image: Add `FROM us.gcr.io/broad-dsp-gcr-public/terra-jupyter-base:0.0.1` to your dockerfile (`terra-jupyter-base` is the smallest image you can extend from)
+  * `terra-base` is the smallest image, but doesn't include any scientific packages on top of Jupyter and R
+* Here is an example of how to build off of a base image: Add `FROM us.gcr.io/broad-dsp-gcr-public/terra-base:0.0.1` to your dockerfile
 * Customize your image (see the [terra-jupyter-python](terra-jupyter-python/Dockerfile) dockerfile for an example of how to extend from one of our base images
-* Publish the image to either GCR or Dockerhub; the image must be public to be used
+* Publish the image to either GAR or Dockerhub;
+  * If using Dockerhub, the image **must be public** to be used
 * Use the published container image location when creating notebook runtime
 * Dockerhub image example: [image name]:[tag]
-* GCR image example: us.gcr.io/repository/[image name]:[tag]
-* Since 6/28/2021, we introduced a few changes that might impact building custom images
-    - Home directory of new images will be `/home/jupyter`. This means if your dockerfile is referencing `/home/jupyter-user` directory, you need to update it to $HOME (recommended) or `/home/jupyter`.
-    - Creating VMs with custom images will take much longer than terra supported images because `docker pull` will take a few min. If the custom image ends up being too large, VM creation may time out. New base images are much larger in size than previous versions.
+* GAR image example: us.gcr.io/repository/[image name]:[tag]
+* Some things to keep in mind when creating custom images:
+  - The home directory of new images will be `/home/jupyter`. This means if your dockerfile is referencing the `/home/jupyter-user` directory, you need to update it to $HOME (recommended) or `/home/jupyter`.
+  - Creating VMs with custom images may take longer than terra supported images because `docker pull` will take a few min. If the custom image ends up being too large, VM creation may time out.
+-
 
 # Development
 ## Using git secrets
@@ -56,25 +59,25 @@ Once you have the container running, you should be able to access jupyter at htt
 Detailed documentation on how to integrate the terra-docker image with Leonardo can be found [here](https://broadworkbench.atlassian.net/wiki/spaces/IA/pages/2519564289/Integrating+new+Terra+docker+images+with+Leonardo)
 
 ### If you are adding a new image:
-- Create a new directory with the Dockerfile and a CHANGELOG.md. 
+- Create a new directory with the Dockerfile and a CHANGELOG.md.
 - Add the directory name (also referred to as the image name) as an entry to the image_data array in the file in config/conf.json. For more info on what is needed for a new image, see the section on the config
 - If you wish the image to be baked into our custom image, which makes the runtime load significantly faster (recommended), make a PR into the leonardo [repo](https://github.com/DataBiosphere/leonardo) doing the following within the `jenkins` folder:
-    - Add the image to the parameter list in the Jenkinsfile
-    - Update the relevant `prepare` script in each subdirectory. Currently there is a prepare script for gce and dataproc.
-    - It is recommended to add a test in the `automation` directory (`automation/src/test/resources/reference.conf`)
-    - Add your image to the `reference.conf` in the automation directory. This will be the only place any future version updates to your image happen. This ensures, along with the test in the previous step, that any changes to the image are tested.
-    - Run the GHA to generate the image, and add it to `reference.conf` in the http directory (`http/src/main/resources/reference.conf`)
+  - Add the image to the parameter list in the Jenkinsfile
+  - Update the relevant `prepare` script in each subdirectory. Currently there is a prepare script for gce and dataproc.
+  - It is recommended to add a test in the `automation` directory (`automation/src/test/resources/reference.conf`)
+  - Add your image to the `reference.conf` in the automation directory. This will be the only place any future version updates to your image happen. This ensures, along with the test in the previous step, that any changes to the image are tested.
+  - Run the GHA to generate the image, and add it to `reference.conf` in the http directory (`http/src/main/resources/reference.conf`)
 
 ### If you are updating an existing image:
 - [Create your terra-docker PR](https://broadworkbench.atlassian.net/wiki/spaces/IA/pages/2519564289/Integrating+new+Terra+docker+images+with+Leonardo#1.-Create-a-terra-docker-PR)
-    - Update the version in config/conf.json
-    - Update CHANGELOG.md and VERSION file
-    - Ensure that no `From` statements need to be updated based on the image you updated (i.e., if you update the base image, you will need to update several other images)
-    - Run updateVersions.sc to bump all images dependent on the base
+  - Update the version in config/conf.json
+  - Update CHANGELOG.md and VERSION file
+  - Ensure that no `From` statements need to be updated based on the image you updated (i.e., if you update the base image, you will need to update several other images)
+  - Run updateVersions.sc to bump all images dependent on the base
 - [Merge your terra-docker PR and check if the image(s) and version json files are created](https://broadworkbench.atlassian.net/wiki/spaces/IA/pages/2519564289/Integrating+new+Terra+docker+images+with+Leonardo#2.-Merge-your-terra-docker-PR-and-check-images-are-created)
 - [Open a PR in leonardo](https://broadworkbench.atlassian.net/wiki/spaces/IA/pages/2519564289/Integrating+new+Terra+docker+images+with+Leonardo#3.-Create-a-new-leo-PR-that-integrates-the-new-images)
-    - Update the relevant `prepare` script within the `jenkins` folder
-    - Update the automation `reference.conf` file
+  - Update the relevant `prepare` script within the `jenkins` folder
+  - Update the automation `reference.conf` file
 - [Run the GHA on your branch to generate the new image](https://broadworkbench.atlassian.net/wiki/spaces/IA/pages/2519564289/Integrating+new+Terra+docker+images+with+Leonardo#4.-Run-the-Github-Action-in-leo-to-generate-a-new-custom-COS-image)
 - [Update the leonardo PR to use the newly generated image](https://broadworkbench.atlassian.net/wiki/spaces/IA/pages/2519564289/Integrating+new+Terra+docker+images+with+Leonardo#5.-Update-the-Leo-PR-to-use-the-generated-OS-images)
 - Ensure that the `terra-docker-versions-candidate.json` file (which is what the UI sources the dropdown from) in the `terra-docjker-image-documentation-[env]` bucket correclty references your new docker image
@@ -85,11 +88,11 @@ Detailed documentation on how to integrate the terra-docker image with Leonardo
 Build the image:
 run `docker build [your_dir] -t [name]`.
 
-`docker build terra-jupyter-base -t terra-jupyter-base`
+`docker build terra-base -t terra-base`
 
 If you're on an M1 and building an image from a locally built image, replace the current FROM command:
 
-`FROM --platform=linux/amd64 terra-jupyter-base`
+`FROM --platform=linux/amd64 terra-base`
 
 Apple Silicon chips later then M1 (ex. M3) need:
 
@@ -120,9 +123,9 @@ To launch an image through Terra, navigate to https://app.terra.bio or your BEE'
 
 ## Config
 
-There is a config file located at `config/conf.json` that contains the configuration used by all automated jobs and build scripts that interface with this repo. 
+There is a config file located at `config/conf.json` that contains the configuration used by all automated jobs and build scripts that interface with this repo.
 
-There is a field for "spark_version" top-level which must be updated if we update the debian version used in the custom image. 
+There is a field for "spark_version" top-level which must be updated if we update the debian version used in the custom image.
 Currently it assumes 1.4x https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-1.4
 
 There are some constants included, such as the tools supported by this repo. Of particular interest is the image_data array.
@@ -163,7 +166,7 @@ Each time you update or add an image, you will need to update the appropriate en
 
 The scripts folder has scripts used for building.
 - `generate_package_docs.py` This script is run once by build.sh each time an image is built. It is used to generate a .json with the versions for the packages in the image.
-- `generate_version_docs.py` This script is run each time an image is built. It builds a new file master version file for the UI to look up the current versions to reference. 
+- `generate_version_docs.py` This script is run each time an image is built. It builds a new file master version file for the UI to look up the current versions to reference.
 
 ## Image dependencies
 

build_all.sh

@@ -3,16 +3,17 @@
 # Example: ./build_all.sh
 
 # Create the ordered list of images to build
-# 1- terra-jupyter-base
-# 2- terra-jupyter-python
-# 3- terra-jupyter-r
-# 4- terra-jupyter-gatk
-# 5- terra-jupyter-hail
-# 6- terra-jupyter-aou
-# 7- terra-jupyter-bioconductor
-# 8- terra-rstudio-aou
-# 9- wondershaper
-images=("terra-jupyter-base" "terra-jupyter-python" "terra-jupyter-r" "terra-jupyter-gatk" "terra-jupyter-hail" "terra-jupyter-aou" "terra-jupyter-bioconductor" "terra-rstudio-aou" "wondershaper")
+# 1- terra-base
+# 2- terra-jupyter-base
+# 3- terra-jupyter-python
+# 4- terra-jupyter-r
+# 5- terra-jupyter-gatk
+# 6- terra-jupyter-hail
+# 7- terra-jupyter-aou
+# 8- terra-jupyter-bioconductor
+# 9- terra-rstudio-aou
+# 10- wondershaper
+images=("terra-base", "terra-jupyter-base" "terra-jupyter-python" "terra-jupyter-r" "terra-jupyter-gatk" "terra-jupyter-hail" "terra-jupyter-aou" "terra-jupyter-bioconductor" "terra-rstudio-aou" "wondershaper")
 
 # Loop over each image to build in the correct order
 for image in "${images[@]}"; do
@@ -22,4 +23,4 @@ for image in "${images[@]}"; do
 done
 
 # Once all images have been built, generate and push the 'terra-docker-versions-new' doc
-python scripts/generate_version_docs.py
+python scripts/generate_version_docs.py

config/conf.json

@@ -95,6 +95,24 @@
                 "include_in_custom_gce" : true
             }
         },
+        {
+            "name" : "terra-base",
+            "base_label" : "New Base",
+            "tools" : [
+                "python"
+            ],
+            "packages" : {
+
+            },
+            "version" : "1.0.0",
+            "automated_flags" : {
+                "generate_docs" : true,
+                "include_in_ui" : true,
+                "build" : true,
+                "include_in_custom_dataproc" : true,
+                "include_in_custom_gce" : true
+            }
+        },
         {
             "name" : "terra-jupyter-r",
             "base_label" : "R",

terra-base/CHANGELOG.md

@@ -0,0 +1,10 @@
+## 1.0.0 - 12/16/2025
+
+- Extends GPU-enabled Ubuntu base image
+- Uses Python 3.10
+- Add UV
+- Add conda
+- Add Jupyter
+- Add Leonardo customizations/extensions
+
+Image URL: `us.gcr.io/broad-dsp-gcr-public/terra-base:1.0.0`