Reworked docker docs (nhs-england-tools#155)

<!-- markdownlint-disable-next-line first-line-heading -->
## Description

Minor (but important!) changes to how docker is run, `DOCKER_IMAGE`
handling, and some docs.

## Context

This change wraps up a few different changes I needed to make when using
the template.

### Docker context


[Here](https://github.com/nhs-england-tools/repository-template/compare/alyo12-docker-docs?expand=1#diff-dddd47b45acd050e3b06771b95c68eee8ebcbfcf8a019b3f0186725a18cfcab6L33)
we were `cd`ing into the `infrastructure/images/whatever` directory
before running Docker. That restricts the docker context to only that
directory and below. You can't `COPY ../../. .` or whatever because
that's treated as a security problem.

This meant that you couldn't put your application code into a container
inside the `infrastructure/images` structure. That's fine if you've got
a single application container, and if you're happy with it living at
the root of the repo, but as soon as you've got more than one it gets a
bit messy (and I've never liked things that clutter the repo root at the
best of times).

The change I've made gets rid of the `cd` in and out; `docker` is
*always* run from the repo root. That means paths in `Dockerfile`s are
relative to the root, not to themselves; same with `.dockerignore`
(which I've made optional so that it doesn't break if there isn't one).

### `$DOCKER_IMAGE` variable handling

Minor tweak but a good quality of life improvement here. If you specify
`$DOCKER_IMAGE` to the `make docker-*` tasks but *not* `$dir` then it
defaults `$dir` to `infrastructure/images/$DOCKER_IMAGE`. That means the
`make` invocation, the filesystem, and the docker image list are all
synchronised to the same name, by convention.

### Updated docker usage docs

The example in the existing docs was specific to repackaging a third
party tool, but gave no guidance about how to package your own
application code. I've rewritten the `Quick start` section to show
packaging a trivial python script. I'm a little worried that the docs
are a bit repetitive now, but I don't think it's worth reworking right
now.

I've also removed the python example code. It was a good start, but in
practice having the tutorial docs is better because what I want to know
as a new user is *where to put my own stuff*, and the example wasn't a
huge help there and in fact was a bit of a confusion.

## Type of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply. -->

- [x] Refactoring (non-breaking change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would change existing
functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply. -->

- [ ] I am familiar with the [contributing
guidelines](../docs/CONTRIBUTING.md)
- [ ] I have followed the code style of the project
- [ ] I have added tests to cover my changes
- [ ] I have updated the documentation accordingly
- [ ] This PR is a result of pair or mob programming

---

## Sensitive Information Declaration

To ensure the utmost confidentiality and protect your and others
privacy, we kindly ask you to NOT including [PII (Personal Identifiable
Information) / PID (Personal Identifiable
Data)](https://digital.nhs.uk/data-and-information/keeping-data-safe-and-benefitting-the-public)
or any other sensitive data in this PR (Pull Request) and the codebase
changes. We will remove any PR that do contain any sensitive
information. We really appreciate your cooperation in this matter.

- [ ] I confirm that neither PII/PID nor sensitive data are included in
this PR and the codebase changes.

Author: regularfry

docs/developer-guides/Scripting_Docker.md

@@ -11,10 +11,13 @@
     - [Versioning](#versioning)
     - [Variables](#variables)
     - [Platform architecture](#platform-architecture)
+    - [`Dockerignore` file](#dockerignore-file)
   - [FAQ](#faq)
 
 ## Overview
 
+This document provides instructions on how to build Docker images using our automated build process. You'll learn how to specify version tags, commit changes, and understand the build output.
+
 Docker is a tool for developing, shipping and running applications inside containers for Serverless and Kubernetes-based workloads. It has grown in popularity due to its ability to address several challenges faced by engineers, like:
 
 - **Consistency across environments**: One of the common challenges in software development is the "it works on my machine" problem. Docker containers ensure that applications run the same regardless of where the container is run, be it a developer's local machine, a test environment or a production server.
@@ -43,7 +46,6 @@ Here are some key features built into this repository's Docker module:
 - Incorporates metadata through `Dockerfile` labels for enhanced documentation and to conform to standards
 - Integrates a linting routine to ensure `Dockerfile` code quality
 - Includes an automated test suite to validate Docker scripts
-- Provides a ready-to-run example to demonstrate the module's functionality
 - Incorporates a best practice guide
 
 ## Key files
@@ -61,50 +63,108 @@ Here are some key features built into this repository's Docker module:
   - [`docker.test.sh`](../../scripts/docker/tests/docker.test.sh): Main file containing all the tests
   - [`Dockerfile`](../../scripts/docker/tests/Dockerfile): Image definition for the test suite
   - [`VERSION`](../../scripts/docker/tests/VERSION): Version patterns for the test suite
-- Usage example
-  - Python-based example [`hello_world`](../../scripts/docker/examples/python) app showing a multi-staged build
-  - A set of [make targets](https://github.com/nhs-england-tools/repository-template/blob/main/scripts/docker/docker.mk#L18) to run the example
 
 ## Usage
 
 ### Quick start
 
-Run the test suite:
+The Repository Template assumes that you will want to build more than one docker image as part of your project.  As such, we do not use a `Dockerfile` at the root of the project.  Instead, each docker image that you create should go in its own folder under `infrastructure/images`.  So, if your application has a docker image called `my-shiny-app`, you should create the file `infrastructure/images/my-shiny-app/Dockerfile`.  Let's do that.
+
+First, we need an application to package.  Let's do the simplest possible thing, and create a file called `main.py` in the root of the template with a familiar command in it:
+
+```python
+print("hello world")
+```
+
+Run this command to make the directory:
+
+```shell
+mkdir -p infrastructure/images/my-shiny-app
+```
+
+Now, edit `infrastructure/images/my-shiny-app/Dockerfile` and put this into it:
+
+```dockerfile
+FROM python
+
+COPY ./main.py .
+
+CMD ["python", "main.py"]
+```
+
+Note the paths in the `COPY` command.  The `Dockerfile` is stored in a subdirectory, but when `docker` runs it is executed in the root of the repository so that's where all paths are relative to.  This is because you can't `COPY` from parent directories. `COPY ../../main.py .` wouldn't work.
+
+The name of the folder is also significant. It should match the name of the docker image that you want to create.  With that name, you can run the following `make` task to run `hadolint` over your `Dockerfile` to check for common anti-patterns:
+
+```shell
+ $ DOCKER_IMAGE=my-shiny-app make docker-lint
+/workdir/./infrastructure/images/my-shiny-app/Dockerfile.effective:1 DL3006 warning: Always tag the version of an image explicitly
+make[1]: *** [scripts/docker/docker.mk:34: _docker] Error 1
+make: *** [scripts/docker/docker.mk:20: docker-lint] Error 2
+```
+
+All the provided docker `make` tasks take the `DOCKER_IMAGE` parameter.
+
+`hadolint` found a problem, so let's fix that.  It's complaining that we've not specified which version of the `python` docker container we want. Change the first line of the `Dockerfile` to:
+
+```dockerfile
+FROM python:3.12-slim-bookworm
+```
+
+Run `DOCKER_IMAGE=my-shiny-app make docker-lint` again, and you will see that it is silent.
+
+Now let's actually build the image.  Run the following:
+
+```shell
+DOCKER_IMAGE=my-shiny-app make docker-build
+```
+
+And now we can run it:
+
+```shell
+ $ DOCKER_IMAGE=my-shiny-app make docker-run
+hello world
+```
+
+If you list your images, you'll see that the image name matches the directory name under `infrastructure/images`:
 
 ```shell
-$ make docker-test-suite-run
+ $ docker image ls
+REPOSITORY                   TAG                 IMAGE ID      CREATED        SIZE
+localhost/my-shiny-app       latest              6a0adeb5348c  2 hours ago    135 MB
+docker.io/library/python     3.12-slim-bookworm  d9f1825e4d49  5 weeks ago    135 MB
+localhost/hadolint/hadolint  2.12.0-alpine       19b38dcec411  16 months ago  8.3 MB
+```
 
-test-docker-build PASS
-test-docker-test PASS
-test-docker-run PASS
-test-docker-clean PASS
+Your process might want to add specific tag formats so you can identify docker images by date-stamps, or git hashes.  The Repository Template supports that with a `VERSION` file.  Create a new file called `infrastructure/images/my-shiny-app/VERSION`, and put the following into it:
+
+```text
+${yyyy}${mm}${dd}-${hash}
 ```
 
-Run the example:
+Now, run the `docker-build` command again, and towards the end of the output you will see something that looks like this:
 
 ```shell
-$ make docker-example-build
+Successfully tagged localhost/my-shiny-app:20240314-07ee679
+```
 
-#0 building with "desktop-linux" instance using docker driver
-...
-#12 DONE 0.0s
+Obviously the specific values will be different for you.  See the Versioning section below for more on this.
 
-$ make docker-example-run
+It is usually the case that there is a specific image that you will most often want to build, run, and deploy.  You should edit the root-level `Makefile` to document this and to provide shortcuts.  Edit `Makefile`, and change the `build` task to look like this:
 
- * Serving Flask app 'app'
- * Debug mode: off
-WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
- * Running on all addresses (0.0.0.0)
- * Running on http://127.0.0.1:8000
- * Running on http://172.17.0.2:8000
-Press CTRL+C to quit
+```make
+build: # Build the project artefact @Pipeline
+	DOCKER_IMAGE=my-shiny-app
+	make docker-build
 ```
 
+Now when you run `make build`, it will do the right thing.  Keeping this convention consistent across projects means that new starters can be on-boarded quickly, without needing to learn a new set of conventions each time.
+
 ### Your image implementation
 
-Always follow [Docker best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) while developing images. Start with creating your container definition for the service and store it in the `infrastructure/images` directory.
+Always follow [Docker best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) while developing images.
 
-Here is a step-by-step guide:
+Here is a step-by-step guide for an image which packages a third-party tool.  It is mostly similar to the example above, but demonstrates the `.tool-versions` mechanism.
 
 1. Create `infrastructure/images/cypress/Dockerfile`
 
@@ -212,6 +272,10 @@ Set the `docker_image` or `DOCKER_IMAGE` variable for your image. Alternatively,
 
 For cross-platform image support, the `--platform linux/amd64` flag is used to build Docker images, enabling containers to run without any changes on both `amd64` and `arm64` architectures (via emulation).
 
+### `Dockerignore` file
+
+If you need to exclude files from a `COPY` command, put a [`Dockerfile.dockerignore`](https://docs.docker.com/build/building/context/#filename-and-location) file next to the relevant `Dockerfile`.  They do not live in the root directory.  Any paths within `Dockerfile.dockerignore` must be relative to the repository root.
+
 ## FAQ
 
 1. _We built our serverless workloads based on AWS Lambda and package them as `.zip` archives. Why do we need Docker?_

scripts/config/markdownlint.yaml

@@ -1,5 +1,11 @@
 # SEE: https://github.com/DavidAnson/markdownlint/blob/main/schema/.markdownlint.yaml
 
+# https://github.com/DavidAnson/markdownlint/blob/main/doc/md010.md
+MD010: # no-hard-tabs
+  ignore_code_languages:
+    - make
+    - console
+
 # https://github.com/DavidAnson/markdownlint/blob/main/doc/md013.md
 MD013: false
 

scripts/docker/docker.lib.sh

@@ -27,10 +27,9 @@ function docker-build() {
 
   version-create-effective-file
   _create-effective-dockerfile
-  # The current directory must be changed for the image build script to access
-  # assets that need to be copied
-  current_dir=$(pwd)
-  cd "$dir"
+
+  tag=$(_get-effective-tag)
+
   docker build \
     --progress=plain \
     --platform linux/amd64 \
@@ -43,16 +42,36 @@ function docker-build() {
     --build-arg GIT_COMMIT_HASH="$(git rev-parse --short HEAD)" \
     --build-arg BUILD_DATE="$(date -u +"%Y-%m-%dT%H:%M:%S%z")" \
     --build-arg BUILD_VERSION="$(_get-effective-version)" \
-    --tag "${DOCKER_IMAGE}:$(_get-effective-version)" \
+    --tag "${tag}" \
     --rm \
     --file "${dir}/Dockerfile.effective" \
     .
-  cd "$current_dir"
+
   # Tag the image with all the stated versions, see the documentation for more details
   for version in $(_get-all-effective-versions) latest; do
-    docker tag "${DOCKER_IMAGE}:$(_get-effective-version)" "${DOCKER_IMAGE}:${version}"
+    if [ ! -z "$version" ]; then
+      docker tag "${tag}" "${DOCKER_IMAGE}:${version}"
+    fi
   done
-  docker rmi --force "$(docker images | grep "<none>" | awk '{print $3}')" 2> /dev/null ||:
+}
+
+# Create the Dockerfile.effective file to bake in version info
+# Arguments (provided as environment variables):
+#   dir=[path to the Dockerfile to use, default is '.']
+function docker-bake-dockerfile() {
+
+  local dir=${dir:-$PWD}
+
+  version-create-effective-file
+  _create-effective-dockerfile
+}
+
+# Run hadolint over the generated Dockerfile.
+# Arguments (provided as environment variables):
+#  dir=[path to the image directory where the Dockerfile.effective is located, default is '.']
+function docker-lint() {
+  local dir=${dir:-$PWD}
+  file=${dir}/Dockerfile.effective ./scripts/docker/dockerfile-linter.sh
 }
 
 # Check test Docker image.
@@ -81,12 +100,13 @@ function docker-check-test() {
 function docker-run() {
 
   local dir=${dir:-$PWD}
+  local tag=$(dir="$dir" _get-effective-tag)
 
   # shellcheck disable=SC2086
   docker run --rm --platform linux/amd64 \
     ${args:-} \
-    "${DOCKER_IMAGE}:$(dir="$dir" _get-effective-version)" \
-    ${cmd:-}
+    "${tag}" \
+    ${DOCKER_CMD:-}
 }
 
 # Push Docker image.
@@ -114,7 +134,8 @@ function docker-clean() {
   done
   rm -f \
     .version \
-    Dockerfile.effective
+    Dockerfile.effective \
+    Dockerfile.effective.dockerignore
 }
 
 # Create effective version from the VERSION file.
@@ -207,6 +228,13 @@ function _create-effective-dockerfile() {
 
   local dir=${dir:-$PWD}
 
+  # If it exists, we need to copy the .dockerignore file to match the prefix of the
+  # Dockerfile.effective file, otherwise docker won't use it.
+  # See https://docs.docker.com/build/building/context/#filename-and-location
+  # If using podman, this requires v5.0.0 or later.
+  if [ -f "${dir}/Dockerfile.dockerignore" ]; then
+    cp "${dir}/Dockerfile.dockerignore" "${dir}/Dockerfile.effective.dockerignore"
+  fi
   cp "${dir}/Dockerfile" "${dir}/Dockerfile.effective"
   _replace-image-latest-by-specific-version
   _append-metadata
@@ -276,6 +304,20 @@ function _get-effective-version() {
   head -n 1 "${dir}/.version" 2> /dev/null ||:
 }
 
+# Print the effective tag for the image with the version. If you don't have a VERSION file
+# then the tag will be just the image name.  Otherwise it will be the image name with the version.
+# Arguments (provided as environment variables):
+#   dir=[path to the image directory where the Dockerfile is located, default is '.']
+function _get-effective-tag() {
+
+  local tag=$DOCKER_IMAGE
+  version=$(_get-effective-version)
+  if [ ! -z "$version" ]; then
+    tag="${tag}:${version}"
+  fi
+  echo "$tag"
+}
+
 # Print all Docker image versions.
 # Arguments (provided as environment variables):
 #   dir=[path to the image directory where the Dockerfile is located, default is '.']

scripts/docker/docker.mk

@@ -4,26 +4,39 @@
 # Custom implementation - implementation of a make target should not exceed 5 lines of effective code.
 # In most cases there should be no need to modify the existing make targets.
 
-docker-build: # Build Docker image - optional: docker_dir|dir=[path to the Dockerfile to use, default is '.'] @Development
-	make _docker cmd="build" \
+DOCKER_IMAGE ?= $(or ${docker_image}, $(or ${IMAGE}, $(or ${image}, ghcr.io/org/repo)))
+DOCKER_TITLE ?= $(or "${docker_title}", $(or "${TITLE}", $(or "${title}", "Service Docker image")))
+
+docker-bake-dockerfile: # Create Dockerfile.effective - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
+	make _docker cmd="bake-dockerfile" \
 		dir=$(or ${docker_dir}, ${dir})
-	file=$(or ${docker_dir}, ${dir})/Dockerfile.effective
-	scripts/docker/dockerfile-linter.sh
+
+docker-build: # Build Docker image - optional: docker_dir|dir=[path to the Dockerfile to use, default is '.'] @Development
+	dir=$(or ${docker_dir}, ${dir})
+	make _docker cmd="build"
+docker-build: docker-lint
+
+docker-lint: # Run hadolint over the Dockerfile - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
+	dir=$(or ${docker_dir}, ${dir})
+	make _docker cmd="lint"
+docker-lint: docker-bake-dockerfile
 
 docker-push: # Push Docker image - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
 	make _docker cmd="push" \
 		dir=$(or ${docker_dir}, ${dir})
 
+docker-run: # Run Docker image - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
+	make _docker cmd="run" \
+		dir=$(or ${docker_dir}, ${dir})
+
 clean:: # Remove Docker resources (docker) - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Operations
 	make _docker cmd="clean" \
 		dir=$(or ${docker_dir}, ${dir})
 
 _docker: # Docker command wrapper - mandatory: cmd=[command to execute]; optional: dir=[path to the image directory where the Dockerfile is located, relative to the project's top-level directory, default is '.']
 	# 'DOCKER_IMAGE' and 'DOCKER_TITLE' are passed to the functions as environment variables
-	DOCKER_IMAGE=$(or ${DOCKER_IMAGE}, $(or ${docker_image}, $(or ${IMAGE}, $(or ${image}, ghcr.io/org/repo))))
-	DOCKER_TITLE=$(or "${DOCKER_TITLE}", $(or "${docker_title}", $(or "${TITLE}", $(or "${title}", "Service Docker image"))))
+	dir=$(realpath $(or ${dir}, infrastructure/images/${DOCKER_IMAGE}))
 	source scripts/docker/docker.lib.sh
-	dir=$(realpath ${dir})
 	docker-${cmd} # 'dir' is accessible by the function as environment variable
 
 # ==============================================================================
@@ -40,44 +53,15 @@ docker-shellscript-lint: # Lint all Docker module shell scripts @Quality
 docker-test-suite-run: # Run Docker test suite @ExamplesAndTests
 	scripts/docker/tests/docker.test.sh
 
-docker-example-build: # Build Docker example @ExamplesAndTests
-	source scripts/docker/docker.lib.sh
-	cd scripts/docker/examples/python
-	DOCKER_IMAGE=repository-template/docker-example-python
-	DOCKER_TITLE="Repository Template Docker Python Example"
-	TOOL_VERSIONS="$(shell git rev-parse --show-toplevel)/scripts/docker/examples/python/.tool-versions.example"
-	docker-build
-
-docker-example-lint: # Lint Docker example @ExamplesAndTests
-	dockerfile=scripts/docker/examples/python/Dockerfile
-	file=$${dockerfile} scripts/docker/dockerfile-linter.sh
-
-docker-example-run: # Run Docker example @ExamplesAndTests
-	source scripts/docker/docker.lib.sh
-	cd scripts/docker/examples/python
-	DOCKER_IMAGE=repository-template/docker-example-python
-	args=" \
-		-it \
-		--publish 8000:8000 \
-	"
-	docker-run
-
-docker-example-clean: # Remove Docker example resources @ExamplesAndTests
-	source scripts/docker/docker.lib.sh
-	cd scripts/docker/examples/python
-	DOCKER_IMAGE=repository-template/docker-example-python
-	docker-clean
-
 # ==============================================================================
 
 ${VERBOSE}.SILENT: \
 	_docker \
 	clean \
+	docker-bake-dockerfile \
 	docker-build \
-	docker-example-build \
-	docker-example-clean \
-	docker-example-lint \
-	docker-example-run \
+	docker-lint \
 	docker-push \
+	docker-run \
 	docker-shellscript-lint \
 	docker-test-suite-run \