Containerised build and test support for EMS #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

sharon-tickell wants to merge 22 commits into csiro-coasts:dev from eReefs:recom2-dev

.dockerignore

-Original file line number
+Diff line change
@@ -0,0 +1,6 @@
+    .env
+    .gitlab_ci.yaml
+    *.log
+    *.md
+    docker-compose.yml
+    hooks

.gitignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -22,3 +22,17 @@ libjigsaw64r.so @@
     # Misc
     sediments-us
+    # Common test output files
+    # (Test directories may also have their own .gitignore files)
+    model/tests/ems_test.log
+    model/tests/**/alert.*
+    model/tests/**/core.*
+    model/tests/**/diag.txt
+    model/tests/**/in*.nc
+    model/tests/**/loc*.ts
+    model/tests/**/obc_spec.txt
+    model/tests/**/out*.nc
+    model/tests/**/runlog
+    model/tests/**/setup.txt
+    model/tests/**/out/*
+    !model/tests/**/out/.empty

Dockerfile

-Original file line number
+Diff line change
@@ -0,0 +1,69 @@
+    #------------------------------------------------------------------------------
+    # CSIRO EMS Image Build Script
+    #------------------------------------------------------------------------------
+    # The recommended base image is one of the onaci/ereefs-netcdf-base variants,
+    # as those have been designed to include all the EMS dependencies.
+    # Allow it to be overridden in order to choose *which* variant, or even
+    # something completely different.
+    ARG BASE_IMAGE="onaci/ereefs-netcdf-base:python-3.11-slim-bullseye"
+    FROM ${BASE_IMAGE}
+    # Record the actual base image used from the FROM command as a label.
+    ARG BASE_IMAGE
+    LABEL org.opencontainers.image.base.name=${BASE_IMAGE}
+    # Enable Bash in RUN commands, and ensure that any commands with
+    # pipes exit on the first failure.
+    SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+    # Prepare a base directory for all the EMS code
+    ARG EMS_BASE="/usr/local/src/ems"
+    ENV EMS_BASE="${EMS_BASE}"
+    WORKDIR ${EMS_BASE}
+    # Install all the EMS Source Code
+    COPY ./ ./
+    # Selectively remove some subdirectories that we want to omit
+    # (List can be overridden by a build argument)
+    ARG EMS_RM_DIRS=""
+    RUN if [ -n "${EMS_RM_DIRS}" ]; then for d in $EMS_RM_DIRS; do rm -rf "${EMS_BASE}/${d}"; done; fi
+    # Compile the EMS components
+    # We build with OpenMP support by default for use in a docker container
+    # Different options can be specified by overriding the EMS_CONFIGURE_OPTS build argument
+    # (The base image *does* include NetCDF and HDF5 with Open MPI support, so that is an option)
+    ENV EMS_BUILD_LOG="${EMS_BASE}/ems_build.log"
+    ARG EMS_CONFIGURE_OPTS="--enable-omp"
+    RUN make distclean || true \
+        && conf/configure ${EMS_CONFIGURE_OPTS} 2>&1 | tee -a "${EMS_BUILD_LOG}"
+    RUN make clean \
+        && make 2>&1 | tee -a "${EMS_BUILD_LOG}"
+    RUN make check install 2>&1 | tee -a "${EMS_BUILD_LOG}"
+    # Symlink the EMS executable(s) into the default path
+    # (This will fail if the executable did not build for some reason)
+    ENV SHOC="${EMS_BASE}/model/hd/shoc" \
+        COMPAS="${EMS_BASE}/model/hd-us/compas"
+    RUN if [ -n "${SHOC}" ]; then chmod 0755 "${SHOC}" && ln -s "${SHOC}" /usr/local/bin/shoc; fi; \
+        if [ -n "${COMPAS}" ]; then chmod 0755 "${COMPAS}" && ln -s "${COMPAS}" /usr/local/bin/compas; fi; \
+        ln -s "${EMS_BASE}/model/tests/run-tests.sh" /usr/local/bin/run-ems-tests
+    # Optionally run all available unit tests
+    # (Override the EMS_TEST_RUN build argument to have a value of 0 to skip testing at build time)
+    ARG EMS_TEST_RUN=1
+    ARG EMS_TEST_INCLUDE="${EMS_BASE}/model/tests"
+    ENV EMS_TEST_INCLUDE="${EMS_TEST_INCLUDE}"
+    ARG EMS_TEST_EXCLUDE=""
+    ENV EMS_TEST_EXCLUDE="${EMS_TEST_EXCLUDE}"
+    ENV EMS_TEST_LOG="${EMS_BASE}/model/tests/ems_test.log"
+    RUN if [ $EMS_TEST_RUN -eq 1 ]; then run-ems-tests; fi
+    # Encode EMS metadata in labels
+    LABEL au.csiro.ems.base=${EMS_BASE} \
+          au.csiro.ems.shoc=${SHOC} \
+          au.csiro.ems.compas=${COMPAS}
+    # Configure the default entrypoint to be a default EMS executable
+    ENTRYPOINT ["/bin/bash", "-c" ]
+    CMD [ "${SHOC:-$COMPAS}", "-v"]

docker-compose.test.yml

-Original file line number
+Diff line change
@@ -0,0 +1,22 @@
+    ---
+    #-------------------------------------------------------------------------------
+    # Docker *test* environment composition for EMS
+    # This exists to support the autotest option for a Dockerhub repository build
+    # (See: https://docs.docker.com/docker-hub/builds/automated-testing/)
+    #
+    # If this repository and rbanch are configured as Dockerhub autotest
+    # repositories, this will automatically build and test EMS with all default
+    # settings every time a push is made to the repository.
+    #
+    # The resulting image is not automatically pushed to the DockerHub registry
+    # unless the same respository is automatically configured for autobuild.
+    #-------------------------------------------------------------------------------
+    version: "3.8"
+    services:
+      ems:
+        build:
+          context: .
+          args:
+            EMS_TEST_RUN: 0
+        command: "run-ems-tests"

docker-compose.yml

-Original file line number
+Diff line change
@@ -0,0 +1,23 @@
+    ---
+    #-------------------------------------------------------------------------------
+    # Docker development environment composition for ems
+    # (Not for production use!)
+    #
+    # This default compose stack is designed for use on a local development host.
+    #-------------------------------------------------------------------------------
+    version: "3.8"
+    services:
+      ems:
+        build:
+          context: .
+          args:
+            #EMS_RM_DIRS: "ext/swan model/lib/da model/lib/exchange"
+            EMS_TEST_RUN: 0
+        command: "run-ems-tests"
+        container_name: ems
+        environment:
+          EMS_TEST_INCLUDE: '/usr/local/src/ems/model/tests'
+        image: onaci/ereefs-ems-shoc:dev
+        volumes:
+          - ./model/tests/:/usr/local/src/ems/model/tests/

hooks/README.md

-Original file line number
+Diff line change
@@ -0,0 +1,7 @@
+    # csiro-coasts/EMS/hooks
+    The files in this directory are custom build phase hooks for the automated dockerhub builder,
+    and are used to set up custom build configurations and docker registry push options for when
+    this repository is set up as a DockerHub repository with autobuild enabled.
+    See <https://docs.docker.com/docker-hub/builds/advanced/#custom-build-phase-hooks> for more information about how they work.

hooks/build

-Original file line number
+Diff line change
@@ -0,0 +1,62 @@
+    #!/bin/bash
+    # Derive useful variables from the dockerhub build environment
+    BUILD_TIMESTAMP="$(date --rfc-3339=seconds)"
+    DOCKER_TAG="${DOCKER_TAG:-}"
+    DOCKERFILE_PATH="${DOCKERFILE_PATH:-./Dockerfile}"
+    IMAGE_NAME="${IMAGE_NAME:-csiro-coasts/ems}"
+    SAFE_TIMESTAMP="$(echo $BUILD_TIMESTAMP | sed 's/ /T/g' | sed 's/:/-/g' | sed 's/\+.*//')"
+    SOURCE_URL="$(git remote get-url origin || true)"
+    VERSION_TAG="${SOURCE_BRANCH}_v${SAFE_TIMESTAMP}-${SOURCE_COMMIT}"
+    # Ensure we have values for all required build arguments.
+    # These can be configured as environment variables on the dockerhub build,
+    # but if they are not, then we parse the defaults out of the Dockerfile.
+    BASE_IMAGE="${BASE_IMAGE:-}"
+    if [ -z "${BASE_IMAGE:-}" ]; then
+        BASE_IMAGE=$(grep 'ARG BASE_IMAGE=' ./Dockerfile | tr '"' ' ' | tr '=' ' ' | awk '{print $3}')
+    fi
+    EMS_CONFIGURE_OPTS="${EMS_CONFIGURE_OPTS:-}"
+    if [ -z "${EMS_CONFIGURE_OPTS:-}" ]; then
+        EMS_CONFIGURE_OPTS=$(grep 'ARG EMS_CONFIGURE_OPTS=' ./Dockerfile | tr '"' ' ' | tr '=' ' ' | awk '{print $3}')
+    fi
+    EMS_RM_DIRS="${EMS_RM_DIRS:-}"
+    if [ -z "${EMS_RM_DIRS:-}" ]; then
+        EMS_RM_DIRS=$(grep 'ARG EMS_RM_DIRS=' ./Dockerfile | tr '"' ' ' | tr '=' ' ' | awk '{print $3}')
+    fi
+    EMS_TEST_RUN="${EMS_TEST_RUN:-}"
+    if [ -z "${EMS_TEST_RUN:-}" ]; then
+        EMS_TEST_RUN=$(grep 'ARG EMS_TEST_RUN=' ./Dockerfile | tr '"' ' ' | tr '=' ' ' | awk '{print $3}')
+    fi
+    # Treat most DOCKER_TAG values as overrides for the
+    # BASE_IMAGE variable sourced from the .env file
+    if [[ -n "${DOCKER_TAG}" ]] && [[ "${DOCKER_TAG}" != "latest" ]] && [[ "${DOCKER_TAG}" != "${SOURCE_BRANCH}" ]]; then
+        # Assume the custom docker tag is an override for the tag part of
+        # the BASE_IMAGE configured in the Dockerfile and any .env file.
+        BASE_IMAGE="$(echo $BASE_IMAGE | sed -E 's/^([^:]*):.*$/\1/'):${DOCKER_TAG}"
+        # Override the VERSION_TAG to include it there too.
+        VERSION_TAG="${SOURCE_BRANCH}_${DOCKER_TAG}_v${SAFE_TIMESTAMP}-${SOURCE_COMMIT}"
+    fi
+    # Build our customised docker image
+    docker build --pull \
+        --build-arg "BASE_IMAGE=${BASE_IMAGE}" \
+        --build-arg "EMS_CONFIGURE_OPTS=${EMS_CONFIGURE_OPTS}" \
+        --build-arg "EMS_RM_DIRS=${EMS_RM_DIRS}" \
+        --build-arg "EMS_TEST_RUN=${EMS_TEST_RUN}" \
+        --label "org.opencontainers.image.authors=${BUILD_AUTHORS:-CSIRO Coastal Group}" \
+        --label "org.opencontainers.image.branch=${SOURCE_BRANCH}" \
+        --label "org.opencontainers.image.buildhost=${DOCKER_REPO}" \
+        --label "org.opencontainers.image.created=${BUILD_TIMESTAMP}" \
+        --label "org.opencontainers.image.licenses=https://github.com/csiro-coasts/EMS/blob/master/LICENSE.md" \
+        --label "org.opencontainers.image.revision=${SOURCE_COMMIT}" \
+        --label "org.opencontainers.image.source=${SOURCE_URL}" \
+        --label "org.opencontainers.image.title=${DOCKER_REPO:-csiro-coasts/EMS}" \
+        --label "org.opencontainers.image.url=https://github.com/csiro-coasts/EMS" \
+        --label "org.opencontainers.image.vendor=CSIRO" \
+        --label "org.opencontainers.image.version=${VERSION_TAG}" \
+        -f ${DOCKERFILE_PATH} \
+        -t ${IMAGE_NAME} \
+        .

hooks/post_push

-Original file line number
+Diff line change
@@ -0,0 +1,8 @@
+    #!/bin/bash
+    VERSION_TAG="$(docker image inspect  --format '{{ index .Config.Labels "org.opencontainers.image.version"}}' $IMAGE_NAME || true)"
+    if [ -n "${VERSION_TAG}" ]; then
+        IMAGE_TAGGED="${DOCKER_REPO}:${VERSION_TAG}"
+        docker tag $IMAGE_NAME $IMAGE_TAGGED
+        docker push $IMAGE_TAGGED
+    fi

model/tests/hd-us/.gitignore

-Original file line number
+Diff line change
@@ -0,0 +1,18 @@
+    boundary.txt
+    closed*.nc
+    closed*.txt
+    closed*.us
+    est.nc
+    est_hex*.txt
+    est_hex*.us
+    est_quad*.nc
+    est_quad*.txt
+    plot_*_hex.m
+    in1*.txt
+    in1*.us
+    points.txt
+    pt.nc
+    s_*_quad.nc
+    segment.txt
+    test2*.txt
+    totals.ts

model/tests/hd-us/cetas/.gitignore

-Original file line number
+Diff line change
@@ -0,0 +1,10 @@
+    *.msh
+    *.txt
+    *.us
+    *.jig
+    *.site
+    in.nc
+    in.prm
+    outputs/*
+    !outputs/.empty
+    plot_auto.m

model/tests/hd-us/cetas/run_cetas

-Original file line number
+Diff line change
@@ -1,15 +1,19 @@
-    #!/bin/csh -f
+    #!/bin/csh -exf
     echo "Testing COMPAS cetas..."
-    rm *.msh
-    rm *.txt
-    rm *.us
-    rm *.jig
-    rm *.site
-    rm in.nc
-    ../compas -ag auto.prm
+    if (! $?COMPAS || "$COMPAS" == "") then
+        set COMPAS='../compas'
+    endif
-    echo "DONE"
+    set nonomatch
+    rm -f *.msh
+    rm -f *.txt
+    rm -f *.us
+    rm -f *.jig
+    rm -f *.site
+    rm -f in.nc
+    $COMPAS -ag auto.prm
+    echo "DONE"

model/tests/hd-us/closed/run_closed

-Original file line number
+Diff line change
@@ -1,8 +1,13 @@
     #!/bin/csh -ef
-    # Set relative path to compas executable
-    set COMPAS='../../../hd-us/compas'
-    set SHOC='../../../hd/shoc'
+    if (! $?COMPAS || "$COMPAS" == "") then
+        # Set relative path to compas executable
+        set COMPAS='../../../hd-us/compas'
+    endif
+    if (! $?SHOC || "$SHOC" == "") then
+        # Set relative path to shoc executable
+        set SHOC='../../../hd/shoc'
+    endif
     echo "Testing SHOC..."
     rm -f closed.nc || true
@@ Expand All / @@ -14,7 +19,7 @@ if (-f $SHOC) then @@
     else
         echo "SHOC not found, skipping ..."
     endif
     echo "DONE"
     echo "Testing COMPAS quad..."
@@ Expand Down Expand Up / @@ -42,6 +47,3 @@ $COMPAS -g closed_hex.prm closed_hex.nc @@
     $COMPAS -p closed_hex.prm
     echo "DONE"

model/tests/hd-us/est/run_est

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,47 +1,55 @@
  
    #!/bin/csh -f

    #!/bin/csh -ef

    if (! $?SHOC || "$SHOC" == "") then

        set SHOC='../shoc'

    endif

    if (! $?COMPAS || "$COMPAS" == "") then

        set COMPAS='../compas'

    endif

    echo "Testing SHOC..."

    rm est.nc

    rm out1.nc

    rm -f est.nc

    rm -f out1.nc

    ../shoc -g est.prm est.nc

    $SHOC -g est.prm est.nc

    ../shoc -p est.prm

    $SHOC -p est.prm

    echo "DONE"

    echo "Testing COMPAS quad..."

    rm est_quad.nc

    rm out1_quad.nc

    rm -f est_quad.nc

    rm -f out1_quad.nc

    rm -f s_est_quad.nc

    ../compas -g est_quad.prm est_quad.nc

    $COMPAS -g est_quad.prm est_quad.nc

    ../compas -p est_quad.prm

    $COMPAS -p est_quad.prm

    echo "DONE"

    echo "Testing COMPAS quad 5 window..."

    rm out1_quad5w.nc

    rm -f out1_quad5w.nc

    ../compas -p est_quad5w.prm

    $COMPAS -p est_quad5w.prm

    echo "DONE"

    echo "Testing COMPAS hex..."

    rm est_hex.nc

    rm out1_hex.nc

    rm -f est_hex.nc

    rm -f out1_hex.nc

    ../compas -g est_hex.prm est_hex.nc

    $COMPAS -g est_hex.prm est_hex.nc

    ../compas -p est_hex.prm

    $COMPAS -p est_hex.prm

    echo "DONE"

    echo "Testing COMPAS hex 5 window..."

    rm out1_hex5w.nc

    rm -f out1_hex5w.nc

    ../compas -p est_hex5w.prm

    $COMPAS -p est_hex5w.prm

    echo "DONE"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Containerised build and test support for EMS #25

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Containerised build and test support for EMS #25

Are you sure you want to change the base?

Uh oh!

Containerised build and test support for EMS #25

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!