Updates for Docker (#947, #948)

* Update Dockerfiles and image build script to match output
* Add full Dockerfile
* Update Docker docs

Author: tahorst

README.md

@@ -9,12 +9,23 @@ You can reach us at [AllenCenterCovertLab](mailto:[email protected]
 
 See [docs/README.md](docs/README.md) for more info on setting up and running the model.
 
-In short, there are two alternative setups to run the model: inside a Docker container vs. in a manually constructed `pyenv` virtual environment.
-
+In short, there are two alternative setups to run the model: inside a Docker container vs. in a manually constructed `pyenv` virtual environment.  With Docker, you can start running a simulation with three commands:
+1.  Pull the Docker image:
+    ```shell script
+    docker pull docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full:latest
+    ```
+1. Run the Docker container:
+    ```shell script
+    docker run --name=wcm -it --rm docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full
+    ```
+1. Inside the container, run the model:
+    ```shell script
+    python runscripts/manual/runSim.py
+    ```
 
 ## Quick start
 
-When running this code, prepare with these steps (the wcm-code Docker container already prepares this for you):
+When running this code, prepare with these steps (the wcm-full Docker container already prepares this for you):
 
 1. `cd` to the top level of your `wcEcoli` directory.
 2. Set the `$PYTHONPATH`:
@@ -46,7 +57,7 @@ These scripts have command line interfaces built on Python's `argparse`, so you
 **NOTE:** _Use the `-h` or `--help` switch to get complete, up-to-date documentation on the command line options._ Below are just _some_ of the command line options.
 
 
-To run the parameter calculator (ParCa), which is needed to prepare input data for the simulation:
+To run the parameter calculator (ParCa), which is needed to prepare input data for the simulation (this step has already been run when building the wcm-full Docker image and can be skipped if running a container from that image):
 ```bash
 python runscripts/manual/runFitter.py [-h] [--cpus CPUS] [sim_outdir]
 ```

cloud/build-containers.sh

@@ -1,13 +1,22 @@
-#!/bin/sh
+#! /usr/bin/env bash
 # Build the WCM Docker container images locally.
 #
 # ASSUMES: The current working dir is the wcEcoli/ project root.
 
 set -eu
 
+# On macOS, build with `NO_AVX2=1` to avoid the OpenBLAS 0.3.6+ self-test failures
+# and bad results when building with Docker Desktop on macOS.
+if [ "$(uname -s)" == Darwin ]; then NO_AVX2=1; else NO_AVX2=0; fi
+
 # Docker image #1: The Python runtime environment.
-docker build -f cloud/docker/runtime/Dockerfile -t wcm-runtime .
+docker build -f cloud/docker/runtime/Dockerfile -t wcm-runtime \
+  --build-arg NO_AVX2=$NO_AVX2 .
 
 # Docker image #2: The Whole Cell Model code on the runtime environment.
 # See this Dockerfile for usage instructions.
 docker build -f cloud/docker/wholecell/Dockerfile -t wcm-code .
+
+# Docker image #3: The Whole Cell Model code with parameters calculated, ready for sims.
+# See this Dockerfile for usage instructions.
+docker build -f cloud/docker/full/Dockerfile -t wcm-full .

cloud/docker/full/Dockerfile

@@ -0,0 +1,27 @@
+# Container image #3: wcm-full.
+# This Dockerfile builds a container image with the full Whole Cell Model, ready
+# to run simulations, layered on the wcm-code image.
+#
+# To build locally from the wcEcoli/ project root directory:
+#
+#     > docker build -f cloud/docker/full/Dockerfile -t wcm-full .
+#
+# After building locally you can start up a new container from the image:
+#
+#     > docker run --name wcm -it --rm wcm-full
+#
+# It will start a shell where you can execute commands:
+#
+#     # nosetests
+#
+# If this succeeds you should be good to go, e.g.:
+#
+#     # python runscripts/manual/runSim.py
+
+
+ARG from=wcm-code:latest
+FROM ${from}
+
+RUN python runscripts/manual/runFitter.py
+
+CMD ["/bin/bash"]

cloud/docker/runtime/Dockerfile

@@ -17,21 +17,21 @@
 # Add option `--build-arg from=ABC` to build from a different base image but
 # DO NOT USE an alpine base since the simulation math comes out different!
 # See https://pythonspeed.com/articles/alpine-docker-python/ for more reasons.
-ARG from=python:2.7.16
+ARG from=python:2.7.16-stretch
 FROM ${from}
 
 # Option `--build-arg NO_AVX2=1` controls OpenBLAS' use of AVX2 instructions:
 #
-# * NO_AVX2=1 is needed to build or run properly in Docker-for-Mac due to a
+# * NO_AVX2=1 is needed to build properly in Docker-for-Mac due to a
 #   Docker bug.
 # * NO_AVX2=0 reportedly runs 20-30% faster BLAS if you're sure it'll only
-#   build and run in Docker-for-Linux, but this only saves ~7% in a cell sim.
+#   build in Docker-for-Linux, but this only saves ~7% in a cell sim.
 #
 # Docker-for-Mac bug:
 # https://github.com/xianyi/OpenBLAS/issues/2244
 # https://github.com/docker/for-mac/issues/4576
 # https://github.com/machyve/xhyve/issues/171
-ARG NO_AVX2=1
+ARG NO_AVX2=0
 ENV NO_AVX2="$NO_AVX2"
 
 RUN apt-get update \

cloud/docker/wholecell/Dockerfile

@@ -41,6 +41,9 @@ ENV PYTHONPATH=/wcEcoli
 RUN umask 000 && rm -rf fixtures && mkdir fixtures \
 	&& chmod a+rwx reconstruction/ecoli/dataclasses/process
 
-RUN echo 'cat cloud/docker/welcome.txt' >> $HOME/.bashrc
+# Show a welcome message with tips on wcEcoli.
+# Copy .bashrc into / for users w/o home dirs such as `docker run --user ...`
+RUN (echo 'cat cloud/docker/welcome.txt' >> ~/.bashrc \
+    && cp ~/.bashrc /)
 
 CMD ["/bin/bash"]

docs/README.md

@@ -7,20 +7,18 @@ These are the docs for a variety of topics on the Whole Cell Model.
 There are two alternative ways to set up to run the model:
 
 1. **Docker Container:** Install the
-   [Docker Desktop software](https://www.docker.com/products/docker-desktop),
-   build a Docker Image, then run the Docker Image as a Container.
-
-   Build the wcEcoli Docker Image this way:
+   [Docker Desktop software](https://www.docker.com/products/docker-desktop)
+   and run the Docker Image as a Container.
 
+   Pull the full docker image from the package registry in this repo:
    ```shell script
-   cd $YOUR_CODE_PROJECTS_DIR/wcEcoli  # or wherever you cloned the wcEcoli project to
-   cloud/build-containers.sh
+   docker pull docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full:latest
    ```
 
    You can then run the wcEcoli model inside the Container like this:
 
    ```shell script
-   docker run --name=wcm -it --rm wcm-code
+   docker run --name=wcm -it --rm docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full
    ```
 
    The `-it` option starts an interactive shell.
@@ -30,10 +28,11 @@ There are two alternative ways to set up to run the model:
    exit so you don't have to remember to delete old Containers.
 
    You can mount your local directory `wcEcoli/out/` into the Container to preserve the
-   program's output files when the Container exits:
+   program's output files when the Container exits - just be sure to provide a full path
+   to `out/` (eg. `$PWD/out`), not just a relative path from your current directory:
 
    ```shell script
-   docker run --name=wcm -v $PWD/out:/wcEcoli/out -it wcm-code
+   docker run --name=wcm -v $PWD/out:/wcEcoli/out -it docker.pkg.github.com/covertlab/wholecellecolirelease/wcm-full
    ```
 
    In this case, the output files will be owned by root. You can fix
@@ -45,7 +44,7 @@ There are two alternative ways to set up to run the model:
 
    **NOTE:** If you encounter memory issues while using Docker Desktop (the default allocated memory is 2GB) and the simulation processes get killed midway, click the Docker icon > Preferences > Advanced > adjust memory to 4GB.
 
-   **NOTE:** Docker Desktop for Windows is not currently compatible with VirtualBox.  If you use VirtualBox, try installing the legacy [Docker Toolbox](https://github.com/docker/toolbox/releases) instead.  You may also need to adjust the memory allocated to the VirtualBox VM (named 'default') that gets created.  In VirtualBox, select the 'default' VM and under system, change the base memory from 1 GB to 4 GB.
+   **NOTE:** When setting up Docker Desktop for Windows, it is best to [use the WSL2 backend](https://docs.docker.com/docker-for-windows/wsl/). If not, you might run into compatability issues when trying to run Docker, especially if you have VirtualBox installed, which could require using the legacy [Docker Toolbox](https://github.com/docker/toolbox/releases).
 
    Inside the Container you can then run commands like these:
 
@@ -58,6 +57,13 @@ There are two alternative ways to set up to run the model:
    **Tip:** Eventually, you'll want to delete the Docker Image. Refer to the
    commands `docker image prune`, `docker image ls`, and `docker image rm`.
 
+   **Tip:** You can build your own Docker image instead of the one provided using these steps:
+
+   ```shell script
+   cd $YOUR_CODE_PROJECTS_DIR/wcEcoli  # or wherever you cloned the wcEcoli project to
+   cloud/build-containers.sh
+   ```
+
 2. **Python virtual environment:** Follow [Required development tools](dev-tools.md) to install the development tools including pyenv, gcc, make, and git, then follow [Creating the "pyenv" runtime environment](create-pyenv.md) to set up the Python runtime virtual environment for the model including binary libraries and Python packages.
 
    You can then run wcEcoli in this Python virtualenv.