fixing up script to allow for multiple formats and correct metadata creation

vsoch · vsoch · commit 9f3e9cb349b2 · 2017-10-28T08:30:26.000-07:00
diff --git a/README.md b/README.md
@@ -1,6 +1,43 @@
 # `docker2singularity`
 
-Are you developing Docker images and you would like to run them on an HPC cluster supporting [Singularity](http://singularity.lbl.gov)? Are you working on Mac or Windows with no easy access to a Linux machine? If the pull, import, and general commands to [work with docker images provided by Singularity](http://singularity.lbl.gov/docs-docker) natively do not fit your needs, `docker2singularity` is an alternative way to generate Singularity images. This particular branch is intended for Singularity 2.4, which builds a squashfs image. If you want a legacy version to build ext3 images, see the following other branches:
+Are you developing Docker images and you would like to run them on an HPC cluster supporting [Singularity](http://singularity.lbl.gov)? Are you working on Mac or Windows with no easy access to a Linux machine? If the pull, import, and general commands to [work with docker images provided by Singularity](http://singularity.lbl.gov/docs-docker) natively do not fit your needs, `docker2singularity` is an alternative way to generate Singularity images. This particular branch is intended for Singularity 2.4, which gives you a selection of image formats to build.
+
+## Usage
+
+```
+docker run docker2singularity
+USAGE: docker2singularity [-m "/mount_point1 /mount_point2"] [options] docker_image_name
+OPTIONS:
+
+          Image Format
+              -f: build development sandbox (folder)
+              -w: non-production writable image (ext3)         
+
+              Default is squashfs (recommended)
+```
+
+### Options
+
+**Image Format**
+
+ - `default` (no arguments specified) gives you a squashfs (`*.simg`) image. This is a compressed, reliable, and read only format that is recommended for production images.
+ - `-f` builds your image into a sandbox folder. This is ideal for development, as it will produce a working image in a folder on your system.
+ - `-w` builds an older format (ext3) image (`*.img`) with `--writable`. This format is not recommended for production images as we have observed degradation of the images over time, and they tend to be upwards of 1.5x to 2x the size of squashfs.
+
+Note that you are able to convert easily from a folder or ext3 image using Singularity 2.4, if your choice is to develop, making changes, and then finalize. This approach is **not** recommended - your changes are not recorded and thus the image not reproducible.
+
+**Mount Points**
+
+ - `-m` specify one or more mount points to create in the image.
+
+**Image Name**
+
+The last argument (without a letter) is the name of the docker image, as you would specify to run with Docker (e.g., `docker run ubuntu:latest`)
+
+
+## Legacy
+
+If you want a legacy version, see the following other branches:
 
  - [v2.3](https://github.com/singularityware/docker2singularity/tree/v2.3): Version 2.3 of Singularity. The image format is ext3.
  - [v2.4](https://github.com/singularityware/docker2singularity/tree/v2.4): Version 2.4 of Singularity. The default image format is squashfs.
@@ -12,7 +49,7 @@ Intermediate versions built on [Docker Hub](https://hub.docker.com/r/singularity
  - Docker (native Linux or Docker for Mac or Docker for Windows) - to create the Singularity image.
  - Singularity >= 2.1 - to run the Singularity image (**versions 2.0 and older are not supported!**). Note that if running a 2.4 image using earlier versions, not all (later developed) features may be available.
 
-## Usage
+## Examples
 
 No need to download anything from this repository! Simply type:
 
@@ -29,6 +66,58 @@ Replace `D:\host\path\where\to\output\singularity\image` with a path on the host
 
 `docker2singularity` uses the Docker daemon located on the host system. It will access the Docker image cache from the host system avoiding having to redownload images that are already present locally.
 
+## Build a Squashfs Image
+Squashfs is the recommended image type, it is compressed and less prone to degradation over time. You don't need to specify anything special to create it:
+
+```
+docker run -v /var/run/docker.sock:/var/run/docker.sock \
+-v /host/path/change/me:/output \
+--privileged -t --rm \
+singularityware/docker2singularity:2.4
+ubuntu:14.04
+```
+If you ever need to make changes, you can easily export the squashfs image into either a sandbox folder or ext3 (legacy) image, both of which have writable.
+
+```
+sudo singularity build --sandbox sandbox/ production.simg
+sudo singularity build --writable ext3.img production.simg
+```
+
+## Build a Sandbox Image
+A sandbox image is a folder that is ideal for development. You can view it on your desktop, cd inside and browse, and it works like a Singularity image. To create a sandbox, specify the `-f` flag:
+
+```
+docker run -v /var/run/docker.sock:/var/run/docker.sock \
+-v /host/path/change/me:/output \
+--privileged -t --rm \
+singularityware/docker2singularity:2.4 \
+-f \
+ubuntu:14.04
+```
+Importantly, you can use `--writable`, and if needed, you can convert a sandbox folder into a production image:
+
+```
+sudo singularity build sandbox/ production.simg
+```
+
+## Build a Legacy (ext3) Image
+You can build a legacy ext3 image (with `--writable`) with the `-w` flag. This is an older image format that is more prone to degradation over time, and (building) may not be supported for future versions of the software.
+
+```
+docker run -v /var/run/docker.sock:/var/run/docker.sock \
+-v /host/path/change/me:/output \
+--privileged -t --rm \
+singularityware/docker2singularity:2.4 \
+-w \
+ubuntu:14.04
+```
+You can also use `--writable` and convert an ext3 image into a production image:
+
+```
+sudo singularity build ext3.img production.simg
+```
+
+
 ## Tips for making Docker images compatible with Singularity
 
  - Define all environmental variables using the `ENV` instruction set. Do not rely on `.bashrc`, `.profile`, etc.
@@ -68,5 +157,6 @@ If you are getting `WARNING: Non existant bind point (directory) in container: '
      -m "/shared_fs /custom_mountpoint2" \
      ubuntu:14.04
 
+
 ## Acknowledgements
 This work is heavily based on the `docker2singularity` work done by [vsoch](https://github.com/vsoch) and [gmkurtzer](https://github.com/gmkurtzer). Hopefully most of the conversion code will be merged into Singularity in the future making this container even leaner!
diff --git a/docker2singularity.sh b/docker2singularity.sh
@@ -36,22 +36,36 @@
 set -o errexit
 set -o nounset
 
-USAGE="Usage: docker2singularity [-m \"/mount_point1 /mount_point2\"] docker_image_name"
+USAGE="USAGE: docker2singularity [-m \"/mount_point1 /mount_point2\"] [options] docker_image_name"
 
 # --- Option processing --------------------------------------------
 if [ $# == 0 ] ; then
     echo $USAGE
+    echo "OPTIONS:
+
+          Image Format
+              -f: build development sandbox (folder)
+              -w: non-production writable image (ext3)         
+
+              Default is squashfs (recommended)
+              "
+
     exit 1;
 fi
 
 mount_points="/oasis /projects /scratch /local-scratch /work /home1 /corral-repl /corral-tacc /beegfs /share/PI /extra /data /oak"
-while getopts ':hm:' option; do
+image_format="squashfs"
+while getopts ':hm:wf' option; do
   case "$option" in
     h) echo "$USAGE"
-       exit
+       exit 0
        ;;
     m) mount_points=$OPTARG
        ;;
+    f) image_format="sandbox"
+       ;;
+    w) image_format="writable"
+       ;;
     :) printf "missing argument for -%s\n" "$OPTARG" >&2
        echo "$usage" >&2
        exit 1
@@ -66,6 +80,9 @@ shift $((OPTIND - 1))
 
 image=$1
 
+echo ""
+echo "Image Format: ${image_format}"
+
 ################################################################################
 ### CONTAINER RUNNING ID #######################################################
 ################################################################################
@@ -103,14 +120,10 @@ creation_date=`docker inspect --format="{{.Created}}" $image`
 ################################################################################
 
 size=`docker inspect --format="{{.Size}}" $image`
-# convert size in MB (it seems too small for singularity containers ...?). Add 1MB to round up (minimum).
+# convert size in MB
 size=`echo $(($size/1000000+1))`
-# adding half of the container size seems to work (do not know why exactly...?)
-# I think it would be Ok by adding 1/3 of the size.
-size=`echo $(($size+$size/2))`
-
-echo "Size: $size MB for the singularity container, although not needed for squashfs."
-
+echo "Inspected Size: $size MB"
+echo ""
 
 ################################################################################
 ### IMAGE CREATION #############################################################
@@ -119,7 +132,7 @@ TMPDIR=$(mktemp -u -d)
 mkdir -p $TMPDIR
 
 creation_date=`echo ${creation_date} | cut -c1-10`
-new_container_name=/tmp/$image_name-$creation_date-$container_id.simg
+new_container_name=/tmp/$image_name-$creation_date-$container_id
 build_sandbox="${new_container_name}.build"
 echo "(1/9) Creating a build sandbox..."
 mkdir -p ${build_sandbox}
@@ -129,6 +142,13 @@ singularity image.import $build_sandbox < $build_sandbox.tar
 docker inspect $container_id >> $build_sandbox/singularity.json
 
 
+################################################################################
+### METADATA ###################################################################
+################################################################################
+
+# For docker2singularity, installation is at /usr/local
+zcat /usr/local/libexec/singularity/bootstrap-scripts/environment.tar | ( cd $build_sandbox; tar -xf - >/dev/null)
+
 ################################################################################
 ### SINGULARITY RUN SCRIPT #####################################################
 ################################################################################
@@ -152,16 +172,16 @@ fi
 # Remove quotes, commas, and braces
 ENTRYPOINT=`echo "${ENTRYPOINT//\"/}" | sed 's/\[//g' | sed 's/\]//g' | sed 's/,/ /g'`
 
-echo '#!/bin/sh' > $build_sandbox/singularity
+echo '#!/bin/sh' > $build_sandbox/.singularity.d/runscript
 if [[ $ENTRYPOINT != "null" ]]; then
-    echo $ENTRYPOINT '$@' >> $build_sandbox/singularity;
+    echo $ENTRYPOINT '$@' >> $build_sandbox/.singularity.d/runscript;
 else
     if [[ $CMD != "null" ]]; then
-        echo $CMD '$@' >> $build_sandbox/singularity;
+        echo $CMD '$@' >> $build_sandbox/.singularity.d/runscript;
     fi
 fi
 
-chmod +x $build_sandbox/singularity
+chmod +x $build_sandbox/.singularity.d/runscript;
 
 
 ################################################################################
@@ -176,8 +196,8 @@ sed -i 's/^/export /' $TMPDIR/docker_environment
 # add quotes around the variable names
 sed -i 's/=/="/' $TMPDIR/docker_environment
 sed -i 's/$/"/' $TMPDIR/docker_environment
-cp $TMPDIR/docker_environment $build_sandbox/docker_environment
-echo '. /docker_environment' >> $build_sandbox/environment
+cp $TMPDIR/docker_environment $build_sandbox/.singularity.d/env/10-docker.sh
+chmod +x $build_sandbox/.singularity.d/env/10-docker.sh;
 rm -rf $TMPDIR
 
 
@@ -198,12 +218,22 @@ find ${build_sandbox}/* -maxdepth 0 -not -path '${build_sandbox}/dev*' -not -pat
 find ${build_sandbox}/* -type f -or -type d -perm -u+x,o-x -not -path '${build_sandbox}/dev*' -not -path '${build_sandbox}/proc*' -not -path '${build_sandbox}/sys*' -exec chmod a+x '{}' \;
 
 echo "(7/9) Stopping and removing the container..."
-docker stop $container_id
-docker rm $container_id
+docker stop $container_id >> /dev/null
+docker rm $container_id >> /dev/null
 
 # Build a final image from the sandbox
-echo "(8/9) Building..."
-singularity build $new_container_name $build_sandbox
+echo "(8/9) Building ${image_format} container..."
+if [ "$image_format" == "squashfs" ]; then
+    new_container_name=${new_container_name}.simg
+    singularity build ${new_container_name} $build_sandbox
+elif [ "$image_format" == "writable" ]; then
+    new_container_name=${new_container_name}.img    
+    singularity build --writable ${new_container_name} $build_sandbox
+else
+    mv $build_sandbox $new_container_name
+fi
 
 echo "(9/9) Moving the image to the output folder..."
-rsync --info=progress2 $new_container_name /output/
+finalsize=`du -shm $new_container_name | cut -f1`
+rsync --info=progress2 -a $new_container_name /output/
+echo "Final Size: ${finalsize}MB"
