adding labels!

Author: vsoch

README.md

@@ -14,17 +14,18 @@ OPTIONS:
               -w: non-production writable image (ext3)         
 
               Default is squashfs (recommended)
+
 ```
 
 ### Options
 
 **Image Format**
 
- - `default` (no arguments specified) gives you a squashfs (`*.simg`) image. This is a compressed, reliable, and read only format that is recommended for production images.
- - `-f` builds your image into a sandbox folder. This is ideal for development, as it will produce a working image in a folder on your system.
- - `-w` builds an older format (ext3) image (`*.img`) with `--writable`. This format is not recommended for production images as we have observed degradation of the images over time, and they tend to be upwards of 1.5x to 2x the size of squashfs.
+ - `squashfs` (no arguments specified) gives you a squashfs (`*.simg`) image. This is a compressed, reliable, and read only format that is recommended for production images.
+ - `sandbox` (`-f`) builds your image into a sandbox **folder**. This is ideal for development, as it will produce a working image in a folder on your system.
+ - `ext3` (`-w`) builds an older format (ext3) image (`*.img`). This format is not recommended for production images as we have observed degradation of the images over time, and they tend to be upwards of 1.5x to 2x the size of squashfs.
 
-Note that you are able to convert easily from a folder or ext3 image using Singularity 2.4, if your choice is to develop, making changes, and then finalize. This approach is **not** recommended - your changes are not recorded and thus the image not reproducible.
+Note that you are able to convert easily from a folder or ext3 image using Singularity 2.4. If your choice is to develop, making changes, and then finalize, this approach is **not** recommended - your changes are not recorded and thus the image not reproducible.
 
 **Mount Points**
 
@@ -51,21 +52,6 @@ Intermediate versions built on [Docker Hub](https://hub.docker.com/r/singularity
 
 ## Examples
 
-No need to download anything from this repository! Simply type:
-
-```
-docker run \
--v /var/run/docker.sock:/var/run/docker.sock \
--v D:\host\path\where\to\output\singularity\image:/output \
---privileged -t --rm \
-singularityware/docker2singularity:2.4 \
-ubuntu:14.04
-```
-
-Replace `D:\host\path\where\to\output\singularity\image` with a path on the host filesystem where your Singularity image will be created. Replace `ubuntu:14.04` with the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).
-
-`docker2singularity` uses the Docker daemon located on the host system. It will access the Docker image cache from the host system avoiding having to redownload images that are already present locally.
-
 ## Build a Squashfs Image
 Squashfs is the recommended image type, it is compressed and less prone to degradation over time. You don't need to specify anything special to create it:
 
@@ -75,14 +61,76 @@ docker run -v /var/run/docker.sock:/var/run/docker.sock \
 --privileged -t --rm \
 singularityware/docker2singularity:2.4
 ubuntu:14.04
-```
+
+Image Format: squashfs
+Inspected Size: 188 MB
+
+(1/10) Creating a build sandbox...
+(2/10) Exporting filesystem...
+(3/10) Creating labels...
+(4/10) Adding run script...
+(5/10) Setting ENV variables...
+(6/10) Adding mount points...
+(7/10) Fixing permissions...
+(8/10) Stopping and removing the container...
+(9/10) Building squashfs container...
+Building image from sandbox: /tmp/ubuntu_14.04-2017-09-13-3e51deeadc7b.build
+Building Singularity image...
+Singularity container built: /tmp/ubuntu_14.04-2017-09-13-3e51deeadc7b.simg
+Cleaning up...
+(10/10) Moving the image to the output folder...
+     62,591,007 100%  340.92MB/s    0:00:00 (xfr#1, to-chk=0/1)
+Final Size: 60MB
+```
+
+Notice how the image went from 188MB to 60MB? This reduction is even more impressive when we are dealing with
+very large images (e.g., ~3600 down to ~1800). A few notes on the inputs shown above that you should edit:
+
+ - `/host/path/change/me`: the path you want to have the final image reside. If you are on windows this might look like `D:\host\path\where\to\output\singularity\image`.
+ -`ubuntu:14.04`: the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).
+
+`docker2singularity` uses the Docker daemon located on the host system. It will access the Docker image cache from the host system avoiding having to redownload images that are already present locally.
+
 If you ever need to make changes, you can easily export the squashfs image into either a sandbox folder or ext3 (legacy) image, both of which have writable.
 
 ```
 sudo singularity build --sandbox sandbox/ production.simg
 sudo singularity build --writable ext3.img production.simg
 ```
 
+## Inspect Your Image
+New with `docker2singularity` 2.4, the labels for the container are available with `inspect`:
+
+```
+ singularity inspect ubuntu_14.04-2017-09-13-3e51deeadc7b.simg 
+{
+    "org.label-schema.singularity.build": "squashfs",
+    "org.label-schema.docker.version": "17.06.2-ce",
+    "org.label-schema.schema-version": "1.0",
+    "org.label-schema.singularity.build-type": "docker2singularity",
+    "org.label-schema.docker.id": "sha256:dea1945146b96542e6e20642830c78df702d524a113605a906397db1db022703",
+    "org.label-schema.build-date": "2017-10-28-17:19:18",
+    "org.label-schema.singularity.version": "2.4-dist",
+    "org.label-schema.docker.created": "2017-09-13"
+}
+```
+
+as is the runscript and environment
+
+```
+singularity inspect --json -e -r ubuntu_14.04-2017-09-13-3e51deeadc7b.simg 
+{
+    "data": {
+        "attributes": {
+            "environment": "# Custom environment shell code should follow\n\n",
+            "runscript": "#!/bin/sh\n/bin/bash $@\n"
+        },
+        "type": "container"
+    }
+}
+
+```
+
 ## Build a Sandbox Image
 A sandbox image is a folder that is ideal for development. You can view it on your desktop, cd inside and browse, and it works like a Singularity image. To create a sandbox, specify the `-f` flag:
 

docker2singularity.sh

@@ -134,9 +134,9 @@ mkdir -p $TMPDIR
 creation_date=`echo ${creation_date} | cut -c1-10`
 new_container_name=/tmp/$image_name-$creation_date-$container_id
 build_sandbox="${new_container_name}.build"
-echo "(1/9) Creating a build sandbox..."
+echo "(1/10) Creating a build sandbox..."
 mkdir -p ${build_sandbox}
-echo "(2/9) Exporting filesystem..."
+echo "(2/10) Exporting filesystem..."
 docker export $container_id >> $build_sandbox.tar
 singularity image.import $build_sandbox < $build_sandbox.tar
 docker inspect $container_id >> $build_sandbox/singularity.json
@@ -146,13 +146,46 @@ docker inspect $container_id >> $build_sandbox/singularity.json
 ### METADATA ###################################################################
 ################################################################################
 
+# Quiet Singularity debug output when adding labels
+SINGULARITY_MESSAGELEVEL=0
+export SINGULARITY_MESSAGELEVEL
+
 # For docker2singularity, installation is at /usr/local
-zcat /usr/local/libexec/singularity/bootstrap-scripts/environment.tar | ( cd $build_sandbox; tar -xf - >/dev/null)
+echo "(3/10) Creating labels..."
+libexec="/usr/local/libexec/singularity"
+zcat "${libexec}/bootstrap-scripts/environment.tar" | ( cd $build_sandbox; tar -xf - >/dev/null)
+LABELS=$(docker inspect --format='{{json .Config.Labels}}' $image)
+LABELFILE=$(printf "%q" "$build_sandbox/.singularity.d/labels.json")
+ADD_LABEL="${libexec}/python/helpers/json/add.py -f --file ${LABELFILE}"
+
+# Labels could be null
+if [ "${LABELS}" == "null" ]; then
+    LABELS="{}"
+fi
+
+# Extract some other "nice to know" metadata from docker
+SINGULARITY_version=`singularity --version`
+SINGULARITY_VERSION=$(printf "%q" "$SINGULARITY_version")
+DOCKER_VERSION=$(docker inspect --format='{{json .DockerVersion}}' $image)
+DOCKER_ID=$(docker inspect --format='{{json .Id}}' $image)
+
+# Add labels from Docker, then relevant to Singularity build
+echo $LABELS > $LABELFILE;
+eval $ADD_LABEL --key "org.label-schema.schema-version" --value "1.0"
+eval $ADD_LABEL --key "org.label-schema.singularity.build-type" --value "docker2singularity" 
+eval $ADD_LABEL --key "org.label-schema.singularity.build" --value "${image_format}" 
+eval $ADD_LABEL --key "org.label-schema.build-date" --value $(date +%Y-%m-%d-%H:%M:%S)
+eval $ADD_LABEL --key "org.label-schema.singularity.version" --value "${SINGULARITY_VERSION}"
+eval $ADD_LABEL --key "org.label-schema.docker.version" --value "${DOCKER_VERSION}"
+eval $ADD_LABEL --key "org.label-schema.docker.Created" --value "${creation_date}"
+eval $ADD_LABEL --key "org.label-schema.docker.Id" --value "${DOCKER_ID}"
+
+unset SINGULARITY_MESSAGELEVEL
 
 ################################################################################
 ### SINGULARITY RUN SCRIPT #####################################################
 ################################################################################
-echo "(3/9) Adding run script..."
+echo "(4/10) Adding run script..."
 CMD=$(docker inspect --format='{{json .Config.Cmd}}' $image)
 if [[ $CMD != [* ]]; then
     if [[ $CMD != "null" ]]; then
@@ -187,7 +220,8 @@ chmod +x $build_sandbox/.singularity.d/runscript;
 ################################################################################
 ### SINGULARITY ENVIRONMENT ####################################################
 ################################################################################
-echo "(4/9) Setting ENV variables..."
+
+echo "(5/10) Setting ENV variables..."
 docker run --rm --entrypoint="/usr/bin/env" $image > $TMPDIR/docker_environment
 # do not include HOME and HOSTNAME - they mess with local config
 sed -i '/^HOME/d' $TMPDIR/docker_environment
@@ -205,24 +239,24 @@ rm -rf $TMPDIR
 ### Permissions ################################################################
 ################################################################################
 if [ "${mount_points}" ] ; then
-    echo "(5/9) Adding mount points..."
+    echo "(6/10) Adding mount points..."
     mkdir -p "${build_sandbox}/${mount_points}"
 else
-    echo "(5/9) Skipping mount points..."
+    echo "(6/10) Skipping mount points..."
 fi 
 
 # making sure that any user can read and execute everything in the container
-echo "(6/9) Fixing permissions..."
+echo "(7/10) Fixing permissions..."
 
 find ${build_sandbox}/* -maxdepth 0 -not -path '${build_sandbox}/dev*' -not -path '${build_sandbox}/proc*' -not -path '${build_sandbox}/sys*' -exec chmod a+r -R '{}' \;
 find ${build_sandbox}/* -type f -or -type d -perm -u+x,o-x -not -path '${build_sandbox}/dev*' -not -path '${build_sandbox}/proc*' -not -path '${build_sandbox}/sys*' -exec chmod a+x '{}' \;
 
-echo "(7/9) Stopping and removing the container..."
+echo "(8/10) Stopping and removing the container..."
 docker stop $container_id >> /dev/null
 docker rm $container_id >> /dev/null
 
 # Build a final image from the sandbox
-echo "(8/9) Building ${image_format} container..."
+echo "(9/10) Building ${image_format} container..."
 if [ "$image_format" == "squashfs" ]; then
     new_container_name=${new_container_name}.simg
     singularity build ${new_container_name} $build_sandbox
@@ -233,7 +267,7 @@ else
     mv $build_sandbox $new_container_name
 fi
 
-echo "(9/9) Moving the image to the output folder..."
+echo "(10/10) Moving the image to the output folder..."
 finalsize=`du -shm $new_container_name | cut -f1`
 rsync --info=progress2 -a $new_container_name /output/
 echo "Final Size: ${finalsize}MB"