Merge pull request #45 from vsoch/master

Updates to singularity python to support advanced shub functions

Author: vsoch

MANIFEST.in

@@ -1,4 +1,6 @@
 recursive-include singularity/templates *
 recursive-include singularity/static *
 recursive-include singularity/build *
+recursive-include singularity/analysis *
+recursive-include singularity/views *
 recursive-include singularity/testing *

README.md

@@ -32,62 +32,70 @@ After installation, you should be able to run `shub` on the command line, withou
 
       $  shub --help
 	usage: shub [-h] [--image IMAGE] [--images IMAGES] [--debug]
-		    [--outfolder OUTFOLDER] [--package] [--tree] [--simtree]
-		    [--subtract] [--simcalc] [--size SIZE]
+		    [--outfolder OUTFOLDER] [--package] [--os] [--oscalc] [--tags]
+		    [--tree] [--simtree] [--subtract] [--simcalc] [--size SIZE]
 
 	Singularity Hub command line tool
 
 	optional arguments:
 	  -h, --help            show this help message and exit
 	  --image IMAGE         full path to singularity image (for use with --package
 		                and --tree)
-	  --images IMAGES       images, separated by commas (for use with --simtree)
+	  --images IMAGES       images, separated by commas (for use with --simtree
+		                and --subtract
 	  --debug               use verbose logging to debug.
 	  --outfolder OUTFOLDER
 		                full path to folder for output, stays in tmp (or pwd)
 		                if not specified
 	  --package             package a singularity container for singularity hub
+	  --os                  estimate the operating system of your container.
+	  --oscalc              calculate similarity score for your container vs.
+		                docker library OS.
+	  --tags                retrieve list of software tags for an image, itself
+		                minus it's base
 	  --tree                view the guts of an singularity image (use --image)
 	  --simtree             view common guts between two images (use --images)
-	  --subtract            subtract one container image from the second to make
-                                a difference tree (use --images first,subtract)
+	  --subtract            subtract one container image from the second to make a
+		                difference tree (use --images first,subtract)
 	  --simcalc             calculate similarity (number) between images based on
 		                file contents.
 	  --size SIZE           If using Docker or shub image, you can change size
 		                (default is 1024)
 
 
 
-### Package your container
+### Classify your container
+Singularity python provides functions for quickly assessing the base operating system of your container, retrieving a list of software tags that are relevant when this base is subtracted, and getting similarity scores of your container to a library of base software.
 
-A package is a zipped up file that contains the image, the singularity runscript as `runscript`, a `VERSION` file, and a list of files `files.txt` and folders `folders.txt` in the container. 
+#### Estimate the OS
 
-![img/singularity-package.png](img/singularity-package.png)
+You can do this on the command line as follows:
 
-The example package can be [downloaded for inspection](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img.zip), as can the [image used to create it](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img). This is one of the drivers underlying [singularity hub](http://www.singularity-hub.org) (under development).
+      shub --image docker://python:latest --os
+      [sudo] password for vanessa
+      Most similar OS found to be  debian:7.11
+      debian:7.11
 
-  - **files.txt** and **folders.txt**: are simple text file lists with paths in the container, and this choice is currently done to provide the rawest form of the container contents. These files also are used to generate interactive visualizations, and calculate similarity between containers.
-  - **VERSION**: is a text file with one line, an md5 hash generated for the image when it was packaged. 
-  - **{{image}}.img**: is of course the original singularity container (usually a .img file)
+or to do this from within Python, see the [provided example](examples/classify_image/estimate_os.py). From within python, you can export the sudopw as the environmental variable "pancakes" and it won't need to ask. This is not ideal, but it's required for now since we are using Singularity to export the image. This will likely be changed soon.
 
-First, go to where you have some images:
 
-      ls
-      ubuntu.img
-      
+#### Get software tags
+Singularity Hub uses a simple algorithm to obtain a likely list of software that is important to your image. It assumes that (most) core installed software is in a folder called `bin`, and returns the list of these files with the estimated base image subtracted. You can do this as follows:
 
-You can now use the `shub` command line tool to package your image. Note that you must have [singularity installed](https://singularityware.lbl.gov/install-linux), and depending on the function you use, you will likely need to use sudo. We can use the `--package` argument to package our image:
 
-      shub --image ubuntu.img --package
+      shub --image docker://python:latest --tags
+    
 
+We also provide an [example for Python](examples/classify_image/derive_tags.py). If you do this programatically, you can change the folder(s) that are included, meaning that you could get a custom list of software (eg, libraries in `lib`, or python packages in `site-packages`).
 
-If no output folder is specified, the resulting image (named in the format `ubuntu.img.zip` will be output in the present working directory. You can also specify an output folder:
 
-      shub --image ubuntu.img --package --outfolder /tmp
+#### Compare to base OS
+If you want to get a complete list of scores for your image against a core set of latest [docker-os](singularity/analysis/packages/docker-os) images:
 
-For the package command, you will need to put in your password to grant sudo priviledges, as packaging requires using the singularity `export` functionality.
+      shub --image docker://python:latest --oscalc
 
-For more details, and a walkthrough with sample data, please see [examples/package_image](examples/package_image)
+
+or again see [this example](examples/classify_image/estimate_os.py) for doing this from within python.
 
 
 ### View the inside of a container
@@ -114,6 +122,14 @@ An [interactive demo](https://singularityware.github.io/singularity-python/examp
 
 ### Visualize Containers
 
+#### Container Similarity Clustering
+Do you have sets of containers or packages, and want to cluster them based on similarities?
+
+![examples/package_tree/docker-os.png](examples/package_tree/docker-os.png)
+
+We have examples for both deriving scores and producing plots like the above, see [examples/package_tree](examples/package_tree)
+
+
 #### Container Similarity Tree
 
 ![examples/similar_tree/simtree.png](examples/similar_tree/simtree.png)
@@ -146,7 +162,7 @@ An [interactive demo](https://singularityware.github.io/singularity-python/examp
 What files and folders differ between two containers? What does it look like if I subtract one image from the second? `shub` provides a command line tool to generate a visualization to do exactly this.
 
 
-      shub --subtract --images docker://ubuntu:latest.docker://centos:latest
+      shub --subtract --images docker://ubuntu:latest,docker://centos:latest
 
 As with `simtree`, this function supports both docker and singularity images as inputs.
 
@@ -159,7 +175,7 @@ An [interactive demo](https://singularityware.github.io/singularity-python/examp
 The same functions above can be used to show the exact similarities (intersect) and differences (files and/or folders unique to two images) between two images. You can get a data structure with this information as follows:
 
 
-      from shub.views import compare_containers
+      from singularity.analysis.compare import compare_containers
 
       image1 = 'ubuntu.img'
       image2 = 'centos.img'
@@ -168,6 +184,12 @@ The same functions above can be used to show the exact similarities (intersect)
       comparison = compare_containers(image1,image2,by=by)
 
 
+Note that you can also compare packages, or a container to a package:
+
+
+      def compare_containers(container1=None,container2=None,by=None,
+                             image_package1=None,image_package2=None)
+
 
 #### Calculate similarity of images
 
@@ -179,6 +201,38 @@ and the same applies for specification of Docker images, as in the previous exam
 
 
 
+### Package your container
+The driver of much of the above is the simple container package. A package is a zipped up file that contains the image, the singularity runscript as `runscript`, a `VERSION` file, and a list of files `files.txt` and folders `folders.txt` in the container. 
+
+![img/singularity-package.png](img/singularity-package.png)
+
+The example package can be [downloaded for inspection](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img.zip), as can the [image used to create it](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img). This is one of the drivers underlying [singularity hub](http://www.singularity-hub.org) (under development).
+
+  - **files.txt** and **folders.txt**: are simple text file lists with paths in the container, and this choice is currently done to provide the rawest form of the container contents. These files also are used to generate interactive visualizations, and calculate similarity between containers.
+  - **VERSION**: is a text file with one line, an md5 hash generated for the image when it was packaged. 
+  - **{{image}}.img**: is of course the original singularity container (usually a .img file)
+
+First, go to where you have some images:
+
+      ls
+      ubuntu.img
+      
+
+You can now use the `shub` command line tool to package your image. Note that you must have [singularity installed](https://singularityware.lbl.gov/install-linux), and depending on the function you use, you will likely need to use sudo. We can use the `--package` argument to package our image:
+
+      shub --image ubuntu.img --package
+
+
+If no output folder is specified, the resulting image (named in the format `ubuntu.img.zip` will be output in the present working directory. You can also specify an output folder:
+
+      shub --image ubuntu.img --package --outfolder /tmp
+
+For the package command, you will need to put in your password to grant sudo priviledges, as packaging requires using the singularity `export` functionality.
+
+For more details, and a walkthrough with sample data, please see [examples/package_image](examples/package_image)
+
+
+
 ### Build your container
 More information coming soon.
 

examples/classify_image/count_files.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+
+# This is an example of counting files using an image diff
+
+from singularity.analysis.classify import (
+    get_diff,
+    file_counts,
+    extension_counts
+)
+
+image_package = "python:3.6.0.img.zip"
+
+# The diff is a dict of folders --> files that differ between 
+# image and it's closest OS
+diff = get_diff(image_package=image_package)
+
+# Now we might be interested in counting different things
+readme_count = file_counts(diff=diff)
+copyright_count = file_counts(diff=diff,patterns=['copyright'])
+authors_count = file_counts(diff=diff,patterns=['authors','thanks','credit'])
+todo_count = file_counts(diff=diff,patterns=['todo'])
+
+# Or getting a complete dict of extensions
+extensions = extension_counts(diff=diff)
+
+# Return files instead of counts
+extensions = extension_counts(diff=diff,return_counts=False)

examples/classify_image/derive_tags.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+
+# This is an example of generating image packages from within python
+
+import os
+os.environ['MESSAGELEVEL'] = 'CRITICAL'
+
+from singularity.analysis.classify import get_tags
+
+image_package = "python:3.6.0.img.zip"
+
+# The algorithm works as follows:
+#      1) first compare package to set of base OS (provided with shub)
+#      2) subtract the most similar os from image, leaving "custom" files
+#      3) organize custom files into dict based on folder name
+#      4) return search_folders as tags
+
+# Default tags will be returned as software in "bin"
+tags = get_tags(image_package=image_package)
+
+# Most similar OS found to be %s debian:7.11

examples/classify_image/estimate_os.py

@@ -0,0 +1,14 @@
+#!/usr/bin/env python
+
+# This is an example of generating image packages from within python
+
+from singularity.analysis.classify import estimate_os
+
+image_package = "python:3.6.0.img.zip"
+
+# We can obtain the estimated os (top match)
+estimated_os = estimate_os(image_package=image_package)
+# Most similar OS found to be %s debian:7.11
+
+# We can also get the whole list and values
+os_similarity = estimate_os(image_package=image_package,return_top=False)

examples/classify_image/python:3.6.0.img.zip

@@ -0,0 +1,87 @@
+#!/usr/bin/env python
+
+# This is an example of generating image packages from within python. The docker image packages (top) will
+# retrieve all of docker official library (as of 1/2017), and the os specific images (bottom) will 
+# retrieve base os (both included with singularity-python)
+
+import os
+import shutil
+from singularity.cli import get_image
+from singularity.package import package
+
+# Save packages here
+output_base = '/home/vanessa/Documents/Dropbox/Code/singularity/singularity-python/examples/package_image/packages'
+
+## DOCKER LIBBRARY IMAGES
+# https://github.com/docker-library/official-images/tree/master/library
+images = ['aerospike:3.10.1.1','alpine:3.5','amazonlinux:2016.09','arangodb:3.1.7','backdrop:1.5.2',
+          'bash:4.4.5','bonita:7.3.3','buildpack-deps:jessie',
+          'busybox:1.26.1-glibc','cassandra:3.9','celery:3.1.3','centos:7',
+          'chronograf:0.13.0','cirros:0.3.4','clearlinux:base','clojure:lein-2.7.1',
+          'composer:1.3.0','consul:0.7.2','couchbase:enterprise-4.5.1','couchdb:1.6.1','crate:1.0.1',
+          'debian:8.6','docker:1.13.0-rc5','drupal:8.2.5-apache','eclipse-mosquitto:1.4.10','eggdrop:1.8.0',
+          'elasticsearch:5.1.1','elixir:1.4.0','erlang:19.2','fedora:25','fsharp:4.0.0.4','gazebo:libgazebo7',
+          'gcc:4.9.4','ghost:0.11.3','golang:1.6.4','haproxy:1.7.1','haskell:8.0.1','hello-world:latest',
+          'httpd:2.2.31','hylang:0.11.1','ibmjava:8-jre','influxdb:1.1.1','iojs:3.3.0','irssi:1.0.0',
+          'jenkins:2.32.1','jetty:9.3.15','joomla:3.6.5','jruby:9.1.6.0-jre','julia:0.5.0',
+          'kaazing-gateway:5.3.2','kapacitor:1.1.1','kibana:5.1.1','known:0.9.2','kong:0.9.7','lightstreamer:6.0.3',
+          'logstash:5.1.1','mageia:5','mariadb:10.1.20','maven:3.3.9-jdk-8','memcached:1.4.33','mongo:3.4.1',
+          'mongo-express:0.34.0','mono:4.6.2.16','mysql:5.7.17','nats:0.9.6',
+          'nats-streaming:0.3.6','neo4j:3.1.0','neurodebian:nd70','nextcloud:10.0.0','nginx:1.11.8',
+          'node:7.4.0','notary:server-0.5.0','notary:signer-0.5.0','nuxeo-LTS-2016','odoo:10.0','openjdk:8u111-jdk',
+          'opensuse:42.2','oraclelinux:7.3','orientdb:2.2.14','owncloud:9.1.3-apache','percona:5.7.16',
+          'perl:5.24.0','photon:1.0','php:7.1.0-cli','php-zendserver:9.0.1-php7',
+          'piwik:3.0.0','plone:5.0.6','postgres:9.6.1','pypy:3-5.5.0-alpha','python:3.6.0',
+          'r-base:3.3.2','rabbitmq:3.6.6','rakudo-star:2016.11','redis:3.0.7','redmine:3.1.7','registry:2.5.1',
+          'rethinkdb:2.3.5','rocket.chat:0.48.2','ros:kinetic-ros-base','ruby:2.4.0','sentry:8.12.0',
+          'solr:6.3.0','sonarqube:6.2','sourcemage:0.62','spiped:1.5.0','storm:1.0.2','swarm:1.2.5',
+          'telegraf:1.1.2','thrift:0.9.3','tomcat:8.0.39-jre7',
+          'tomee:8-jdk-7.0.1-webprofile','traefik:v1.1.2','ubuntu:16.04','vault:0.6.4',
+          'websphere-liberty:javaee7','wordpress:4.7.0','zookeeper:3.4.9']
+
+image_type = 'docker'
+
+
+## OS IMAGES (provided via official docker library)
+images = ['alpine:3.1','alpine:3.2','alpine:3.3','alpine:3.4','alpine:3.5',
+          'busybox:1.26.1-glibc','busybox:1.26.1-musl','busybox:1.26.1-uclibc',
+          'amazonlinux:2016:09',
+          'centos:7','centos:6','centos:5',
+          'cirros:0.3,4','cirros:0.3,3',
+          'clearlinux:latest',
+          'crux:3.1',
+          'debian:8.6','debian:sid','debian:stretch','debian:7.11', # 8.6 is jessie, 7.11 is wheezy
+          'fedora:25','fedora:24','fedora:23','fedora:22','fedora:21','fedora:20',
+          'mageia:5',
+          'opensuse:42.2','opensuse:42.1','opensuse:13.2','opensuse:tumbleweed' #  42.2 is leap, 13.2 harlequin
+          'oraclelinux:7.3','oraclelinux:7.2','oraclelinux:7.1','oraclelinux:7.0','oraclelinux:6.8',
+          'oraclelinux:6.7','oraclelinux:6.6','oraclelinux:5.11',
+          'photon:1.0',
+          'sourcemage:0.62',
+          'swarm:1.2.6-rc1',
+          'ubuntu:12.04.5','ubuntu:14.04.5','ubuntu:16.04','ubuntu:16.10','ubuntu:17.04']
+
+image_type = 'os'
+
+# You will need to export your sudopw to an environment variable called pancakes for it to not ask you :)
+os.environ['pancakes'] = 'yoursecretpass'
+
+# We will make subdirectories in package folder
+output_folder = "%s/%s" %(output_base,image_type)
+
+if not os.path.exists(output_folder):
+   os.mkdir(output_folder)
+
+for name in images:
+    docker_image = "docker://%s" %(name)
+    image = get_image(docker_image)
+    package_name = "%s/%s.zip" %(output_folder,os.path.basename(image))
+    if not os.path.exists(package_name):
+        image_package = package(image_path=image,
+                                output_folder=output_folder,
+                                remove_image=True,
+                                runscript=True,
+                                software=True)
+    tmpfolder = os.path.dirname(image)
+    shutil.rmtree(tmpfolder)
+