Merge pull request #92 from vsoch/vsoch-v2.5

Updates for 2.5 (with merge conflicts resolved)

Author: vsoch

.travis/before_install

@@ -6,7 +6,8 @@ sudo apt-get update && sudo apt-get install -y wget git build-essential squashfs
                                                     automake \
                                                     autoconf \
                                                     uuid-dev \
-                                                    libssl-dev
+                                                    libssl-dev \
+                                                    libarchive-dev
 
 
 sudo sed -i -e 's/^Defaults\tsecure_path.*$//' /etc/sudoers
@@ -31,9 +32,7 @@ which python
 
 conda install -y pandas scikit-learn pygments
 cd $TRAVIS_BUILD_DIR && sudo python -m pip install -r requirements.txt
-python setup.py sdist && python setup.py install
-pylint --version
+python python setup.py install
 
 # Install Singularity (development)
-cd /tmp && git clone -b development https://github.com/singularityware/singularity.git && cd singularity && ./autogen.sh && ./configure --prefix=/usr/local && make && sudo make install
-
+cd /tmp && git clone https://github.com/singularityware/singularity.git && cd singularity && ./autogen.sh && ./configure --prefix=/usr/local && make && sudo make install

.travis/script

@@ -1,3 +1,2 @@
 #!/bin/bash
-sudo apt-get install -y squashfs-tools
 python -m unittest discover -s $TRAVIS_BUILD_DIR/singularity/tests/ -p '[t|T]est*.py'

CHANGELOG.md

@@ -13,18 +13,18 @@ and changes prior to that are (unfortunately) done retrospectively. Critical ite
  - migration guidance (how to convert images?)
  - changed behaviour (recipe sections work differently)
 
-## [vxx](https://github.com/singularityware/singularity-python/tree/development) (development)
 
-**changed user experience**
- - to address this [sregisty issue](https://github.com/singularityhub/sregistry/issues/56) the push client does not always state that the upload is finished. In the case that an image is frozen (and 403 status) this message is misleading.
-**pull bug**
- - changing function to rename image after download to `shutil.move` to allow for cross device downloads.
+## [master](https://github.com/singularityware/singularity-python/tree/master)
+ - removed all client functionality in favor of using spython (2.5)
+ - build on Google will not extract file counts, etc.
 
 ## [v2.4.1](https://github.com/singularityware/singularity-python/releases/tag/v2.4.1) (v2.4.1)
 
-**changed behavior**
+ - changing function to rename image after download to `shutil.move` to allow for cross device downloads.
  - *apps extraction* files limited to those in bin and lib, as the response to the server was too large.
  - **updated tests** to handle secure build
+ - to address this [sregisty issue](https://github.com/singularityhub/sregistry/issues/56) the push client does not always state that the upload is finished. In the case that an image is frozen (and 403 status) this message is misleading.
+
 
 ## [v2.4](https://github.com/singularityware/singularity-python/releases/tag/v2.4) (v2.4)
 

README.md

@@ -2,7 +2,25 @@
 
 [![Build Status](https://travis-ci.org/singularityware/singularity-python.svg?branch=master)](https://travis-ci.org/singularityware/singularity-python)
 
-Singularity Python is a python module and command line tool for working with <a href="https://singularityware.github.io" target="_blank">Singularity</a> containers, specifically providing functions to visualize, package, and compare containers. Currently, most functions use Singularity on the command line, and some require sudo.
+Singularity Python is a python module and command line tool to provide helpers for working with <a href="https://singularityware.github.io" target="_blank">Singularity</a> containers, specifically providing functions to visualize, package, and compare containers. 
+
+ - If you are looking for the Singularity Python client to pull, build, and otherwise wrap Singularity functions, then please see the repository [singularity-cli](https://singularityhub.github.io/singularity-cli) for the [spython](https://pypi.org/project/spython/) module.
+ - If you are looking for local management and interaction with various storage locations (e.g., to pull and inspect containers in Singularity Hub, Registry, or other cloud resource) then please see the [Global Client](https://singularityhub.github.io/sregistry-cli) for the `sregistry` module.
+
+## Install
+You have the option to install only the dependencies that are needed for your functionality of interest.
+
+```bash
+# All
+pip install singularity
+
+# Metrics and analysis dependencies
+pip install singularity[metrics]
+
+# Building on Google Cloud
+pip install singularity[google[
+```
+
 
 We currently require Python > version 3 to use various timezone functions. If you are unable to install version 3.0, we provide a [Singularity.container](Singularity.container) for you to use instead. This is the recommended approach as some older versions of Python do not support generation of the timestamp. See the [installation docs](https://github.com/singularityware/singularity-python/wiki/Installation) for your different options.
 

Singularity.container

@@ -2,13 +2,7 @@ Bootstrap: docker
 From: continuumio/miniconda3
 
 %runscript
-exec /opt/conda/bin/sregistry "$@"
-
-%apprun shub
-exec /opt/conda/bin/shub "$@"
-
-%apprun sregistry
-exec /opt/conda/bin/sregistry "$@"
+exec /opt/conda/bin/python "$@"
 
 %labels
 maintainer [email protected]
@@ -21,10 +15,8 @@ apt-get update && apt-get install -y git
 
 # Install Singularity Python
 cd /opt
-git clone -b development https://www.github.com/vsoch/singularity-python
+git clone https://www.github.com/singularityware/singularity-python
 cd singularity-python
 /opt/conda/bin/pip install setuptools
 /opt/conda/bin/pip install -r requirements.txt
-/opt/conda/bin/pip install pyasn1==0.4.1
-/opt/conda/bin/python setup.py sdist
 /opt/conda/bin/python setup.py install

examples/classify/classify_image/count_files.py

@@ -3,25 +3,20 @@
 # This is an example of counting files using an image diff
 
 from singularity.analysis.classify import (
-    get_diff,
     file_counts,
     extension_counts
 )
 
-image_package = "python:3.6.0.img.zip"
-
-# The diff is a dict of folders --> files that differ between 
-# image and it's closest OS
-diff = get_diff(image_package=image_package)
+container = "ubuntu.simg"
 
 # Now we might be interested in counting different things
-readme_count = file_counts(diff=diff)
-copyright_count = file_counts(diff=diff,patterns=['copyright'])
-authors_count = file_counts(diff=diff,patterns=['authors','thanks','credit'])
-todo_count = file_counts(diff=diff,patterns=['todo'])
+readme_count = file_counts(container)
+copyright_count = file_counts(container, patterns=['copyright'])
+authors_count = file_counts(container, patterns=['authors','thanks','credit'])
+todo_count = file_counts(container, patterns=['todo'])
 
 # Or getting a complete dict of extensions
-extensions = extension_counts(diff=diff)
+extensions = extension_counts(container)
 
 # Return files instead of counts
-extensions = extension_counts(diff=diff,return_counts=False)
+extensions = extension_counts(container, return_counts=False)

examples/classify/classify_image/derive_tags.py

@@ -5,28 +5,19 @@
 import os
 os.environ['MESSAGELEVEL'] = 'CRITICAL'
 
-from singularity.analysis.classify import (
-    get_tags,
-    get_diff
-)
-
-image_package = "python:3.6.0.img.zip"
+from singularity.analysis.classify import get_tags
+from singularity.package import get_container_contents
 
 # The algorithm works as follows:
-#      1) first compare package to set of base OS (provided with shub)
-#      2) subtract the most similar os from image, leaving "custom" files
-#      3) organize custom files into dict based on folder name
-#      4) return search_folders as tags
+#      1) organize custom files into dict based on folder name
+#      2) return search_folders as tags
 
 # Default tags will be returned as software in "bin"
-tags = get_tags(image_package=image_package)
+tags = get_tags(container)
 
-# We can also get the raw "diff" between the image and it's base
-# which is usable in other functions (and we don't have to calc 
-# it again)
-diff = get_diff(image_package=image_package)
+# We can also get the raw list of flies
+file_list = get_container_contents(container)['all']
 
 # We can specify other folders of interest
 folders = ['init','init.d','bin','systemd']
-tags = get_tags(search_folders=folders,diff=diff)
-# Most similar OS found to be %s debian:7.11
+tags = get_tags(container, search_folders=folders)