|
1 | | -# Singularity Python |
2 | | - |
3 | | -[](https://travis-ci.org/singularityware/singularity-python) |
4 | | - |
5 | | -This is a python command line tool for working with [Singularity](singularityware.github.io) containers, specifically providing functions to visualize, package, and containers. Currently, most functions use Singularity on the commnd line, and require sudo, and this is expected to change when the software is updated to obtain image folders and files without an export that uses `Singularity export`. |
6 | | - |
7 | | -The Singularity-Python code is licensed under the MIT open source license, which is a highly permissive license that places few limits upon reuse. This ensures that the code will be usable by the greatest number of researchers, in both academia and industry. |
8 | | - |
9 | | - |
10 | | -## Installation (current) |
11 | | - |
12 | | - pip install singularity |
13 | | - |
14 | | - |
15 | | -## Installation (dev) |
16 | | - |
17 | | - pip install git+git://github.com/singularityware/singularity-python.git |
18 | | - |
19 | | - |
20 | | -## Quick Start |
21 | | - |
22 | | -Installation will place an executable, `shub` in your bin folder. |
23 | | - |
24 | | -### Test your installation |
25 | | - |
26 | | -After installation, you should be able to run `shub` on the command line, without any input args, to see if you have Singularity installed (and test the package): |
27 | | - |
28 | | - |
29 | | - $ shub |
30 | | - Please specify one or more containers with --image(s) |
31 | | - |
32 | | - |
33 | | - $ shub --help |
34 | | - usage: shub [-h] [--image IMAGE] [--images IMAGES] [--debug] |
35 | | - [--outfolder OUTFOLDER] [--package] [--os] [--oscalc] [--tags] |
36 | | - [--tree] [--simtree] [--subtract] [--simcalc] [--size SIZE] |
37 | | - |
38 | | - Singularity Hub command line tool |
39 | | - |
40 | | - optional arguments: |
41 | | - -h, --help show this help message and exit |
42 | | - --image IMAGE full path to singularity image (for use with --package |
43 | | - and --tree) |
44 | | - --images IMAGES images, separated by commas (for use with --simtree |
45 | | - and --subtract |
46 | | - --debug use verbose logging to debug. |
47 | | - --outfolder OUTFOLDER |
48 | | - full path to folder for output, stays in tmp (or pwd) |
49 | | - if not specified |
50 | | - --package package a singularity container for singularity hub |
51 | | - --os estimate the operating system of your container. |
52 | | - --oscalc calculate similarity score for your container vs. |
53 | | - docker library OS. |
54 | | - --osplot plot similarity scores for your container vs. docker |
55 | | - library OS. |
56 | | - --tags retrieve list of software tags for an image, itself |
57 | | - minus it's base |
58 | | - --tree view the guts of an singularity image (use --image) |
59 | | - --simtree view common guts between two images (use --images) |
60 | | - --subtract subtract one container image from the second to make a |
61 | | - difference tree (use --images first,subtract) |
62 | | - --simcalc calculate similarity (number) between images based on |
63 | | - file contents. |
64 | | - --size SIZE If using Docker or shub image, you can change size |
65 | | - (default is 1024) |
66 | | - |
67 | | - |
68 | | - |
69 | | -### Classify your container |
70 | | -Singularity python provides functions for quickly assessing the base operating system of your container, retrieving a list of software tags that are relevant when this base is subtracted, and getting similarity scores of your container to a library of base software. |
71 | | - |
72 | | -#### Estimate the OS |
73 | | - |
74 | | -You can do this on the command line as follows: |
75 | | - |
76 | | - shub --image docker://python:latest --os |
77 | | - [sudo] password for vanessa |
78 | | - Most similar OS found to be debian:7.11 |
79 | | - debian:7.11 |
80 | | - |
81 | | -or to do this from within Python, see the [provided example](examples/classify_image/estimate_os.py). From within python, you can export the sudopw as the environmental variable "pancakes" and it won't need to ask. This is not ideal, but it's required for now since we are using Singularity to export the image. This will likely be changed soon. |
82 | | - |
83 | | - |
84 | | -#### Get software tags |
85 | | -Singularity Hub uses a simple algorithm to obtain a likely list of software that is important to your image. It assumes that (most) core installed software is in a folder called `bin`, and returns the list of these files with the estimated base image subtracted. You can do this as follows: |
86 | | - |
87 | | - |
88 | | - shub --image docker://python:latest --tags |
89 | | - |
90 | | - |
91 | | -We also provide an [example for Python](examples/classify_image/derive_tags.py). If you do this programatically, you can change the folder(s) that are included, meaning that you could get a custom list of software (eg, libraries in `lib`, or python packages in `site-packages`). |
92 | | - |
93 | | - |
94 | | -#### Compare to base OS |
95 | | -If you want to get a complete list of scores for your image against a core set of latest [docker-os](singularity/analysis/packages/docker-os) images: |
96 | | - |
97 | | - shub --image docker://python:latest --oscalc |
98 | | - |
99 | | -or again see [this example](examples/classify_image/estimate_os.py) for doing this from within python. |
100 | | - |
101 | | -You can also generate a [dynamic plot](https://singularityware.github.io/singularity-python/examples/classify_image/) for this data: |
102 | | - |
103 | | - shub --image docker://python:latest --osplot |
104 | | - |
105 | | - |
106 | | - |
107 | | -### View the inside of a container |
108 | | -What's inside that container? Right now, the main way to answer this question is to do some equivalent of ssh. shub provides a command line function for rendering a view to (immediately) show the contents of an image (folders and files) in your web browser. **Important** the browser will open, but you will need to use your password to use Singularity on the command line: |
109 | | - |
110 | | - |
111 | | -#### Docker Image |
112 | | -shub will render the guts of any Docker image, even if it's not currently on your system. You don't need |
113 | | - |
114 | | - |
115 | | - shub --image docker://ubuntu:latest --tree |
116 | | - |
117 | | - |
118 | | -#### Singularity Package or Image |
119 | | - |
120 | | - shub --image ubuntu.img --tree |
121 | | - |
122 | | - |
123 | | -This will open up something that looks like this: |
124 | | - |
125 | | - |
126 | | -An [interactive demo](https://singularityware.github.io/singularity-python/examples/container_tree) is also available, and see the [example](examples/container_tree) for updates. |
127 | | - |
128 | | - |
129 | | -### Visualize Containers |
130 | | - |
131 | | -#### Container Similarity Clustering |
132 | | -Do you have sets of containers or packages, and want to cluster them based on similarities? |
133 | | - |
134 | | - |
135 | | - |
136 | | -We have examples for both deriving scores and producing plots like the above, see [examples/package_tree](examples/package_tree) |
137 | | - |
138 | | - |
139 | | -#### Container Similarity Tree |
140 | | - |
141 | | - |
142 | | - |
143 | | -What do two containers have in common, in terms of files and folders? shub provides a command line function for rendering a view to (immediately) show the similarity between to container images: |
144 | | - |
145 | | - |
146 | | - shub --images centos.img,ubuntu.img --simtree --debug |
147 | | - |
148 | | - |
149 | | -We can also compare a local image to a Docker image: |
150 | | - |
151 | | - |
152 | | - shub --images docker://ubuntu:latest,/home/vanessa/Desktop/ubuntu.img --simtree --debug |
153 | | - |
154 | | - |
155 | | -Or two Docker images: |
156 | | - |
157 | | - |
158 | | - shub --images docker://ubuntu:latest,docker://centos:latest --simtree |
159 | | - |
160 | | - |
161 | | -If you need output for any of the following, you can add the `--debug` argument. Note that when generating docker comparisons, the back end is obtaining the layers, creating the images, importing and packaging, so the result is not instantanous. |
162 | | - |
163 | | - |
164 | | -An [interactive demo](https://singularityware.github.io/singularity-python/examples/similar_tree/) is also available. |
165 | | - |
166 | | - |
167 | | -#### Container Difference Tree |
168 | | -What files and folders differ between two containers? What does it look like if I subtract one image from the second? `shub` provides a command line tool to generate a visualization to do exactly this. |
169 | | - |
170 | | - |
171 | | - shub --subtract --images docker://ubuntu:latest,docker://centos:latest |
172 | | - |
173 | | -As with `simtree`, this function supports both docker and singularity images as inputs. |
174 | | - |
175 | | - |
176 | | - |
177 | | -An [interactive demo](https://singularityware.github.io/singularity-python/examples/difference_tree/) is also available. |
178 | | - |
179 | | - |
180 | | -### Compare Containers |
181 | | -The same functions above can be used to show the exact similarities (intersect) and differences (files and/or folders unique to two images) between two images. You can get a data structure with this information as follows: |
182 | | - |
183 | | - |
184 | | - from singularity.analysis.compare import compare_containers |
185 | | - |
186 | | - image1 = 'ubuntu.img' |
187 | | - image2 = 'centos.img' |
188 | | - by = "files.txt" # can also be "folders.txt", or a list with both |
189 | | - |
190 | | - comparison = compare_containers(image1,image2,by=by) |
191 | | - |
192 | | - |
193 | | -Note that you can also compare packages, or a container to a package: |
194 | | - |
195 | | - |
196 | | - def compare_containers(container1=None,container2=None,by=None, |
197 | | - image_package1=None,image_package2=None) |
198 | | - |
199 | | - |
200 | | -#### Calculate similarity of images |
201 | | - |
202 | | -We can calculate similarity of images based on the file content inside. For an example, see [examples/calculate_similarity](examples/calculate_similarity). We can compare two local images as follows: |
203 | | - |
204 | | - $ shub --images /home/vanessa/Desktop/ubuntu.img,/home/vanessa/Desktop/ubuntu.img --simcalc |
205 | | - |
206 | | -and the same applies for specification of Docker images, as in the previous example. Note that we are specifying `images` for the argument instead of `image`, and it's a single string of image names separated by a comma. |
207 | | - |
208 | | - |
209 | | - |
210 | | -### Package your container |
211 | | -The driver of much of the above is the simple container package. A package is a zipped up file that contains the image, the singularity runscript as `runscript`, a `VERSION` file, and a list of files `files.txt` and folders `folders.txt` in the container. |
212 | | - |
213 | | - |
214 | | - |
215 | | -The example package can be [downloaded for inspection](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img.zip), as can the [image used to create it](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img). This is one of the drivers underlying [singularity hub](http://www.singularity-hub.org) (under development). |
216 | | - |
217 | | - - **files.txt** and **folders.txt**: are simple text file lists with paths in the container, and this choice is currently done to provide the rawest form of the container contents. These files also are used to generate interactive visualizations, and calculate similarity between containers. |
218 | | - - **VERSION**: is a text file with one line, an md5 hash generated for the image when it was packaged. |
219 | | - - **{{image}}.img**: is of course the original singularity container (usually a .img file) |
220 | | - |
221 | | -First, go to where you have some images: |
222 | | - |
223 | | - ls |
224 | | - ubuntu.img |
225 | | - |
226 | | - |
227 | | -You can now use the `shub` command line tool to package your image. Note that you must have [singularity installed](https://singularityware.lbl.gov/install-linux), and depending on the function you use, you will likely need to use sudo. We can use the `--package` argument to package our image: |
228 | | - |
229 | | - shub --image ubuntu.img --package |
230 | | - |
231 | | - |
232 | | -If no output folder is specified, the resulting image (named in the format `ubuntu.img.zip` will be output in the present working directory. You can also specify an output folder: |
233 | | - |
234 | | - shub --image ubuntu.img --package --outfolder /tmp |
235 | | - |
236 | | -For the package command, you will need to put in your password to grant sudo priviledges, as packaging requires using the singularity `export` functionality. |
237 | | - |
238 | | -For more details, and a walkthrough with sample data, please see [examples/package_image](examples/package_image) |
239 | 1 |
|
240 | 2 |
|
| 3 | +# Singularity Python |
241 | 4 |
|
242 | | -### Build your container |
243 | | -More information coming soon. |
| 5 | +[](https://travis-ci.org/singularityware/singularity-python) |
244 | 6 |
|
| 7 | +Singularity Python is a python module and command line tool for working with <a href="https://singularityware.github.io" target="_blank">Singularity</a> containers, specifically providing functions to visualize, package, and compare containers. Currently, most functions use Singularity on the command line, and some require sudo. |
245 | 8 |
|
246 | | -### Functions Provided |
247 | | -You can also use the library as a module, and import singularity-python functions into your application. If you would like to see specific examples for something, [please ask](https://github.com/singularityware/singularity-python)! |
| 9 | +The Singularity-Python code is licensed under the MIT open source license, which is a highly permissive license that places few limits upon reuse. This ensures that the code will be usable by the greatest number of researchers, in both academia and industry. The following examples are provided: |
248 | 10 |
|
| 11 | +Please see our [complete docs](https://github.com/singularityware/singularity-python/wiki) |
249 | 12 |
|
250 | 13 | ## Help and Contribution |
251 | | -Please contribute to the package, or post feedback and questions as [issues](https://github.com/singularityware/singularity-python). For points that require discussion of the larger group, please use the [Singularity List](https://groups.google.com/a/lbl.gov/forum/#!forum/singularity). |
| 14 | +Please contribute to the package, or post feedback and questions as <a href="https://github.com/singularityware/singularity-python" target="_blank">issues</a>. For points that require discussion of the larger group, please use the <a href="https://groups.google.com/a/lbl.gov/forum/#!forum/singularity" target="_blank">Singularity List</a> |
| 15 | + |
0 commit comments