You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
--image IMAGE full path to singularity image (for use with --package
43
43
and --tree)
44
-
--images IMAGES images, separated by commas (for use with --simtree)
44
+
--images IMAGES images, separated by commas (for use with --simtree
45
+
and --subtract
45
46
--debug use verbose logging to debug.
46
47
--outfolder OUTFOLDER
47
48
full path to folder for output, stays in tmp (or pwd)
48
49
if not specified
49
50
--package package a singularity container for singularity hub
51
+
--os estimate the operating system of your container.
52
+
--oscalc calculate similarity score for your container vs.
53
+
docker library OS.
54
+
--tags retrieve list of software tags for an image, itself
55
+
minus it's base
50
56
--tree view the guts of an singularity image (use --image)
51
57
--simtree view common guts between two images (use --images)
52
-
--subtract subtract one container image from the second to make
53
-
a difference tree (use --images first,subtract)
58
+
--subtract subtract one container image from the second to make a
59
+
difference tree (use --images first,subtract)
54
60
--simcalc calculate similarity (number) between images based on
55
61
file contents.
56
62
--size SIZE If using Docker or shub image, you can change size
57
63
(default is 1024)
58
64
59
65
60
66
61
-
### Package your container
67
+
### Classify your container
68
+
Singularity python provides functions for quickly assessing the base operating system of your container, retrieving a list of software tags that are relevant when this base is subtracted, and getting similarity scores of your container to a library of base software.
62
69
63
-
A package is a zipped up file that contains the image, the singularity runscript as `runscript`, a `VERSION` file, and a list of files `files.txt` and folders `folders.txt` in the container.
The example package can be [downloaded for inspection](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img.zip), as can the [image used to create it](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img). This is one of the drivers underlying [singularity hub](http://www.singularity-hub.org) (under development).
74
+
shub --image docker://python:latest --os
75
+
[sudo] password for vanessa
76
+
Most similar OS found to be debian:7.11
77
+
debian:7.11
68
78
69
-
-**files.txt** and **folders.txt**: are simple text file lists with paths in the container, and this choice is currently done to provide the rawest form of the container contents. These files also are used to generate interactive visualizations, and calculate similarity between containers.
70
-
-**VERSION**: is a text file with one line, an md5 hash generated for the image when it was packaged.
71
-
-**{{image}}.img**: is of course the original singularity container (usually a .img file)
79
+
or to do this from within Python, see the [provided example](examples/classify_image/estimate_os.py). From within python, you can export the sudopw as the environmental variable "pancakes" and it won't need to ask. This is not ideal, but it's required for now since we are using Singularity to export the image. This will likely be changed soon.
72
80
73
-
First, go to where you have some images:
74
81
75
-
ls
76
-
ubuntu.img
77
-
82
+
#### Get software tags
83
+
Singularity Hub uses a simple algorithm to obtain a likely list of software that is important to your image. It assumes that (most) core installed software is in a folder called `bin`, and returns the list of these files with the estimated base image subtracted. You can do this as follows:
78
84
79
-
You can now use the `shub` command line tool to package your image. Note that you must have [singularity installed](https://singularityware.lbl.gov/install-linux), and depending on the function you use, you will likely need to use sudo. We can use the `--package` argument to package our image:
80
85
81
-
shub --image ubuntu.img --package
86
+
shub --image docker://python:latest --tags
87
+
82
88
89
+
We also provide an [example for Python](examples/classify_image/derive_tags.py). If you do this programatically, you can change the folder(s) that are included, meaning that you could get a custom list of software (eg, libraries in `lib`, or python packages in `site-packages`).
83
90
84
-
If no output folder is specified, the resulting image (named in the format `ubuntu.img.zip` will be output in the present working directory. You can also specify an output folder:
If you want to get a complete list of scores for your image against a core set of latest [docker-os](singularity/analysis/packages/docker-os) images:
87
94
88
-
For the package command, you will need to put in your password to grant sudo priviledges, as packaging requires using the singularity `export` functionality.
95
+
shub --image docker://python:latest --oscalc
89
96
90
-
For more details, and a walkthrough with sample data, please see [examples/package_image](examples/package_image)
97
+
98
+
or again see [this example](examples/classify_image/estimate_os.py) for doing this from within python.
91
99
92
100
93
101
### View the inside of a container
@@ -114,6 +122,14 @@ An [interactive demo](https://singularityware.github.io/singularity-python/examp
114
122
115
123
### Visualize Containers
116
124
125
+
#### Container Similarity Clustering
126
+
Do you have sets of containers or packages, and want to cluster them based on similarities?
We have examples for both deriving scores and producing plots like the above, see [examples/package_tree/docker-os.png](examples/package_tree/docker-os.png)
@@ -185,6 +201,38 @@ and the same applies for specification of Docker images, as in the previous exam
185
201
186
202
187
203
204
+
### Package your container
205
+
The driver of much of the above is the simple container package. A package is a zipped up file that contains the image, the singularity runscript as `runscript`, a `VERSION` file, and a list of files `files.txt` and folders `folders.txt` in the container.
The example package can be [downloaded for inspection](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img.zip), as can the [image used to create it](http://www.vbmis.com/bmi/project/singularity/package_image/ubuntu:latest-2016-04-06.img). This is one of the drivers underlying [singularity hub](http://www.singularity-hub.org) (under development).
210
+
211
+
-**files.txt** and **folders.txt**: are simple text file lists with paths in the container, and this choice is currently done to provide the rawest form of the container contents. These files also are used to generate interactive visualizations, and calculate similarity between containers.
212
+
-**VERSION**: is a text file with one line, an md5 hash generated for the image when it was packaged.
213
+
-**{{image}}.img**: is of course the original singularity container (usually a .img file)
214
+
215
+
First, go to where you have some images:
216
+
217
+
ls
218
+
ubuntu.img
219
+
220
+
221
+
You can now use the `shub` command line tool to package your image. Note that you must have [singularity installed](https://singularityware.lbl.gov/install-linux), and depending on the function you use, you will likely need to use sudo. We can use the `--package` argument to package our image:
222
+
223
+
shub --image ubuntu.img --package
224
+
225
+
226
+
If no output folder is specified, the resulting image (named in the format `ubuntu.img.zip` will be output in the present working directory. You can also specify an output folder:
For the package command, you will need to put in your password to grant sudo priviledges, as packaging requires using the singularity `export` functionality.
231
+
232
+
For more details, and a walkthrough with sample data, please see [examples/package_image](examples/package_image)
Copy file name to clipboardExpand all lines: examples/package_tree/README.md
+10Lines changed: 10 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,16 @@
1
1
# How similar are my operating systems?
2
2
A question that has spun out of one of my projects that I suspect would be useful in many applications but hasn't been fully explored is comparison of operating systems. If you think about it, for the last few decades we've generated many methods for comparing differences between files. We have md5 sums to make sure our downloads didn't poop out, and command line tools to quickly look for differences. We now have to take this up a level, because our new level of operation isn't on a single "file", it's on an entire operating system. It's not just your Mom's computer, it's a container-based thing (e.g., Docker or Singularity that contains a base OS plus additional libraries and packages and then the special sauce, the application or analysis that the container was birthed into existence to carry out. It's not good enough to have message storage places to dump these containers, we need simple and consistent methods to computationally compare them, organize them, and let us explore them.
3
3
4
+
We have provided this simple method in Singularity Python, which can produce plots like the following
5
+
6
+
## Cluster Docker (Library) Images based on Base OS
7
+
8
+
9
+
## Cluster Base OS Versions
10
+
11
+
12
+
The derivation of the scores can be seen in [calculate_similarity.py](calculate_similarity.py), and the simple plot in [plot_similarity.py](plot_similarity.py).
13
+
4
14
5
15
# Similarity of File Paths
6
16
When I think about it, an entire understanding of an "image" (or more generally, a computer or operating system) comes down to the programs installed, and files included. Yes, there might be various environmental variables, but I would hypothesize that the environmental variables found in an image have a rather strong correlation with the software installed, and we would do pretty well to understand the guts of an image from the body without the electricity flowing through it. This would need to be tested, but not quite yet.
0 commit comments