simple extractor onbuild

Author: bingzhang

README.md

@@ -225,9 +225,9 @@ If you need any python packages installed you will need to create file called re
 the docker build process will use `pip install -r requirements.txt` to install these packages.
 
 ## SimpleExtractor
-Motivation: design and implement an simple extractor to bridge Python developer and knowledge of PyClowder library. It is inevitable for us to assume a learning curve for Python developer to be familiar with PyClowder api library and start to code extractor.
+Motivation: design and implement a simple extractor to bridge Python developer and knowledge of PyClowder library. It requires little effort for Python developers to wrap their python code into Clowder's extractors.
 
-Instead of making a complete extractor, SimpleExtractor provides a way to decouple the function of metadata computation from the full implementation of extractor.  Developers will just focus on their own extract function body. Meanwhile, simple extractor will take developer defined function as input to do extraction and then parse and organize the metadata output into Clowder defined metadata data-struct and submit back to Clowder.
+Simple extractors take developer defined main function as input parameter to do extraction and then parse and pack extraction's output into Simple extractor defined metadata data-struct and submit back to Clowder.
 
 Users' function must have to return a ``dict'' object containing metdata and previews.
 ```markdown
@@ -241,17 +241,12 @@ result = {
 ```
 
 ### Example: 
-Extraction on single file is most common extractor type. In Clowder extractors' repositories, we would think 90% or more extractors are computing extraction on single file(e.g., wordcount, meangrey, ocr, etc.), which means user uploads a file onto Clowder, and then that uploaded file will be forwarded to an applicable extractor to compute metadata and extractor will post back Clowder the attached computing metadata and previews for this particular file.
+`wordcount-simpleextractor` is the simplest example to illustrate how to wrap existing Python code as a Simple Extractor.
 
-
-`wordcount-simpleextractor` is the simplest example to use SimpleExtractor. It consists of three files, which will be illustrated respectivaly.
-
-
-wordcount.py is regular python file which is defined and provided by users. In the code, wordcount invoke `wc` command to process input file and extractor lines, words, characters. It packs metadata into python dict.
+wordcount.py is regular python file which is defined and provided by Python developers. In the code, wordcount invoke `wc` command to process input file to extract lines, words, characters. It packs metadata into python dict.
 ```markdown
 import subprocess
 
-
 def wordcount(input_file):
     result = subprocess.check_output(['wc', input_file], stderr=subprocess.STDOUT)
     (lines, words, characters, _) = result.split()
@@ -266,13 +261,12 @@ def wordcount(input_file):
     return result
 ```
 
-To build wordcount simple extractor as docker image, we provide the template Dockerfile shown below. EXTRACTION_FUNC is environment variable and must be assigned to extraction function, where in wordcount.py, the function is `wordcount`.
+To build wordcount as a Simpel extractor docker image, users just simply assign two environment variables in Dockerfile shown below. EXTRACTION_FUNC is environment variable and has to be assigned as extraction function, where in wordcount.py, the extraction function is `wordcount`. Environment variable EXTRACTION_MODULE is the name of module file containing the definition of extraction function.
 ```markdown
-FROM clowder/pyclowder:onbuild
-  
-ENV EXTRACTION_FUNC="wordcount"
+FROM clowder/extractors-simple-extractor:onbuild
 
-CMD python -c "from pyclowder.simpleextractor import SimpleExtractor; from wordcount import *; SimpleExtractor(${EXTRACTION_FUNC}).start()"
+ENV EXTRACTION_FUNC="wordcount"
+ENV EXTRACTION_MODULE="wordcount"
 ```
 
 

docker.sh

@@ -10,8 +10,8 @@ export DEBUG=${DEBUG:-""}
 ${DEBUG} docker build --tag clowder/pyclowder:latest .
 ${DEBUG} docker build --tag clowder/pyclowder:onbuild --file Dockerfile.onbuild .
 ${DEBUG} docker build --tag clowder/extractors-binary-preview:onbuild sample-extractors/binary-preview
-${DEBUG} docker build --tag clowder/extractors-simple-extractor:latest sample-extractors/simple-extractor
+${DEBUG} docker build --tag clowder/extractors-simple-extractor:onbuild sample-extractors/simple-extractor
 
 # build sample extractors
 ${DEBUG} docker build --tag clowder/extractors-wordcount:latest sample-extractors/wordcount
-${DEBUG} docker build --tag clowder/extractors-wordcount-simpleextractor:latest sample-extractors/wordcount-simpleextractor
+${DEBUG} docker build --tag clowder/extractors-wordcount-simpleextractor:latest sample-extractors/wordcount-simple-extractor

sample-extractors/simple-extractor/Dockerfile

@@ -3,4 +3,21 @@ FROM clowder/pyclowder:onbuild
 ENV EXTRACTION_FUNC=""
 ENV EXTRACTION_MODULE=""
 
-CMD python -c "from simple_extractor import SimpleExtractor; from ${EXTRACTION_MODULE} import *; SimpleExtractor(${EXTRACTION_FUNC}).start()"
+# install any packages
+ONBUILD COPY packages.* Dockerfile /home/clowder/
+ONBUILD RUN if [ -e packages.apt ]; then \
+                apt-get -q -q update \
+                && xargs apt-get -y install --no-install-recommends < packages.apt \
+                && rm -rf /var/lib/apt/lists/*; \
+            fi
+
+# install any python packages
+ONBUILD COPY requirements.txt* Dockerfile /home/clowder/
+ONBUILD RUN if [ -e requirements.txt ]; then \
+                pip install --no-cache-dir -r requirements.txt; \
+            fi
+
+# copy all files
+ONBUILD ADD . /home/clowder/
+
+CMD python -c "from simple_extractor import SimpleExtractor; from ${EXTRACTION_MODULE} import *; SimpleExtractor(${EXTRACTION_FUNC}).start()"

sample-extractors/simple-extractor/README.md

@@ -1,23 +1,23 @@
 # Simple Extractor
 
 The goal of the simple extractor is to make writing of an extractor as easy as possible. It wraps almost all of the 
-complexities in itself and exposes only one environment variable called ```EXTRACTION_FUNC```. This environment 
-variable needs to contain the name of the method that needs to be called when this extractor receives a message from 
-the message broker.
+complexities in itself and exposes only two environment variables: ```EXTRACTION_FUNC``` and ```EXTRACTION_MODULE```. Environment 
+variable ```EXTRACTION_FUNC``` is the name of the method that needs to be called when this extractor receives a message from 
+the message broker. The other environment variable ```EXTRACTION_MODULE`` is the module name of python file where ```EXTRACTION_FUNC``` function has been declared.
 
 # When to Use This
 
 1. This simple extractor is meant to be used in those situations when there is already some Python code available that 
 needs to be wrapped as an extractor as quickly as possible.
-2. This extractor ONLY generates JSON format metadata or a list of preview files. If your extractor generates 
+2. This extractor CURRENTLY outputs JSON format metadata or a list of preview files. If your extractor generates 
 any additional information like generated files, datasets, collections, thumbnails, etc., this method cannot be use and 
-you need to write your extractor the normal way using [PyClowder2](https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse)
+you have to write your extractor the normal way using [PyClowder2](https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/browse)
 3. [Docker](https://www.docker.com/) is the recommended way of developing / wrapping your code using the Simple Extractor.
 
 ## Steps for Writing an Extractor Using the Simple Extractor
 
 To write an extractor using the Simple Extractor, you need to have your Python program available. The main function of 
-this Python program needs to accept an input file path as its parameter. It needs to return a Python dictionary that 
+this Python program is supposed to take an input file path as its parameter. It needs to return a Python dictionary that 
 can contain either metadata information ("metadata"), details about file previews ("previews") or both. For example:
 
 ``` json
@@ -29,10 +29,11 @@ can contain either metadata information ("metadata"), details about file preview
 
 1. Let's call your main Python program file ```your_python_program.py``` and the main function ```your_main_function```.
 
-2. Let's create a Dockerfile for your extractor. Its contents need to be:
+2. Let's create a Dockerfile for your extractor. Dockerfile contents need to be:
+
+        FROM clowder/extractors-simple-extractor:onbuild
+        ENV EXTRACTION_FUNC="your_main_function"
+        ENV EXTRACTION_MODULE="your_python_program"
 
-        FROM clowder/extractors-simple-extractor:latest
-        ENV EXTRACTION_FUNC="your_python_program.your_main_function"
 
-TODO: Complete this.
 

sample-extractors/wordcount-simple-extractor/Dockerfile

@@ -1,3 +1,4 @@
-FROM clowder/extractors-simple-extractor:latest
+FROM clowder/extractors-simple-extractor:onbuild
 
 ENV EXTRACTION_FUNC="wordcount"
+ENV EXTRACTION_MODULE="wordcount"