|
| 1 | +# Introduction |
| 2 | + |
| 3 | +Clowder is an open-source research data management system that supports curation of long-tail data and metadata across |
| 4 | +multiple research domains and diverse data types. It uses a metadata extraction bus to perform data curation. Extractors |
| 5 | +are software programs that do the extraction of specific metadata from a file or dataset (a group of related files). |
| 6 | +The Simple Extractor Wrapper is a piece of software being developed to make the process of developing an extractor |
| 7 | +easier. This document will provide the details of writing an extractor program using the Simple Extractor Wrapper. |
| 8 | + |
| 9 | +# Goals of Simple Extractor Wrapper |
| 10 | + |
| 11 | +An extractor can be written in any programming language as long as it can communicate with Clowder using a simple HTTP |
| 12 | +web service API and RabbitMQ. It can be hard to develop an extractor from the scratch when you also consider the code |
| 13 | +that is needed for this communication. To reduce this effort and to avoid code duplication, we created libraries written |
| 14 | +in Python (PyClowder) and Java (JClowder) to make the processing of writing extractors easy in these languages. We chose |
| 15 | +these languages since they are among the most popular ones and they continue to remain to so. Though this is the case, |
| 16 | +there is still some overhead in terms of developing an extractor using these libraries. In order to make the process of |
| 17 | +writing extractors even easier, we created a Simple Extractor Wrapper, that wraps around your existing source code and |
| 18 | +converts your code into an extractor. As the name says, the extractor itself needs to be simple in nature. The extractor |
| 19 | +will process a file and generate metadata in JSON format and/or create a file preview. Any other Clowder API endpoints |
| 20 | +are not currently available through the Simple Extractor and the developer would have to fall back to using PyClowder, |
| 21 | +JClowder or writing the extractor from scratch. |
| 22 | + |
| 23 | +# Creating an Extractor |
| 24 | + |
| 25 | +The main function of your program needs to accept the string format file path of the input file. It also needs to |
| 26 | +return an object containing either metadata information ("metadata"), details about file previews ("previews") or both |
| 27 | +in the following format: |
| 28 | + |
| 29 | +```json |
| 30 | +{ |
| 31 | + "metadata": {}, |
| 32 | + "previews": [] |
| 33 | +} |
| 34 | +``` |
| 35 | + |
| 36 | +The metadata sub document will contain the metadata that is directly uploaded back to clowder and will be associated |
| 37 | +with the file. The previews array is a list of filenames that are previews that will be uploaded to clowder and |
| 38 | +associated with file. Once the previews are uploaded they will be removed from the drive. |
| 39 | + |
| 40 | +When writing the code for the extractor you don't have to worry about interaction with clowder and any subpieces, you |
| 41 | +can test your code locally in your development environment by calling the function that will process the file and see |
| 42 | +if the result matches the output described above. |
| 43 | + |
| 44 | +# Using Extractor in Clowder |
| 45 | + |
| 46 | +Once you are done with the extractor and you have tested your code you can wrap the extractor in a docker image and test |
| 47 | +this image in the full clowder environment. To do this you will need to create a Dockerfile as well as an |
| 48 | +extractor_info.json file as well as some optional additional files need by the docker build process. Once you have these |
| 49 | +files you can build you image using `docker build -t extractor-example .`. This will build the docker image and tag it |
| 50 | +with the name extractor-example (you should replace this with a better name). |
| 51 | + |
| 52 | +The dockerfile has 2 environment variables that need to be set: |
| 53 | +- R_SCRIPT : the path on disk to the actual file that needs to be sourced for the function. This can be left blank if |
| 54 | + no file needs to be sourced (for example in case when the file is installed as a package). |
| 55 | +- R_FUNCTION : the name of the function that needs to be called that takes a file as input and returns an object that |
| 56 | + contains the data described above. |
| 57 | +There can be 2 additional files that are used when creating the docker image: |
| 58 | +- packages.apt : a list of ubuntu packages that need to be installed for the default ubuntu repositories. |
| 59 | +- docker.R : an R script that is run during the docker build process. This can be used to install any required R |
| 60 | + packages. Another option is to install the code if it is provided as an R package. |
| 61 | + |
| 62 | +There also has to be an extractor_info.json file which contains information about the extractor and is used to by the |
| 63 | +extractor framework to initialize the extractor as well as upload information to clowder about the extractor. |
| 64 | + |
| 65 | +```json |
| 66 | +{ |
| 67 | + "@context": "<context root URL>", |
| 68 | + "name": "<extractor name>", |
| 69 | + "version": "<version number>", |
| 70 | + "description": "<extractor description>", |
| 71 | + "author": "<first name> <last name> <<email address>>", |
| 72 | + "contributors": [ |
| 73 | + "<first name> <last name> <<email address>>", |
| 74 | + "<first name> <last name> <<email address>>", |
| 75 | + ], |
| 76 | + "contexts": [ |
| 77 | + { |
| 78 | + "<metadata term 1>": "<URL definition of metadata term 1>", |
| 79 | + "<metadata term 2>": "<URL definition of metadata term 2>", |
| 80 | + } |
| 81 | + ], |
| 82 | + "repository": [ |
| 83 | + { |
| 84 | + "repType": "git", |
| 85 | + "repUrl": "<source code URL>" |
| 86 | + } |
| 87 | + ], |
| 88 | + "process": { |
| 89 | + "file": [ |
| 90 | + "<MIME type/subtype>", |
| 91 | + "<MIME type/subtype>" |
| 92 | + ] |
| 93 | + }, |
| 94 | + "external_services": [], |
| 95 | + "dependencies": [], |
| 96 | + "bibtex": [] |
| 97 | + } |
| 98 | +``` |
| 99 | + |
| 100 | +Once the image with the extractor is build you can test this extractor in the clowder environment. To do this you will |
| 101 | +need to start clowder first. This can be done using a single [docker-compose file](https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder2/raw/docker-compose.yml). |
| 102 | +You can start the full clowder stack using `docker-compose up -p clowder` in the same folder where you downloaded the |
| 103 | +docker-compose file. After some time you will have an instance of clowder running that you can access using: |
| 104 | +http://localhost:9000/ (if you use docker with virtualbox the url will probably be http://192.168.99.100:9000/). |
| 105 | + |
| 106 | +If this is the first time you have started clowder you will need to create an account. You will be asked to enter an |
| 107 | +email address (use [email protected]). If you look at the console where you started clowder using docker-compose you |
| 108 | +will some text and a url of the form http://localhost:9000/signup/57d93076-7eca-418e-be7e-4a06c06f3259. If you follow |
| 109 | +this URL you will be able to create an account for clowder. If you used the [email protected] email address this will |
| 110 | +have admin privileges. |
| 111 | + |
| 112 | +Once you have the full clowder stack running, you can start your extractor using |
| 113 | +`docker run --rm -ti --network clowder_clowder extractor-example`. This will start the extractor and show the output |
| 114 | +of the extractor on the command line. Once the extractor has started successfully, you can upload the appropriate file |
| 115 | +and it should show that it is being processed by the extractor. At this point you have successfully created an |
| 116 | +extractor and deployed it in clowder. |
| 117 | + |
| 118 | + |
0 commit comments