Plugin description

Introduciton

jobcomp/elasticsearch is a SLURM job completion plugin that inserts the information related to finished jobs (COMPLETED, CANCELLED, FAILED, TIMEOUT or NODE_FAIL) in a custom ElasticSearch server.

If data can't be indexed for whatever reason (server not reachable, index in readonly mode, etc.), the plugin saves the job information in a state file for future retries.

It is a good idea to have a web layer over your ElasticSearch server, such as Kibana, in order to visualize the data.

Note that the plugin has libcurl-devel library as a dependency, so you can use these two configure options:

--with-libcurl (used by default, so not necessary)
--without-libcurl

Here you can find some Screenshots.

Any suggestions are more than welcome to asanchez1987@gmail.com

Installation

# change directory to your slurm source directory
cd slurm-14.11.3/
wget -O jobcomp_es-0.1.tar.gz https://github.com/asanchez1987/jobcomp-elasticsearch/archive/v0.1.tar.gz
tar xf ./jobcomp_es-0.1.tar.gz --strip-components=1
./autogen.sh
./configure
# Ensure libcurl is usable:
# ...
# checking for curl-config... /usr/bin/curl-config
# checking for the version of libcurl... 7.19.7
# checking whether libcurl is usable... yes
# checking for curl_free... yes
# ...
make
make install

Plugin can be enabled and configured through slurm.conf, here is an example:

JobCompType=jobcomp/elasticsearch
JobCompLoc=http://YOUR_ELASTICSEARCH_SERVER:9200

Make sure that the ElasticSearch server is reachable from the Slurm controller host.

All the jobs will be stored in the slurm index with jobcomp type.