Merge branch 'master' into build-docker-in-travis

Author: szarnyasg

.travis.yml

@@ -16,10 +16,25 @@ script:
 - mvn test | grep "^\\[" | tee mvn.log
 # test if the output of the mvn command contained an "[INFO] BUILD SUCCESS" entry
 - grep 'BUILD SUCCESS' mvn.log
-- ls -al test_data
-#- if [ "$TRAVIS_BRANCH" = "master" -a "$TRAVIS_PULL_REQUEST" = "false" ]; then ./deploy.sh; fi
+# generate SF1 data set
+- wget -q http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz
+- tar xf hadoop-2.6.0.tar.gz
+- export HADOOP_CLIENT_OPTS="-Xmx2G"
+- export HADOOP_HOME=`pwd`/hadoop-2.6.0
+- export LDBC_SNB_DATAGEN_HOME=`pwd`
+- ./run.sh
+- mkdir out
+- cp -r substitution_parameters out/
 notifications:
   slack: ldbcouncil:OrBanrJ7l0EHQbj8T5YdJYhd
   email: false
   on_success: change
   on_failure: always
+deploy:
+  provider: pages
+  skip-cleanup: true
+  local-dir: out
+  github-token: $GITHUB_TOKEN
+  keep: true
+  on:
+    branch: master

README.md

@@ -6,11 +6,11 @@ LDBC-SNB Data Generator
 [![Build Status](https://travis-ci.org/ldbc/ldbc_snb_datagen.svg?branch=master)](https://travis-ci.org/ldbc/ldbc_snb_datagen)
 [![Codacy Badge](https://api.codacy.com/project/badge/Grade/5b0c677c9c4c4de3b6af15f118c9212c)](https://www.codacy.com/app/ArnauPrat/ldbc_snb_datagen?utm_source=github.com&utm_medium=referral&utm_content=ldbc/ldbc_snb_datagen&utm_campaign=Badge_Grade)
 
-The LDBC-SNB Data Generator (DATAGEN) is the responsible of providing the data sets used by all the LDBC benchmarks. This data generator is designed to produce directed labeled graphs that mimic the characteristics of those graphs of real data. A detailed description of the schema produced by datagen, as well as the format of the output files, can be found in the latest version of official [LDBC SNB specification document](https://github.com/ldbc/ldbc_snb_docs).
+The LDBC-SNB Data Generator (Datagen) is the responsible of providing the data sets used by all the LDBC benchmarks. This data generator is designed to produce directed labeled graphs that mimic the characteristics of those graphs of real data. A detailed description of the schema produced by Datagen, as well as the format of the output files, can be found in the latest version of official [LDBC SNB specification document](https://github.com/ldbc/ldbc_snb_docs).
 
 
-ldbc_snb_datagen is part of the [LDBC project](http://www.ldbcouncil.org/).
-ldbc_snb_datagen is GPLv3 licensed, to see detailed information about this license read the `LICENSE.txt` file.
+`ldbc_snb_datagen` is part of the [LDBC project](http://www.ldbcouncil.org/).
+`ldbc_snb_datagen` is GPLv3 licensed, to see detailed information about this license read the `LICENSE.txt` file.
 
 * **[Releases](https://github.com/ldbc/ldbc_snb_datagen/releases)**
 * **[Configuration](https://github.com/ldbc/ldbc_snb_datagen/wiki/Configuration)**
@@ -21,6 +21,13 @@ ldbc_snb_datagen is GPLv3 licensed, to see detailed information about this licen
 
 ## Quick start
 
+There are three main ways to run Datagen:
+(1) using a pseudo-distributed Hadoop installation,
+(2) running the same setup in a Docker image,
+(3) running on a distributed Hadoop cluster.
+
+### Pseudo-distributed Hadoop node
+
 ```bash
 wget http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz
 tar xf hadoop-2.6.0.tar.gz
@@ -33,36 +40,36 @@ cd $LDBC_SNB_DATAGEN_HOME
 ./run.sh
 ```
 
-## Docker image
+### Docker image
 
 The image can be simply built with the provided Dockerfile.
 To build, execute the following command from the repository directory:
-```
+
+```bash
 docker build . --tag ldbc/datagen
 ```
 
-### Options
+#### Configuration
 
 To configure the amount of memory available, set the `HADOOP_CLIENT_OPTS` variable in the Dockerfile. The default value is `-Xmx8G`.
 
-### Running
+#### Running
 
 In order to run the container, a `params.ini` file is required. For reference, please see the `params*.ini` files in the repository. The file will be mounted in the container by the `--mount type=bind,source="$(pwd)/params.ini,target="/opt/ldbc_snb_datagen/params.ini"` option. If required, the source path can be set to a different path.
 
-The container will output it's results in the `/opt/ldbc_snb_datagen/social_network/` directory. In order to save the results of the generation, a directory must be mounted in the container from the host:
+The container will output its results in the `/opt/ldbc_snb_datagen/social_network/` directory. In order to save the results of the generation, a directory must be mounted in the container from the host:
 
-```
+```bash
 mkdir datagen_output
-
 docker run --rm --mount type=bind,source="$(pwd)/datagen_output/",target="/opt/ldbc_snb_datagen/social_network/" --mount type=bind,source="$(pwd)/params.ini",target="/opt/ldbc_snb_datagen/params.ini" ldbc/datagen
 ```
 
 If the memory limit has to be raised, the `-e HADOOP_CLIENT_OPTS="-Xmx..."` parameter can override the default `-Xmx8G` value.
 
-<!-- Publicly available datasets can be found at the LDBC-SNB Amazon Bucket. These datasets are the official SNB datasets and were  generated using version 0.2.6. They are available in the three official supported serializers: CSV, CSVMergeForeign and TTL. The bucket is configured in "Requester Pays" mode, thus in order to access them you need a properly set up AWS client.
-* http://ldbc-snb.s3.amazonaws.com/ -->
+### Hadoop cluster
 
-**Community provided tools**
+Instructions are currently not provided. (TBD)
 
+### Community provided tools
 
 * **[Apache Flink Loader:](https://github.com/s1ck/ldbc-flink-import)** A loader of LDBC datasets for Apache Flink.

deploy.sh

@@ -1,4 +1,4 @@
-#o!/bin/bash
+#!/bin/bash
 
 set -e