Merge pull request #94 from ldbc/batching

Major refactoring to prepare for batching the dynamic part of the graph

Author: szarnyasg

.gitignore

@@ -24,6 +24,7 @@ scripts/
 /social_network/
 *.log
 test_log
+.DS_Store
 
 # Eclipse stuff
 .metadata
@@ -47,5 +48,8 @@ out/
 datagen_output/
 
 # Hadoop directory extracted as instructed in the Quick Start guide
-/hadoop-2.6.0.tar.gz
-/hadoop-2.6.0/
+/hadoop-*.tar.gz
+/hadoop-*/
+
+# configuration file
+params.ini

.travis.yml

@@ -13,7 +13,10 @@ before_install:
 - docker build . --tag ldbc/datagen
 install: true
 script:
+- cp params-csv.ini params.ini
 - docker run --rm --mount type=bind,source="$(pwd)/",target="/opt/ldbc_snb_datagen/out" --mount type=bind,source="$(pwd)/params.ini",target="/opt/ldbc_snb_datagen/params.ini" ldbc/datagen
+- "[[ `md5sum social_network/*.csv | sort | md5sum` == 'fa046e4c44e4c3e8f6858720c45d80ed  -' ]]"
+- "[[ `md5sum substitution_parameters/interactive_* | sort | md5sum` == '5cba23795df372c19688b05c5a9f318f  -' ]]"
 - mkdir out
 - cp -r substitution_parameters out/
 notifications:

README.md

@@ -21,42 +21,50 @@ The LDBC-SNB Data Generator (Datagen) is the responsible of providing the data s
 
 ## Quick start
 
-There are three main ways to run Datagen:
-(1) using a pseudo-distributed Hadoop installation,
-(2) running the same setup in a Docker image,
-(3) running on a distributed Hadoop cluster.
+### Configuration
+
+Initialize the `params.ini` file as needed. For example, to generate the basic CSV files, issue:
+
+```bash
+cp params-csv.ini params.ini
+```
+
+There are three main ways to run Datagen, each using a different approach to configure the amount of memory available.
+
+1. using a pseudo-distributed Hadoop installation,
+2. running the same setup in a Docker image,
+3. running on a distributed Hadoop cluster.
 
 ### Pseudo-distributed Hadoop node
 
-To grab Hadoop, extract it, and set the environment values to sensible defaults, and generate the data as specified in the `params.ini` file, run the following script:
+To configure the amount of memory available, set the `HADOOP_CLIENT_OPTS` environment variable.
+To grab Hadoop, extract it, and set the environment values to sensible defaults, and generate the data as specified in the `params-csv.ini` file, run the following script:
 
 ```bash
-wget http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz
-tar xf hadoop-2.6.0.tar.gz
+cp params-csv.ini params.ini
+wget http://archive.apache.org/dist/hadoop/core/hadoop-2.9.2/hadoop-2.9.2.tar.gz
+tar xf hadoop-2.9.2.tar.gz
 export HADOOP_CLIENT_OPTS="-Xmx2G"
-# set this to the Hadoop 2.6.0 directory
-export HADOOP_HOME=`pwd`/hadoop-2.6.0
+# set this to the Hadoop 2.9.2 directory
+export HADOOP_HOME=`pwd`/hadoop-2.9.2
 # set this to the repository's directory
 export LDBC_SNB_DATAGEN_HOME=`pwd`
 ./run.sh
 ```
 
 ### Docker image
 
-The image can be simply built with the provided Dockerfile.
-To build, execute the following command from the repository directory:
+SNB datagen images are available via [Docker Hub](https://hub.docker.com/r/ldbc/datagen/) where you may find both the latest version of the generator as well as previous stable versions.
+
+Alternatively, the image can be built with the provided Dockerfile. To build, execute the following command from the repository directory:
 
 ```bash
 docker build . --tag ldbc/datagen
 ```
 
-#### Configuration
-
-To configure the amount of memory available, set the `HADOOP_CLIENT_OPTS` variable in the Dockerfile. The default value is `-Xmx8G`.
-
 #### Running
 
-In order to run the container, a `params.ini` file is required. For reference, please see the `params*.ini` files in the repository. The file will be mounted in the container by the `--mount type=bind,source="$(pwd)/params.ini,target="/opt/ldbc_snb_datagen/params.ini"` option. If required, the source path can be set to a different path.
+Set the `params.ini` in the repository as for the pseudo-distributed case. The file will be mounted in the container by the `--mount type=bind,source="$(pwd)/params.ini,target="/opt/ldbc_snb_datagen/params.ini"` option. If required, the source path can be set to a different path.
 
 The container outputs its results in the `/opt/ldbc_snb_datagen/out/` directory which contains two sub-directories, `social_network/` and `subsitution_parameters`. In order to save the results of the generation, a directory must be mounted in the container from the host. The driver requires the results be in the datagen repository directory. To generate the data, run the following command which includes changing the owner (`chown`) of the Docker-mounted volumes:
 
@@ -65,11 +73,11 @@ docker run --rm --mount type=bind,source="$(pwd)/",target="/opt/ldbc_snb_datagen
   sudo chown -R $USER:$USER social_network/ substitution_parameters/
 ```
 
-If you need to raise the memory limit, use the `-e HADOOP_CLIENT_OPTS="-Xmx..."` parameter to override the default value (`-Xmx8G`).
+If you need to raise the memory limit, use the `-e HADOOP_CLIENT_OPTS="-Xmx..."` parameter to override the default value (`-Xmx2G`).
 
 ### Hadoop cluster
 
-Instructions are currently not provided. (TBD)
+Instructions are currently not provided.
 
 ### Community provided tools
 

docker_run.sh

@@ -8,7 +8,7 @@ if [ ! -f /opt/ldbc_snb_datagen/params.ini ]; then
 fi
 
 # Running the generator
-/opt/hadoop-2.6.0/bin/hadoop jar /opt/ldbc_snb_datagen/target/ldbc_snb_datagen-0.2.7-jar-with-dependencies.jar /opt/ldbc_snb_datagen/params.ini
+/opt/hadoop-2.6.0/bin/hadoop jar /opt/ldbc_snb_datagen/target/ldbc_snb_datagen-0.4.0-SNAPSHOT-jar-with-dependencies.jar /opt/ldbc_snb_datagen/params.ini
 
 # Cleanup
 rm -f m*personFactors*

graphalytics-generate-old.sh

@@ -51,9 +51,9 @@ for VERSION in v0.2.1 v0.2.2 v0.2.3 v0.2.4 v0.2.5; do
     # vertices
     echo > params.ini
     echo ldbc.snb.datagen.generator.scaleFactor:graphalytics.$SCALE_FACTOR >> params.ini
-    echo ldbc.snb.datagen.serializer.personSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVPersonSerializer >> params.ini
-    echo ldbc.snb.datagen.serializer.invariantSerializer:ldbc.snb.datagen.serializer.empty.EmptyInvariantSerializer >> params.ini
-    echo ldbc.snb.datagen.serializer.personActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyPersonActivitySerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVDynamicPersonSerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.empty.EmptyStaticSerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyDynamicActivitySerializer >> params.ini
 
     ./run.sh
     tail -n +2 social_network/person_0_0.csv | wc -l >> ../datagen-graphalytics.log
@@ -63,12 +63,12 @@ for VERSION in v0.2.1 v0.2.2 v0.2.3 v0.2.4 v0.2.5; do
     fi
 
     # edges
-    # from version 0.2.2, it's also possible to use the CSVPersonSerializerWithWeights serializer, which adds edge weights
+    # from version 0.2.2, it's also possible to use the CSVDynamicPersonSerializerWithWeights serializer, which adds edge weights
     echo > params.ini
     echo ldbc.snb.datagen.generator.scaleFactor:graphalytics.$SCALE_FACTOR >> params.ini
-    echo ldbc.snb.datagen.serializer.personSerializer:ldbc.snb.datagen.serializer.graphalytics.CSVPersonSerializer >> params.ini
-    echo ldbc.snb.datagen.serializer.invariantSerializer:ldbc.snb.datagen.serializer.empty.EmptyInvariantSerializer >> params.ini
-    echo ldbc.snb.datagen.serializer.personActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyPersonActivitySerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.graphalytics.CSVDynamicPersonSerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.empty.EmptyStaticSerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyDynamicActivitySerializer >> params.ini
 
     ./run.sh
     tail -n +2 social_network/person_knows_person_0_0.csv | wc -l >> ../datagen-graphalytics.log
@@ -79,7 +79,7 @@ for VERSION in v0.2.1 v0.2.2 v0.2.3 v0.2.4 v0.2.5; do
 done
 
 # For versions 0.2.6-0.2.8, we only need a single run, which produces both the vertices and the edges
-# using the CSVPersonSerializerExtended class, which also produces edge weights
+# using the CSVDynamicPersonSerializerExtended class, which also produces edge weights
 for VERSION in v0.2.6 v0.2.7 v0.2.8; do
     echo $VERSION >> ../datagen-graphalytics.log
 
@@ -90,9 +90,9 @@ for VERSION in v0.2.6 v0.2.7 v0.2.8; do
     # vertices and edges
     echo > params.ini
     echo ldbc.snb.datagen.generator.scaleFactor:graphalytics.$SCALE_FACTOR >> params.ini
-    echo ldbc.snb.datagen.serializer.personSerializer:ldbc.snb.datagen.serializer.graphalytics.CSVPersonSerializerExtended >> params.ini
-    echo ldbc.snb.datagen.serializer.invariantSerializer:ldbc.snb.datagen.serializer.empty.EmptyInvariantSerializer >> params.ini
-    echo ldbc.snb.datagen.serializer.personActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyPersonActivitySerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.graphalytics.CSVDynamicPersonSerializerExtended >> params.ini
+    echo ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.empty.EmptyStaticSerializer >> params.ini
+    echo ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyDynamicActivitySerializer >> params.ini
 
     ./run.sh
     tail -n +2 social_network/person_0_0.csv | wc -l >> ../datagen-graphalytics.log

params-composite-foreign-key.ini

@@ -1,5 +1,5 @@
 ldbc.snb.datagen.generator.scaleFactor:snb.interactive.1
 
-ldbc.snb.datagen.serializer.personSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVCompositeMergeForeignPersonSerializer
-ldbc.snb.datagen.serializer.invariantSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVCompositeMergeForeignInvariantSerializer
-ldbc.snb.datagen.serializer.personActivitySerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVCompositeMergeForeignPersonActivitySerializer
+ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.snb.csv.compositemergeforeign.CSVCompositeMergeForeignDynamicPersonSerializer
+ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.snb.csv.compositemergeforeign.CSVCompositeMergeForeignStaticSerializer
+ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.snb.csv.compositemergeforeign.CSVCompositeMergeForeignDynamicActivitySerializer

params-composite.ini

@@ -1,5 +1,5 @@
 ldbc.snb.datagen.generator.scaleFactor:snb.interactive.1
 
-ldbc.snb.datagen.serializer.personSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVCompositePersonSerializer
-ldbc.snb.datagen.serializer.invariantSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVCompositeInvariantSerializer
-ldbc.snb.datagen.serializer.personActivitySerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVCompositePersonActivitySerializer
+ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.snb.csv.composite.CSVCompositeDynamicPersonSerializer
+ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.snb.csv.composite.CSVCompositeStaticSerializer
+ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.snb.csv.composite.CSVCompositeDynamicActivitySerializer

params-csv.ini

@@ -0,0 +1,8 @@
+ldbc.snb.datagen.generator.scaleFactor:snb.interactive.1
+
+ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.snb.csv.basic.CSVDynamicPersonSerializer
+ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.snb.csv.basic.CSVStaticSerializer
+ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.snb.csv.basic.CSVDynamicActivitySerializer
+
+# To generate RFC 3339-compliant timestamps (https://tools.ietf.org/html/rfc3339), uncomment the following line:
+#ldbc.snb.datagen.serializer.formatter.StringDateFormatter.dateTimeFormat:yyyy-MM-dd'T'HH:mm:ss.SSS+00:00

params-foreign-key.ini

@@ -1,5 +1,5 @@
 ldbc.snb.datagen.generator.scaleFactor:snb.interactive.1
 
-ldbc.snb.datagen.serializer.personSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVMergeForeignPersonSerializer
-ldbc.snb.datagen.serializer.invariantSerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVMergeForeignInvariantSerializer
-ldbc.snb.datagen.serializer.personActivitySerializer:ldbc.snb.datagen.serializer.snb.interactive.CSVMergeForeignPersonActivitySerializer
+ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.snb.csv.mergeforeign.CSVMergeForeignDynamicPersonSerializer
+ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.snb.csv.mergeforeign.CSVMergeForeignStaticSerializer
+ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.snb.csv.mergeforeign.CSVMergeForeignDynamicActivitySerializer

params-graphalytics.ini

@@ -1,5 +1,5 @@
 ldbc.snb.datagen.generator.scaleFactor:graphalytics.1
 
-ldbc.snb.datagen.serializer.personSerializer:ldbc.snb.datagen.serializer.graphalytics.CSVPersonSerializerExtended
-ldbc.snb.datagen.serializer.invariantSerializer:ldbc.snb.datagen.serializer.empty.EmptyInvariantSerializer
-ldbc.snb.datagen.serializer.personActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyPersonActivitySerializer
+ldbc.snb.datagen.serializer.dynamicPersonSerializer:ldbc.snb.datagen.serializer.graphalytics.CSVDynamicPersonSerializerExtended
+ldbc.snb.datagen.serializer.staticSerializer:ldbc.snb.datagen.serializer.empty.EmptyStaticSerializer
+ldbc.snb.datagen.serializer.dynamicActivitySerializer:ldbc.snb.datagen.serializer.empty.EmptyDynamicActivitySerializer