Skip to content

Commit 992d426

Browse files
committed
Cleanup CI commands and README
1 parent 7655fe8 commit 992d426

File tree

2 files changed

+17
-18
lines changed

2 files changed

+17
-18
lines changed

.circleci/config.yml

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,15 +42,20 @@ jobs:
4242
sudo apt install -y openjdk-8-jdk zip
4343
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/java-8-openjdk-amd64/bin/java 1
4444
sudo update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/bin/java
45-
java -version
46-
- run: docker build . -t ldbc/spark
45+
- run:
46+
name: Build Docker container
47+
command: |
48+
docker build . -t ldbc/spark
4749
- restore_cache:
4850
keys:
4951
- m2-dep-branch:{{ .Branch }}-pom:{{ checksum "pom.xml" }}-
5052
- m2-dep-branch:dev-pom:{{ checksum "pom.xml" }}-
5153
- m2-dep-branch:{{ .Branch }}-
5254
- m2-dep-branch:dev-
53-
- run: mvn -ntp clean test-compile assembly:assembly
55+
- run:
56+
name: Build JAR file
57+
command: |
58+
mvn -ntp clean test-compile assembly:assembly
5459
- save_cache:
5560
key: m2-dep-branch:{{ .Branch }}-pom:{{ checksum "pom.xml" }}-{{ epoch }}
5661
paths:

README.md

Lines changed: 9 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ You can build the JAR with both Maven and SBT.
2525
* To assemble the JAR file with Maven, run:
2626

2727
```bash
28-
tools/build.sh
28+
./tools/build.sh
2929
```
3030

3131
* For faster builds during development, consider using SBT. To assemble the JAR file with SBT, run:
@@ -51,7 +51,7 @@ pip install ./tools
5151
```
5252
### Running locally
5353

54-
The `tools/run.py` is intended for **local runs**. To use it, download and extract Spark as follows.
54+
The `./tools/run.py` script is intended for **local runs**. To use it, download and extract Spark as follows.
5555

5656
#### Spark 3.1.x
5757

@@ -75,26 +75,20 @@ export PATH="$SPARK_HOME/bin":"$PATH"
7575

7676
Both Java 8 and Java 11 are supported.
7777

78-
To build, run
79-
80-
```bash
81-
tools/build.sh
82-
```
83-
84-
Run the script with:
78+
Once you have Spark in place and built the JAR file, run the generator as follows:
8579

8680
```bash
8781
export PLATFORM_VERSION=2.12_spark3.1
8882
export DATAGEN_VERSION=0.5.0-SNAPSHOT
89-
tools/run.py ./target/ldbc_snb_datagen_${PLATFORM_VERSION}-${DATAGEN_VERSION}.jar <runtime configuration arguments> -- <generator configuration arguments>
83+
./tools/run.py ./target/ldbc_snb_datagen_${PLATFORM_VERSION}-${DATAGEN_VERSION}.jar <runtime configuration arguments> -- <generator configuration arguments>
9084
```
9185

9286
#### Runtime configuration arguments
9387

9488
The runtime configuration arguments determine the amount of memory, number of threads, degree of parallelism. For a list of arguments, see:
9589

9690
```bash
97-
tools/run.py --help
91+
./tools/run.py --help
9892
```
9993

10094
To generate a single `part-*.csv` file, reduce the parallelism (number of Spark partitions) to 1.
@@ -158,18 +152,18 @@ It is also possible to pass a parameter file:
158152
The Docker image can be built with the provided Dockerfile. To build, execute the following command from the repository directory:
159153

160154
```bash
161-
tools/docker-build.sh
155+
./tools/docker-build.sh
162156
```
163157

164-
See [Build the JAR](#build-the-jar) to build the library. Then, run the following:
158+
See [Build the JAR](#build-the-jar) to build the library (e.g. by invoking `./tools/build.sh`). Then, run the following:
165159

166160
```bash
167-
tools/docker-run.sh
161+
./tools/docker-run.sh
168162
```
169163

170164
### Elastic MapReduce
171165

172-
We provide scripts to run Datagen on AWS EMR. See the README in the [`tools/emr`](tools/emr) directory for details.
166+
We provide scripts to run Datagen on AWS EMR. See the README in the [`./tools/emr`](tools/emr) directory for details.
173167

174168
## Larger scale factors
175169

0 commit comments

Comments
 (0)