Skip to content

Commit 8dc8be1

Browse files
committed
Remove instructions for Spark 2
1 parent 1fe3d92 commit 8dc8be1

File tree

3 files changed

+7
-10
lines changed

3 files changed

+7
-10
lines changed

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ You can build the JAR with both Maven and SBT.
3636
sbt assembly
3737
```
3838

39-
:warning: When using SBT, change the path of the JAR file in the instructions provided in the README (`target/ldbc_snb_datagen_${PLATFORM_VERSION}-${DATAGEN_VERSION}.jar` -> `./target/scala-2.11/ldbc_snb_datagen-assembly-${DATAGEN_VERSION}.jar`).
39+
:warning: When using SBT, change the path of the JAR file in the instructions provided in the README (`target/ldbc_snb_datagen_${PLATFORM_VERSION}-${DATAGEN_VERSION}.jar` -> `./target/scala-2.12/ldbc_snb_datagen-assembly-${DATAGEN_VERSION}.jar`).
4040

4141
### Install Python tools
4242

tools/emr/README.md

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -56,15 +56,12 @@ To use spot instances, add the `--use-spot` argument:
5656

5757
### Using a different Spark / EMR version
5858

59-
60-
61-
We use EMR 6.3.0 by default, which contains Spark 3.1. You can use a different version by specifying it with the `--emr-version` option.
62-
EMR 5.33.0 is the recommended EMR version to be used with Spark 2.4.
63-
Make sure that you have uploaded the right JAR first!
59+
We use EMR 6.3.0 by default, which packages Spark 3.1. You can use a different version by specifying it with the `--emr-version` option.
60+
Make sure that you have uploaded the right JAR first.
6461

6562
```bash
66-
PLATFORM_VERSION=2.11_spark2.4
67-
./tools/emr/submit_datagen_job.py --bucket ${BUCKET_NAME} --platform-version ${PLATFORM_VERSION} --emr-release emr-5.33.0 ${JOB_NAME} ${SCALE_FACTOR} csv raw
63+
PLATFORM_VERSION=2.12_spark3.1
64+
./tools/emr/submit_datagen_job.py --bucket ${BUCKET_NAME} --platform-version ${PLATFORM_VERSION} --emr-release emr-6.2.0 ${JOB_NAME} ${SCALE_FACTOR} csv raw
6865
```
6966

7067
### Using a parameter file

tools/emr/submit_datagen_job.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -238,13 +238,13 @@ def submit_datagen_job(name,
238238
help='EC2 key name for SSH connection')
239239
parser.add_argument('--platform-version',
240240
default=defaults['platform_version'],
241-
help='The spark platform the JAR is compiled for formatted like {scala.compat.version}_spark{spark.compat.version}, e.g. 2.11_spark2.4, 2.12_spark3.1')
241+
help='The spark platform the JAR is compiled for formatted like {scala.compat.version}_spark{spark.compat.version}, e.g. 2.12_spark3.1')
242242
parser.add_argument('--version',
243243
default=defaults['version'],
244244
help='LDBC SNB Datagen library version')
245245
parser.add_argument('--emr-release',
246246
default=defaults['emr_release'],
247-
help='The EMR release to use. E.g emr-5.33.0, emr-6.3.0')
247+
help='The EMR release to use. E.g. emr-6.3.0')
248248
parser.add_argument('-y', '--yes',
249249
default=defaults['yes'],
250250
action='store_true',

0 commit comments

Comments
 (0)