Skip to content

Commit 723105a

Browse files
authored
Merge pull request #409 from ldbc/remove-maven
Introduce dynamic versioning and remove Maven
2 parents 2b22c43 + 3208700 commit 723105a

File tree

12 files changed

+148
-366
lines changed

12 files changed

+148
-366
lines changed

.dockerignore

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,10 @@
44
*.swp
55
*.crc
66
*.log
7-
87
**/*.iml
98
.idea/*
109
**/.gitignore
1110
.git/*
1211
.circleci
13-
12+
**/.bloop
1413
Dockerfile

Dockerfile

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,13 @@
11
FROM eclipse-temurin:8 as build-jar
2-
ARG MAVEN_VERSION=3.8.6
3-
COPY pom.xml /build/pom.xml
2+
ARG SBT_VERSION=1.5.2
3+
RUN cd /opt && curl -fSsL https://github.com/sbt/sbt/releases/download/v${SBT_VERSION}/sbt-${SBT_VERSION}.tgz | tar xvz
4+
ENV PATH=/opt/sbt/bin:$PATH
45
WORKDIR build
5-
RUN cd /opt && curl https://dlcdn.apache.org/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz | tar xvz
6-
ENV PATH=/opt/apache-maven-${MAVEN_VERSION}/bin:$PATH
7-
RUN mvn install
8-
COPY src /build/src
9-
RUN mvn assembly:assembly -DskipTests
6+
COPY build.sbt build.sbt
7+
COPY project project
8+
RUN sbt update
9+
COPY src src
10+
RUN sbt assembly
1011

1112
FROM scratch as jar
1213
COPY --from=build-jar /build/target/ldbc_snb_datagen_*-jar-with-dependencies.jar /jar

README.md

Lines changed: 13 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -20,21 +20,11 @@ The LDBC SNB Data Generator (Datagen) produces the datasets for the [LDBC Social
2020

2121
### Build the JAR
2222

23-
You can build the JAR with both Maven and SBT.
23+
To assemble the JAR file with SBT, run:
2424

25-
* To assemble the JAR file with Maven, run:
26-
27-
```bash
28-
./tools/build.sh
29-
```
30-
31-
* For faster builds during development, consider using SBT. To assemble the JAR file with SBT, run:
32-
33-
```bash
34-
sbt assembly
35-
```
36-
37-
:warning: When using SBT, change the path of the JAR file in the instructions provided in the README (`target/ldbc_snb_datagen_${PLATFORM_VERSION}-${DATAGEN_VERSION}.jar` -> `./target/scala-2.12/ldbc_snb_datagen-assembly-${DATAGEN_VERSION}.jar`).
25+
```bash
26+
sbt assembly
27+
```
3828

3929
### Install Python tools
4030

@@ -78,9 +68,9 @@ Both Java 8 and Java 11 are supported.
7868
Once you have Spark in place and built the JAR file, run the generator as follows:
7969

8070
```bash
81-
export PLATFORM_VERSION=2.12_spark3.2
82-
export DATAGEN_VERSION=0.5.0-SNAPSHOT
83-
export LDBC_SNB_DATAGEN_JAR=./target/ldbc_snb_datagen_${PLATFORM_VERSION}-${DATAGEN_VERSION}.jar
71+
export PLATFORM_VERSION=$(sbt -batch -error 'print platformVersion')
72+
export DATAGEN_VERSION=$(sbt -batch -error 'print version')
73+
export LDBC_SNB_DATAGEN_JAR=$(sbt -batch -error 'print assembly / assemblyOutputPath')
8474
./tools/run.py <runtime configuration arguments> -- <generator configuration arguments>
8575
```
8676

@@ -166,7 +156,7 @@ It is also possible to pass a parameter file:
166156

167157
### Docker images
168158
SNB Datagen images are available via [Docker Hub](https://hub.docker.com/orgs/ldbc/repositories).
169-
The image tags follow the pattern `${DATAGEN_VERSION}-${PLATFORM_VERSION}`, e.g `ldbc/datagen-standalone:0.5.0-2.12_spark3.1`.
159+
The image tags follow the pattern `${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}`, e.g `ldbc/datagen-standalone:0.5.0-2.12_spark3.2`.
170160

171161
When building images ensure that you [use BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds).
172162

@@ -180,27 +170,28 @@ docker run \
180170
--mount type=bind,source="$(pwd)"/out_sf0.003_interactive,target=/out \
181171
--mount type=bind,source="$(pwd)"/conf,target=/conf,readonly \
182172
-e SPARK_CONF_DIR=/conf \
183-
ldbc/datagen-standalone:latest --parallelism 1 -- --format csv --scale-factor 0.003 --mode interactive
173+
ldbc/datagen-standalone:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION} --parallelism 1 -- --format csv --scale-factor 0.003 --mode interactive
184174
```
185175

186176
The standalone Docker image can be built with the provided Dockerfile. To build, execute the following command from the repository directory:
187177

188178
```bash
189-
docker build . --target=standalone -t ldbc/datagen-standalone:latest
179+
docker build . --target=standalone -t ldbc/datagen-standalone:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}
190180
```
191181

192182
#### JAR-only image
193183
The `ldbc/datagen-jar` image contains the assembly JAR, so it can bundled in your custom container:
194184

195185
```docker
196186
FROM my-spark-image
197-
COPY --from=ldbc/datagen-jar:latest /jar /lib/ldbc-datagen.jar
187+
ARG VERSION
188+
COPY --from=ldbc/datagen-jar:$VERSION /jar /lib/ldbc-datagen.jar
198189
```
199190

200191
The JAR-only Docker image can be built with the provided Dockerfile. To build, execute the following command from the repository directory:
201192

202193
```bash
203-
docker build . --target=jar -t ldbc/datagen-jar:latest
194+
docker build . --target=jar -t ldbc/datagen-jar:${DATAGEN_VERSION/+/-}-${PLATFORM_VERSION}
204195
```
205196
### Elastic MapReduce
206197

build.sbt

Lines changed: 104 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,110 @@
1+
name := "ldbc_snb_datagen"
2+
organization := "ldbc.snb.datagen"
3+
4+
scmInfo := {
5+
val org = "ldbc"
6+
val project = "ldbc_snb_datagen_spark"
7+
Some(ScmInfo(url(s"https://github.com/$org/$project"), s"[email protected]:$org/$project.git"))
8+
}
9+
10+
Global / onChangedBuildSource := ReloadOnSourceChanges
11+
Global / cancelable := true
112
ThisBuild / Test / parallelExecution := false
213
ThisBuild / Test / fork := true
3-
ThisBuild / cancelable := true
414

5-
javacOptions ++= Seq("-source", "1.8", "-target", "1.8", "-Xlint")
15+
val sparkVersion = settingKey[String]("The version of Spark used for building.")
16+
val sparkCompatVersion = taskKey[String]("The compatibility version of Spark")
17+
val platformVersion = taskKey[String]("The version of the target platform")
18+
19+
sparkVersion := "3.2.1"
20+
sparkCompatVersion := { sparkVersion.value.split("\\.", 3).take(2).mkString(".") }
21+
platformVersion := { scalaBinaryVersion.value + "_spark" + sparkCompatVersion.value }
22+
23+
resolvers += "TUDelft Repository" at "https://simulation.tudelft.nl/maven/"
24+
25+
libraryDependencies ++= Seq(
26+
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
27+
"org.apache.spark" %% "spark-core" % sparkVersion.value % "provided",
28+
"com.chuusai" %% "shapeless" % "2.3.3",
29+
"com.github.scopt" %% "scopt" % "3.7.1",
30+
"org.javatuples" % "javatuples" % "1.2",
31+
"ca.umontreal.iro" % "ssj" % "2.5",
32+
"xml-apis" % "xml-apis" % "1.4.01",
33+
"org.specs2" %% "specs2-core" % "4.2.0" % Test,
34+
"org.specs2" %% "specs2-junit" % "4.2.0" % Test,
35+
"org.mockito" % "mockito-core" % "3.3.3" % Test,
36+
"org.scalatest" %% "scalatest" % "3.1.0" % Test withSources(),
37+
"junit" % "junit" % "4.13.1" % Test
38+
)
39+
40+
scalaVersion := "2.12.16"
41+
42+
scalacOptions ++= Seq(
43+
"-deprecation", // Emit warning and location for usages of deprecated APIs.
44+
"-encoding", "utf-8", // Specify character encoding used by source files.
45+
"-explaintypes", // Explain type errors in more detail.
46+
"-feature", // Emit warning and location for usages of features that should be imported explicitly.
47+
"-language:existentials", // Existential types (besides wildcard types) can be written and inferred
48+
"-language:experimental.macros", // Allow macro definition (besides implementation and application)
49+
"-language:higherKinds", // Allow higher-kinded types
50+
"-language:implicitConversions", // Allow definition of implicit functions called views
51+
"-unchecked", // Enable additional warnings where generated code depends on assumptions.
52+
"-Xcheckinit", // Wrap field accessors to throw an exception on uninitialized access.
53+
"-Xlint:adapted-args", // Warn if an argument list is modified to match the receiver.
54+
"-Xlint:constant", // Evaluation of a constant arithmetic expression results in an error.
55+
"-Xlint:delayedinit-select", // Selecting member of DelayedInit.
56+
"-Xlint:doc-detached", // A Scaladoc comment appears to be detached from its element.
57+
"-Xlint:inaccessible", // Warn about inaccessible types in method signatures.
58+
"-Xlint:infer-any", // Warn when a type argument is inferred to be `Any`.
59+
"-Xlint:missing-interpolator", // A string literal appears to be missing an interpolator id.
60+
"-Xlint:nullary-override", // Warn when non-nullary `def f()' overrides nullary `def f'.
61+
"-Xlint:nullary-unit", // Warn when nullary methods return Unit.
62+
"-Xlint:option-implicit", // Option.apply used implicit view.
63+
"-Xlint:package-object-classes", // Class or object defined in package object.
64+
"-Xlint:poly-implicit-overload", // Parameterized overloaded implicit methods are not visible as view bounds.
65+
"-Xlint:private-shadow", // A private field (or class parameter) shadows a superclass field.
66+
"-Xlint:stars-align", // Pattern sequence wildcard must align with sequence component.
67+
"-Xlint:type-parameter-shadow", // A local type parameter shadows a type already in scope.
68+
)
669

7-
assemblyMergeStrategy in assembly := {
70+
scalacOptions ++=
71+
scalaVersion {
72+
case sv if sv.startsWith("2.13") => List(
73+
)
74+
75+
case sv if sv.startsWith("2.12") => List(
76+
"-Yno-adapted-args", // Do not adapt an argument list (either by inserting () or creating a tuple) to match the receiver.
77+
"-Ypartial-unification", // Enable partial unification in type constructor inference
78+
"-Ywarn-extra-implicit", // Warn when more than one implicit parameter section is defined.
79+
"-Ywarn-inaccessible", // Warn about inaccessible types in method signatures.
80+
"-Ywarn-infer-any", // Warn when a type argument is inferred to be `Any`.
81+
"-Ywarn-nullary-override", // Warn when non-nullary `def f()' overrides nullary `def f'.
82+
"-Ywarn-nullary-unit", // Warn when nullary methods return Unit.
83+
"-Ywarn-numeric-widen" // Warn when numerics are widened.
84+
)
85+
86+
case _ => Nil
87+
}.value
88+
89+
// The REPL can’t cope with -Ywarn-unused:imports or -Xfatal-warnings so turn them off for the console
90+
Compile / console / scalacOptions --= Seq("-Ywarn-unused:imports", "-Xfatal-warnings")
91+
92+
javacOptions ++= Seq(
93+
"-Xlint",
94+
"-source", "1.8",
95+
"-target", "1.8",
96+
"-g:vars"
97+
)
98+
99+
assembly / assemblyMergeStrategy := {
8100
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
9-
case x => MergeStrategy.first
101+
case _ => MergeStrategy.first
102+
}
103+
104+
// Override JAR name
105+
assembly / assemblyJarName := {
106+
moduleName.value + "_" + platformVersion.value + "-" + version.value + "-jar-with-dependencies.jar"
10107
}
108+
109+
// Put under target instead of target/<scala-binary-version>
110+
assembly / target := { target.value }

0 commit comments

Comments
 (0)