Skip to content

Commit 92dbd77

Browse files
authored
Added documentation for the Java API. (spark-jobserver#862)
* Added documentation for the Java API. * Update javaapi.md
1 parent b571ca5 commit 92dbd77

File tree

1 file changed

+41
-0
lines changed

1 file changed

+41
-0
lines changed

doc/javaapi.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
##SJS Java Api
2+
3+
The Java API in the Spark Job Server is designed to be a more Java friendly API for users to run Spark Jobs
4+
inside of SJS.
5+
6+
There are a few differences between this API and the standard one.
7+
8+
- The API is entirely Java based
9+
- Type safety is done via generics instead of type aliases
10+
- Requires it's own set of Context factories specific to Java (very important)
11+
12+
###Basic Example
13+
14+
```java
15+
public class JavaHelloWorldJob implements JSparkJob<String> {
16+
17+
public String run(JavaSparkContext sc, JobEnvironment runtime, Config data) {
18+
return "Hi!";
19+
}
20+
21+
public Config verify(JavaSparkContext sc, JobEnvironment runtime, Config config) {
22+
return ConfigFactory.empty();
23+
}
24+
}
25+
```
26+
[Full File Here](https://github.com/spark-jobserver/spark-jobserver/blob/master/job-server-tests/src/main/java/spark/jobserver/JavaHelloWorldJob.java)
27+
28+
This example does nothing Spark based, but gives an idea of the difference in structure. In order to run a job
29+
written like this, the user **MUST** use the `JavaSparkContextFactory`. This context is configured the same as
30+
any of the Context factories in the Scala API. In addition, there are context types for all Java related jobs.
31+
In the Java API, there are both context factories and interfaces that must be used for each kind of job.
32+
For instance the JSparkJob uses the JavaSparkContextFactory.
33+
34+
###Additional Context and Job Types
35+
36+
- `JavaSqlContextFactory` for SQLContext Jobs with `JSqlJob`
37+
- `JavaStreamingContextFactory` for StreamingContext jobs with `JStreamingJob`
38+
- `JavaHiveContextFactory` for HiveContext jobs with `JHiveJob`
39+
40+
**NOTE: If you do not use the correct context, the jobs will probably not run correctly, it is imperative you configure
41+
jobs correctly.**

0 commit comments

Comments
 (0)