Skip to content
This repository was archived by the owner on May 12, 2021. It is now read-only.

[PIO-210] Add support for Scala 2.12; Update dependencies #518

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from
Open

[PIO-210] Add support for Scala 2.12; Update dependencies #518

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5d5f38f
[PIO-210] Add support for Scala 2.12; Update dependencies
EmergentOrder Jun 25, 2019
0496da7
cleanup LICENSE.txt
EmergentOrder Jun 25, 2019
618a36a
Drop specs2 version to fix build
EmergentOrder Jun 25, 2019
9368fe5
Add Scala 2.12 tests to travis
EmergentOrder Jun 25, 2019
54cae9c
Attempt to fix travis build
EmergentOrder Jun 25, 2019
b6262a1
Attempt to fix travis build
EmergentOrder Jun 25, 2019
ec817b0
Don't use 2.11 version of elasticsearch-spark in 2.12
EmergentOrder Jun 26, 2019
4289256
Revert assembly plugin
EmergentOrder Jun 26, 2019
d8c0796
Be explicit about Scala version
EmergentOrder Jun 27, 2019
2ea73d4
Fix pio-core dep in test engine
EmergentOrder Jun 28, 2019
0748158
Merge branch 'develop' into develop
EmergentOrder Nov 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 4 additions & 9 deletions LICENSE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1433,11 +1433,9 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI
Binary distribution bundles

com.esotericsoftware # kryo # 3.0.3 (https://github.com/EsotericSoftware/kryo)
com.esotericsoftware # kryo-shaded # 4.0.2 (https://github.com/EsotericSoftware/kryo)
com.esotericsoftware # minlog # 1.3.0 (https://github.com/EsotericSoftware/minlog)
com.esotericsoftware # reflectasm # 1.10.1 (https://github.com/EsotericSoftware/reflectasm)
com.esotericsoftware.kryo # kryo # 2.21 (https://github.com/EsotericSoftware/kryo)
com.esotericsoftware.minlog # minlog # 1.2 (https://github.com/EsotericSoftware/minlog)
com.esotericsoftware.reflectasm # reflectasm # 1.07 (https://github.com/EsotericSoftware/reflectasm)

which is available under the BSD license (http://www.opensource.org/licenses/bsd-license.php)

Expand Down Expand Up @@ -1529,8 +1527,8 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
Binary distribution bundles

com.thoughtworks.paranamer # paranamer # 2.3 (https://github.com/paul-hammant/paranamer)
com.thoughtworks.paranamer # paranamer # 2.6 (https://github.com/paul-hammant/paranamer)
com.thoughtworks.paranamer # paranamer # 2.3 (https://github.com/paul-hammant/paranamer)
com.thoughtworks.paranamer # paranamer # 2.8 (https://github.com/paul-hammant/paranamer)

which is available under the BSD license (http://www.opensource.org/licenses/bsd-license.php)

Expand Down Expand Up @@ -1632,7 +1630,7 @@ THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
Binary distribution bundles

org.clapper # grizzled-slf4j_2.11 # 1.0.2 (http://software.clapper.org/grizzled-slf4j/)
org.clapper # grizzled-slf4j_2.11 # 1.3.4 (http://software.clapper.org/grizzled-slf4j/)

which is available under the BSD license (http://www.opensource.org/licenses/bsd-license.php)

Expand Down Expand Up @@ -1701,13 +1699,10 @@ EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Binary distribution bundles

org.scala-lang # scala-library # 2.11.12 (http://scala-lang.org/)
org.scala-lang # scala-compiler # 2.11.12 (http://scala-lang.org/)
org.scala-lang # scala-reflect # 2.11.12 (http://scala-lang.org/)
org.scala-lang # scalap # 2.11.12 (http://scala-lang.org/)
org.scala-lang.modules # scala-java8-compat_2.11 # 0.7.0 (http://scala-lang.org/)
org.scala-lang.modules # scala-parser-combinators_2.11 # 1.0.6 (http://scala-lang.org/)
org.scala-lang.modules # scala-parser-combinators_2.11 # 1.1.0 (http://scala-lang.org/)
org.scala-lang.modules # scala-xml_2.11 # 1.0.5 (http://scala-lang.org/)
org.scala-lang.modules # scala-xml_2.11 # 1.0.6 (http://scala-lang.org/)

which is available under the BSD license (http://www.scala-lang.org/downloads/license.html)
Expand Down
4 changes: 2 additions & 2 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ scalaVersion in ThisBuild := sys.props.getOrElse("scala.version", "2.11.12")

scalaBinaryVersion in ThisBuild := binaryVersion(scalaVersion.value)

crossScalaVersions in ThisBuild := Seq("2.11.12")
crossScalaVersions in ThisBuild := Seq("2.11.12", "2.12.8")

scalacOptions in ThisBuild ++= Seq("-deprecation", "-unchecked", "-feature")

Expand All @@ -37,7 +37,7 @@ javacOptions in (ThisBuild, compile) ++= Seq("-source", "1.8", "-target", "1.8",
"-Xlint:deprecation", "-Xlint:unchecked")

// Ignore differentiation of Spark patch levels
sparkVersion in ThisBuild := sys.props.getOrElse("spark.version", "2.1.3")
sparkVersion in ThisBuild := sys.props.getOrElse("spark.version", "2.4.3")

sparkBinaryVersion in ThisBuild := binaryVersion(sparkVersion.value)

Expand Down
8 changes: 4 additions & 4 deletions core/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ name := "apache-predictionio-core"
libraryDependencies ++= Seq(
"com.github.scopt" %% "scopt" % "3.5.0",
"com.google.code.gson" % "gson" % "2.5",
"com.twitter" %% "chill-bijection" % "0.7.2",
"com.twitter" %% "chill-bijection" % "0.9.3",
"de.javakaffee" % "kryo-serializers" % "0.37",
"net.jodah" % "typetools" % "0.3.1",
"org.apache.spark" %% "spark-core" % sparkVersion.value % "provided",
"org.json4s" %% "json4s-ext" % json4sVersion.value,
"org.scalaj" %% "scalaj-http" % "1.1.6",
"org.scalaj" %% "scalaj-http" % "2.4.2",
"org.slf4j" % "slf4j-log4j12" % "1.7.18",
"org.scalatest" %% "scalatest" % "2.1.7" % "test",
"org.specs2" %% "specs2" % "2.3.13" % "test",
"org.scalatest" %% "scalatest" % "3.0.5" % "test",
"org.specs2" %% "specs2-core" % "4.5.1" % "test",
"org.scalamock" %% "scalamock-scalatest-support" % "3.5.0" % "test",
"com.h2database" % "h2" % "1.4.196" % "test"
)
Expand Down
10 changes: 5 additions & 5 deletions data/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,15 @@ name := "apache-predictionio-data"

libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value,
"com.github.nscala-time" %% "nscala-time" % "2.6.0",
"com.github.nscala-time" %% "nscala-time" % "2.22.0",
"com.google.guava" % "guava" % "14.0.1",
"com.typesafe.akka" %% "akka-http-testkit" % "10.1.5" % "test",
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
"org.clapper" %% "grizzled-slf4j" % "1.0.2",
"org.scalatest" %% "scalatest" % "2.1.7" % "test",
"org.specs2" %% "specs2" % "3.3.1" % "test"
"org.clapper" %% "grizzled-slf4j" % "1.3.4",
"org.scalatest" %% "scalatest" % "3.0.5" % "test",
"org.specs2" %% "specs2-core" % "4.5.1" % "test"
exclude("org.scalaz.stream", s"scalaz-stream_${scalaBinaryVersion.value}"),
"org.scalamock" %% "scalamock-specs2-support" % "3.5.0" % "test",
"org.scalamock" %% "scalamock-specs2-support" % "3.6.0" % "test",
"com.h2database" % "h2" % "1.4.196" % "test")

parallelExecution in Test := false
Expand Down
2 changes: 1 addition & 1 deletion e2/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ parallelExecution in Test := false

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-mllib" % sparkVersion.value % "provided",
"org.scalatest" %% "scalatest" % "2.2.5" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

pomExtra := childrenPomExtra.value
2 changes: 1 addition & 1 deletion project/build.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
sbt.version=1.2.3
sbt.version=1.2.8
4 changes: 2 additions & 2 deletions storage/elasticsearch/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ libraryDependencies ++= Seq(
"org.apache.predictionio" %% "apache-predictionio-core" % version.value % "provided",
"org.apache.spark" %% "spark-core" % sparkVersion.value % "provided",
"org.elasticsearch.client" % "elasticsearch-rest-client" % elasticsearchVersion.value,
"org.elasticsearch" %% "elasticsearch-spark-20" % elasticsearchVersion.value
"org.elasticsearch" % "elasticsearch-spark-20_2.11" % elasticsearchVersion.value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this run on Spark 2.4.3 (Scala 2.12)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you pointed out, tests currently only run under 2.11, so I'm not sure. I wanted to ensure the 2.11 build was working first. That's the only 2.11 dep that doesn't have a 2.12 equivalent.
I'll fix .travis.yml now to test under 2.12.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, too good to be true, of course.
It can't work with the 2.11 version in 2.12, so that is a blocker.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

elastic/elasticsearch-hadoop#1224

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is here, so we can pick this back up once they've merged and published:

elastic/elasticsearch-hadoop#1308

exclude("org.apache.spark", "*"),
"org.elasticsearch" % "elasticsearch-hadoop-mr" % elasticsearchVersion.value,
"org.specs2" %% "specs2" % "2.3.13" % "test")
"org.specs2" %% "specs2-core" % "2.4.17" % "test")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why we choose not the latest version (4.5.1) but 2.4.17?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests failed to compile under the latest version.


parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/hbase/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ libraryDependencies ++= Seq(
exclude("org.mortbay.jetty", "servlet-api-2.5")
exclude("org.mortbay.jetty", "jsp-api-2.1")
exclude("org.mortbay.jetty", "jsp-2.1"),
"org.specs2" %% "specs2" % "2.3.13" % "test")
"org.specs2" %% "specs2-core" % "2.4.17" % "test")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above


parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/hdfs/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ libraryDependencies ++= Seq(
exclude("commons-beanutils", "*"),
"org.apache.hadoop" % "hadoop-hdfs" % hadoopVersion.value,
"org.apache.predictionio" %% "apache-predictionio-data" % version.value % "provided",
"org.scalatest" %% "scalatest" % "2.1.7" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/jdbc/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
"org.scalikejdbc" %% "scalikejdbc" % "3.1.0",
"org.postgresql" % "postgresql" % "9.4-1204-jdbc41" % "test",
"org.specs2" %% "specs2" % "2.3.13" % "test")
"org.specs2" %% "specs2-core" % "2.4.17" % "test")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above


parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/localfs/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ name := "apache-predictionio-data-localfs"

libraryDependencies ++= Seq(
"org.apache.predictionio" %% "apache-predictionio-core" % version.value % "provided",
"org.scalatest" %% "scalatest" % "2.1.7" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion storage/s3/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ libraryDependencies ++= Seq(
"org.apache.predictionio" %% "apache-predictionio-core" % version.value % "provided",
"com.google.guava" % "guava" % "14.0.1" % "provided",
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.132",
"org.scalatest" %% "scalatest" % "2.1.7" % "test")
"org.scalatest" %% "scalatest" % "3.0.5" % "test")

parallelExecution in Test := false

Expand Down
2 changes: 1 addition & 1 deletion tools/build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-sql" % sparkVersion.value % "provided",
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion.value,
"com.typesafe.akka" %% "akka-http-testkit" % "10.1.5" % "test",
"org.specs2" %% "specs2-core" % "4.2.0" % "test")
"org.specs2" %% "specs2-core" % "4.5.1" % "test")

assemblyMergeStrategy in assembly := {
case PathList("META-INF", "LICENSE.txt") => MergeStrategy.concat
Expand Down