Skip to content

Commit 00cdfc0

Browse files
olafurpgolafurpg
committed
Add lsif-java command-line tool with Gradle and Maven support.
Previously, users needed to add manual configuration to their Gradle/Maven builds in order to use lsif-java. Now, users can run the new `lsif-java` command-line tool to automatically produce LSIF. In order to automatically produce LSIF, `lsif-java` needs to compile all the Java sources with an additional Java compiler options `-Xplugin:semanticdb`. Most build tools support a way to add custom compiler options but that typically requires updating build configuration files such as `pom.xml` for Maven and `build.gradle` for Gradle. However, we want to avoid updating build configuration files. The exact solution to update the compilation options varies between build tools: * Maven: you can customize the `javac` executable through the command-line flag `mvn -Dmaven.compiler.executable=PATH compile` * Gradle: you can pass in a custom Gradle script via `gradle --init-script=PATH`, which allows you to query information out of the build and configure settings. By default, `lsif-java` will try to configure the build to use a custom `javac` script similar to Maven. However, this approach does not work for Gradle builds that use the the newly released "Java toolchains" feature. For such builds, we can provide a custom `java` binary that registers a custom Java agent that ensures that Gradle always enabled the required `-Xplugin:semanticdb` flag.
1 parent 38565eb commit 00cdfc0

File tree

38 files changed

+1537
-39
lines changed

38 files changed

+1537
-39
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ jobs:
1212
- uses: olafurpg/setup-scala@v10
1313
with:
1414
java-version: [email protected]
15+
- run: go get github.com/sourcegraph/lsif-semanticdb/cmd/lsif-semanticdb
1516
- run: sbt test
1617
check:
1718
runs-on: ubuntu-latest

‎.github/workflows/native.yml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
name: Native Image
2+
on:
3+
push:
4+
branches:
5+
- master
6+
pull_request:
7+
release:
8+
types: [published]
9+
jobs:
10+
unix:
11+
runs-on: ${{ matrix.os }}
12+
strategy:
13+
fail-fast: false
14+
matrix:
15+
os: [macOS-latest, ubuntu-latest, windows-latest]
16+
include:
17+
- os: macOS-latest
18+
uploaded_filename: lsif-java-x86_64-apple-darwin
19+
local_path: lsif-java/target/native-image/lsif-java
20+
- os: ubuntu-latest
21+
uploaded_filename: lsif-java-x86_64-pc-linux
22+
local_path: lsif-java/target/native-image/lsif-java
23+
- os: windows-latest
24+
uploaded_filename: lsif-java-x86_64-pc-win32.exe
25+
local_path: lsif-java\target\native-image\lsif-java.exe
26+
steps:
27+
- uses: actions/checkout@v2
28+
- uses: olafurpg/setup-scala@v10
29+
- run: git fetch --tags || true
30+
- name: sbt nativeImage
31+
shell: bash
32+
if: ${{ matrix.os != 'windows-latest' }}
33+
run: |
34+
sbt cli/nativeImage "cli/nativeImageRun --cwd tests/gradle-example"
35+
- name: sbt nativeImage
36+
shell: cmd
37+
if: ${{ matrix.os == 'windows-latest' }}
38+
run: >-
39+
"C:\Program Files (x86)\Microsoft Visual
40+
Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat" && sbt
41+
cli/nativeImage
42+
- uses: actions/upload-artifact@master
43+
with:
44+
path: ${{ matrix.local_path }}
45+
name: ${{ matrix.uploaded_filename }}
46+
- name: Upload release
47+
if: github.event_name == 'release'
48+
uses: actions/[email protected]
49+
env:
50+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
51+
with:
52+
upload_url: ${{ github.event.release.upload_url }}
53+
asset_path: ${{ matrix.local_path }}
54+
asset_name: ${{ matrix.uploaded_filename }}
55+
asset_content_type: application/zip

‎README.md‎

Lines changed: 33 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -13,17 +13,24 @@ Currently, only Java 8 with the build tool sbt is supported. We hope to increase
1313
compatibility with more Java language versions and build tools as the project
1414
evolves.
1515

16-
| Language version | Support |
17-
| ---------------- | ------- |
18-
| Java 7 | ❌ |
19-
| Java 8 | ✅ |
20-
| Java 11+ | ❌ |
16+
| Language version | Support |
17+
| ---------------- | ------------------ |
18+
| Java 7 | ❌ |
19+
| Java 8 | ✅ |
20+
| Java 11 | ✅ |
21+
| Java 12 | Untested, may work |
22+
| Java 13 | Untested, may work |
23+
| Java 14 | Untested, may work |
24+
| Java 15 | Untested, may work |
25+
| Java 16 | Untested, may work |
26+
| Java 17 | Untested, may work |
2127

2228
| Build tool | Support |
2329
| ---------- | ------- |
24-
| Gradle | ❌ |
25-
| Maven | ❌ |
30+
| Gradle | ✅ |
31+
| Maven | ✅ |
2632
| Bazel | ❌ |
33+
| Buck | ❌ |
2734
| sbt | ✅ |
2835

2936
## Overview
@@ -82,6 +89,17 @@ using SemanticDB as an intermediary representation for LSIF:
8289

8390
The following sections provide tips on how to contribute to this codebase.
8491

92+
### System dependencies
93+
94+
- `java`: any version should work
95+
- `git`: any version should work
96+
- `lsif-semanticdb`:
97+
`go get github.com/sourcegraph/lsif-semanticdb/cmd/lsif-semanticdb`
98+
- `gradle`: `brew install gradle`, or see
99+
[general installation guide](https://gradle.org/install/).
100+
- `mvn`: `brew install maven`, or see
101+
[general installation guide](https://www.baeldung.com/install-maven-on-windows-linux-mac).
102+
85103
### Project structure
86104

87105
These are the main components of the project.
@@ -100,17 +118,19 @@ These are the main components of the project.
100118

101119
### Helpful commands
102120

103-
| Command | Where | Description |
104-
| ------------------------------------------------------------------ | -------- | ----------------------------------------------------------------------------------- |
105-
| `./sbt` | terminal | Start interactive sbt shell with Java 8. Takes a while to load on the first run. |
106-
| `unit/test` | sbt | Run fast unit tests. |
107-
| `~unit/test` | sbt | Start watch mode to run tests on file save, good for local edit-and-test workflows. |
121+
| Command | Where | Description |
122+
| ------------------------------------------------------------------- | -------- | ----------------------------------------------------------------------------------- |
123+
| `./sbt` | terminal | Start interactive sbt shell with Java 11. Takes a while to load on the first run. |
124+
| `unit/test` | sbt | Run fast unit tests. |
125+
| `~unit/test` | sbt | Start watch mode to run tests on file save, good for local edit-and-test workflows. |
126+
| `buildTools/test` | sbt | Run slow build tool tests (Gradle, Maven). |
108127
| `snapshots/testOnly tests.MinimizedSnapshotSuite` | sbt | Runs fast snapshot tests. Indexes a small set of files under `tests/minimized`. |
109128
| `snapshots/testOnly tests.MinimizedSnapshotSuite -- *InnerClasses*` | sbt | Runs only individual tests cases matching the name "InnerClasses". |
110129
| `snapshots/testOnly tests.LibrarySnapshotSuite` | sbt | Runs slow snapshot tests. Indexes a corpus of external Java libraries. |
111130
| `snapshots/test` | sbt | Runs all snapshot tests. |
112131
| `snapshots/run` | sbt | Update snapshot tests. Use this command after you have fixed a bug. |
113-
| `fixAll` | sbt | Run Scalafmt, Scalafix and Javafmt on all sources. Run this before opening a PR. |
132+
| `cli/run --cwd DIRECTORY` | sbt | Run `lsif-java` command-line tool against a given Gradle/Maven build. |
133+
| `fixAll` | sbt | Run Scalafmt, Scalafix and Javafmt on all sources. Run this before opening a PR. |
114134

115135
### Import the project into IntelliJ
116136

‎build.sbt‎

Lines changed: 149 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,31 @@
1-
def scala213 = "2.13.4"
2-
def scalametaVersion = "4.4.8"
1+
import java.io.File
2+
import java.util.Properties
3+
import scala.collection.mutable.ListBuffer
4+
5+
lazy val V =
6+
new {
7+
val coursier = "2.0.8"
8+
val bloop = "1.4.7"
9+
val bsp = "2.0.0-M13"
10+
val moped = "0.1.9"
11+
def scala213 = "2.13.4"
12+
def scala212 = "2.12.12"
13+
def scalameta = "4.4.8"
14+
}
315

416
inThisBuild(
517
List(
6-
scalaVersion := scala213,
7-
crossScalaVersions := List(scala213),
18+
scalaVersion := V.scala213,
19+
crossScalaVersions := List(V.scala213),
820
scalafixDependencies +=
921
"com.github.liancheng" %% "organize-imports" % "0.5.0",
1022
scalafixCaching := true,
1123
scalacOptions ++= List("-Wunused:imports"),
1224
semanticdbEnabled := true,
13-
semanticdbVersion := scalametaVersion,
25+
semanticdbVersion := V.scalameta,
1426
organization := "com.sourcegraph",
1527
homepage := Some(url("https://github.com/sourcegraph/lsif-java")),
28+
dynverSeparator := "-",
1629
licenses :=
1730
List("Apache-2.0" -> url("http://www.apache.org/licenses/LICENSE-2.0")),
1831
developers :=
@@ -49,19 +62,30 @@ commands +=
4962
"javafmtCheckAll" :: "publishLocal" :: s
5063
}
5164

52-
lazy val testSettings = List(
53-
skip.in(publish) := true,
54-
autoScalaLibrary := true,
55-
testFrameworks := List(new TestFramework("munit.Framework")),
56-
libraryDependencies ++=
57-
List(
58-
"org.scalameta" %% "munit" % "0.7.10",
59-
"org.scalameta" %% "scalameta" % scalametaVersion,
60-
"org.scala-lang.modules" %% "scala-parallel-collections" % "1.0.0",
61-
"io.get-coursier" %% "coursier" % "2.0.8",
62-
"com.lihaoyi" %% "pprint" % "0.6.1"
63-
)
64-
)
65+
lazy val agent = project
66+
.in(file("semanticdb-agent"))
67+
.settings(
68+
fatjarPackageSettings,
69+
autoScalaLibrary := false,
70+
moduleName := "semanticdb-agent",
71+
libraryDependencies ++=
72+
List(
73+
"org.javassist" % "javassist" % "3.27.0-GA",
74+
"net.bytebuddy" % "byte-buddy" % "1.10.20",
75+
"net.bytebuddy" % "byte-buddy-agent" % "1.10.20"
76+
),
77+
incOptions ~= { old =>
78+
old.withEnabled(false)
79+
},
80+
crossPaths := false,
81+
Compile / packageBin / packageOptions +=
82+
Package.ManifestAttributes(
83+
"Agent-Class" -> "com.sourcegraph.semanticdb_javac.SemanticdbAgent",
84+
"Can-Redefine-Classes" -> "true",
85+
"Can-Retransform-Classes" -> "true",
86+
"Premain-Class" -> "com.sourcegraph.semanticdb_javac.SemanticdbAgent"
87+
)
88+
)
6589

6690
lazy val plugin = project
6791
.in(file("semanticdb-javac"))
@@ -71,16 +95,69 @@ lazy val plugin = project
7195
incOptions ~= { old =>
7296
old.withEnabled(false)
7397
},
98+
fatjarPackageSettings,
7499
crossPaths := false,
75100
PB.targets.in(Compile) :=
76101
Seq(PB.gens.java -> (Compile / sourceManaged).value)
77102
)
78103

104+
lazy val cli = project
105+
.in(file("cli"))
106+
.settings(
107+
moduleName := "lsif-java",
108+
mainClass.in(Compile) := Some("com.sourcegraph.lsif_java.LsifJava"),
109+
buildInfoKeys :=
110+
Seq[BuildInfoKey](
111+
version,
112+
scalaVersion,
113+
"bloopVersion" -> V.bloop,
114+
"bspVersion" -> V.bsp
115+
),
116+
buildInfoPackage := "com.sourcegraph.lsif_java",
117+
libraryDependencies ++= List("org.scalameta" %% "moped" % V.moped),
118+
resourceGenerators.in(Compile) +=
119+
Def
120+
.task[Seq[File]] {
121+
val outs = ListBuffer.empty[(File, File)]
122+
val out = resourceManaged.in(Compile).value.toPath
123+
IO.delete(out.toFile)
124+
def addJar(jar: File, filename: String): Unit = {
125+
outs += jar -> out.resolve(filename).toFile
126+
}
127+
128+
addJar(
129+
Keys.`package`.in(plugin, Compile).value,
130+
"semanticdb-plugin.jar"
131+
)
132+
addJar(
133+
Keys.`package`.in(agent, Compile).value,
134+
"semanticdb-agent.jar"
135+
)
136+
137+
IO.copy(outs)
138+
val props = new Properties()
139+
val propsFile = out.resolve("lsif-java.properties").toFile
140+
val copiedJars = outs.collect { case (_, out) =>
141+
out
142+
}
143+
val names = copiedJars.map(_.getName).mkString(";")
144+
props.put("jarNames", names)
145+
IO.write(props, "lsif-java", propsFile)
146+
147+
propsFile :: copiedJars.toList
148+
}
149+
.taskValue,
150+
nativeImageOptions ++= List("-H:IncludeResources=^semanticdb-.*jar$"),
151+
nativeImageOutput := target.in(NativeImage).value / "lsif-java"
152+
)
153+
.enablePlugins(NativeImagePlugin, BuildInfoPlugin)
154+
79155
lazy val minimized = project
80156
.in(file("tests/minimized"))
81157
.settings(
82158
autoScalaLibrary := false,
83159
skip.in(publish) := true,
160+
fork.in(run) := true,
84161
javacOptions.in(Compile) ++=
85162
List[String](
86163
s"-Arandomtimestamp=${System.nanoTime()}",
@@ -93,7 +170,7 @@ lazy val minimized = project
93170
).mkString(" ")
94171
)
95172
)
96-
.dependsOn(plugin)
173+
.dependsOn(agent, plugin)
97174

98175
lazy val minimizedScala = project
99176
.in(file("tests/minimized-scala"))
@@ -123,9 +200,23 @@ lazy val unit = project
123200
),
124201
buildInfoPackage := "tests"
125202
)
126-
.dependsOn(plugin)
203+
.dependsOn(plugin, cli)
127204
.enablePlugins(BuildInfoPlugin)
128205

206+
lazy val buildTools = project
207+
.in(file("tests/buildTools"))
208+
.settings(
209+
testSettings,
210+
javaOptions.in(Test) ++=
211+
List(
212+
s"-javaagent:${Keys.`package`.in(agent, Compile).value}",
213+
s"-Dsemanticdb.pluginpath=${Keys.`package`.in(plugin, Compile).value}",
214+
s"-Dsemanticdb.sourceroot=${baseDirectory.in(ThisBuild).value}",
215+
s"-Dsemanticdb.targetroot=${target.in(agent, Compile).value / "semanticdb-targetroot"}"
216+
)
217+
)
218+
.dependsOn(agent, unit)
219+
129220
lazy val snapshots = project
130221
.in(file("tests/snapshots"))
131222
.settings(
@@ -148,3 +239,41 @@ lazy val bench = project
148239
)
149240
.dependsOn(unit)
150241
.enablePlugins(JmhPlugin)
242+
243+
lazy val testSettings = List(
244+
skip.in(publish) := true,
245+
autoScalaLibrary := true,
246+
testFrameworks := List(new TestFramework("munit.Framework")),
247+
libraryDependencies ++=
248+
List(
249+
"org.scalameta" %% "munit" % "0.7.10",
250+
"org.scalameta" %% "moped-testkit" % V.moped,
251+
"org.scalameta" %% "scalameta" % V.scalameta,
252+
"org.scala-lang.modules" %% "scala-parallel-collections" % "1.0.0",
253+
"io.get-coursier" %% "coursier" % V.coursier,
254+
"com.lihaoyi" %% "pprint" % "0.6.1"
255+
)
256+
)
257+
258+
lazy val fatjarPackageSettings = List[Def.Setting[_]](
259+
assemblyMergeStrategy in assembly := {
260+
case PathList("javax", _ @_*) =>
261+
MergeStrategy.discard
262+
case PathList("com", "sun", _ @_*) =>
263+
MergeStrategy.discard
264+
case PathList("sun", _ @_*) =>
265+
MergeStrategy.discard
266+
case PathList("META-INF", "versions", "9", "module-info.class") =>
267+
MergeStrategy.first
268+
case x =>
269+
val oldStrategy = (assemblyMergeStrategy in assembly).value
270+
oldStrategy(x)
271+
},
272+
Keys.`package`.in(Compile) := {
273+
val slimJar = Keys.`package`.in(Compile).value
274+
val fatJar = crossTarget.value / assemblyJarName.in(assembly).value
275+
val _ = assembly.value
276+
IO.copyFile(fatJar, slimJar, CopyOptions().withOverwrite(true))
277+
slimJar
278+
}
279+
)

‎cli/src/main/scala/com/sourcegraph/lsif_java/AbsolutePath.scala‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
package com.sourcegraph.lsif_java
2+
3+
import java.nio.file.Path
4+
import java.nio.file.Paths
5+
6+
object AbsolutePath {
7+
def systemWorkingDirectory: Path = Paths.get(".").toAbsolutePath.normalize()
8+
def of(path: Path): Path = of(path, systemWorkingDirectory)
9+
def of(path: Path, cwd: Path): Path =
10+
if (path.isAbsolute)
11+
path
12+
else if (cwd.isAbsolute)
13+
cwd.resolve(path)
14+
else
15+
systemWorkingDirectory.resolve(cwd).resolve(path)
16+
}

‎tests/unit/src/main/scala/tests/DeleteVisitor.scala‎ renamed to ‎cli/src/main/scala/com/sourcegraph/lsif_java/DeleteVisitor.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
package tests
1+
package com.sourcegraph.lsif_java
22

33
import java.io.IOException
44
import java.nio.file.FileVisitResult

0 commit comments

Comments
 (0)