Skip to content

Commit f98f1c0

Browse files
olafurpgolafurpg
committed
Document the new design in the README
1 parent d870dd9 commit f98f1c0

File tree

4 files changed

+151
-198
lines changed

4 files changed

+151
-198
lines changed

‎README.md‎

Lines changed: 147 additions & 198 deletions
Original file line numberDiff line numberDiff line change
@@ -2,201 +2,150 @@
22

33
Visit https://lsif.dev/ to learn about LSIF.
44

5-
Please note, lsif-java is currently undergoing a large-scale rewrite. Progress can be followed at [tree/nsc/comsunsource](https://github.com/sourcegraph/lsif-java/tree/nsc/comsunsource).
6-
7-
## Installation
8-
9-
- Java 1.8 or higher
10-
- Maven
11-
12-
### macOS ###
13-
14-
We recommend using brew to install dependancies on mac OS, you can install brew using the following command:
15-
```
16-
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
17-
```
18-
19-
If you already have brew installed, you'll need to make sure its up to date:
20-
```
21-
brew update && brew upgrade brew-cask && brew cleanup && brew cask cleanup
22-
```
23-
24-
Install Java:
25-
```
26-
brew tap homebrew/cask-versions
27-
brew update
28-
brew cask install homebrew/cask-versions/adoptopenjdk8
29-
```
30-
31-
Install Maven:
32-
```
33-
brew install maven
34-
```
35-
36-
### Ubuntu 18.04 ###
37-
38-
```
39-
apt-get update && apt-get install -y git default-jdk maven
40-
```
41-
42-
## Build the LSIF indexer
43-
44-
```
45-
git clone https://github.com/sourcegraph/lsif-java
46-
cd lsif-java
47-
./gradlew installDist
48-
```
49-
50-
## Generating an LSIF dump
51-
52-
### **Step 1**
53-
54-
Ensure you have a `pom.xml` (Maven projects already have one):
55-
56-
For single-project Gradle projects **_(experimental)_**:
57-
58-
1. Add [`maven-publish`](https://docs.gradle.org/current/userguide/publishing_maven.html) to your `plugins` in `build.gradle`
59-
```
60-
plugins {
61-
id 'maven-publish'
62-
}
63-
```
64-
2. Specify a publication:
65-
66-
**_with default sourceSets:_**
67-
68-
```groovy
69-
publishing {
70-
model {
71-
tasks.generatePomFileForSourcegraphPublication {
72-
destination = file("$projectDir/pom.xml")
73-
}
74-
}
75-
publications {
76-
sourcegraph(MavenPublication) {
77-
from components.java
78-
}
79-
}
80-
}
81-
```
82-
83-
**_with non-default sourceSets:_**
84-
85-
```groovy
86-
publishing {
87-
model {
88-
tasks.generatePomFileForSourcegraphPublication {
89-
destination = file("$projectDir/pom.xml")
90-
}
91-
}
92-
publications {
93-
sourcegraph(MavenPublication) {
94-
from components.java
95-
def sourceSets = project.sourceSets.main.java.srcDirs
96-
pom.withXml {
97-
def node = asNode();
98-
if (node.get("build").size() == 0) {
99-
node.appendNode("build").with {
100-
if (sourceSets.size() > 0) {
101-
appendNode("sourceDirectory", sourceSets.first())
102-
}
103-
}
104-
}
105-
}
106-
}
107-
}
108-
}
109-
```
110-
111-
3. Run `./gradlew generatePomFileForSourcegraphPublication`
112-
4. You should now see `pom.xml` at the top of your project directory
113-
114-
For multi-project Gradle projects **_(experimental)_**:
115-
116-
1. Modify your `allprojects` block in the root `build.gradle` to include the following:
117-
118-
```groovy
119-
allprojects {
120-
// ...
121-
122-
apply plugin: "maven-publish"
123-
124-
afterEvaluate { project ->
125-
if (!new File(project.projectDir.toString()+"/build.gradle").exists()) return
126-
publishing {
127-
model {
128-
tasks.generatePomFileForSourcegraphPublication {
129-
destination = file("$projectDir/pom.xml")
130-
}
131-
}
132-
publications {
133-
sourcegraph(MavenPublication) {
134-
def projectDirStr = project.projectDir.toString()
135-
def subprojectSet = project.subprojects
136-
def sourceSetsSet = []
137-
if (project.hasProperty("sourceSets.main.java.srcDirs")) {
138-
sourceSetsSet = project.sourceSets.main.java.srcDirs
139-
}
140-
141-
if (!(new File(projectDirStr+"/build.gradle").exists())) return
142-
143-
def javaApplied = components.collect{it.getName()}.contains("java")
144-
if (javaApplied) from components.java
145-
146-
pom.withXml {
147-
def node = asNode();
148-
if (node.get("build").size() == 0 && javaApplied) {
149-
node.appendNode("build").with {
150-
if (sourceSetsSet.size() > 0) {
151-
appendNode("sourceDirectory", sourceSetsSet.first())
152-
} else {
153-
def dirpath = "${projectDirStr}/src/main/java"
154-
if (new File(dirpath).exists()) appendNode("sourceDirectory", dirpath)
155-
}
156-
}
157-
}
158-
159-
if (subprojectSet.size() > 0) {
160-
node.appendNode("modules").with {
161-
for(Project p : subprojectSet) {
162-
if(new File(p.projectDir.toString()+"/build.gradle").exists()) {
163-
def path = p.projectDir.toString().substring(projectDirStr.size()+1)
164-
appendNode("module", path)
165-
}
166-
}
167-
}
168-
}
169-
}
170-
}
171-
}
172-
}
173-
}
174-
}
175-
```
176-
177-
2. Run `./gradlew generatePomFileForSourcegraphPublication`
178-
3. You should now see a `pom.xml` file for each `build.gradle` throughout the project
179-
180-
### **Step 2**
181-
Generate an LSIF dump:
182-
183-
```
184-
<absolute path to lsif-java>/build/install/lsifjava/bin/lsifjava \
185-
-projectRoot <project directory> \
186-
-out dump.lsif
187-
```
188-
189-
## Comparison to [Microsoft/lsif-java](https://github.com/Microsoft/lsif-java)
190-
191-
- sourcegraph/lsif-java is ~10x faster
192-
- sourcegraph/lsif-java supports cross-file hovers/definitions/references (Microsoft/lsif-java does not)
193-
- sourcegraph/lsif-java uses [Spoon](https://github.com/INRIA/spoon), which is built on [eclipse.jdt.core](https://github.com/eclipse/eclipse.jdt.core)
194-
- Microsoft/lsif-java uses [eclipse.jdt.ls](https://github.com/eclipse/eclipse.jdt.ls), which is also built on [eclipse.jdt.core](https://github.com/eclipse/eclipse.jdt.core)
195-
196-
See https://github.com/microsoft/lsif-java/issues/61 for the status of collaboration efforts.
197-
198-
## Development
199-
200-
```
201-
./dev
202-
```
5+
## Usage
6+
7+
⚠️ This project is under development so there is nothing to try out at the
8+
moment.
9+
10+
### Supported tools and versions
11+
12+
Currently, only Java 8 with the build tool sbt is supported. We hope to increase
13+
compatibility with more Java language versions and build tools as the project
14+
evolves.
15+
16+
| Language version | Support |
17+
| ---------------- | ------- |
18+
| Java 7 | ❌ |
19+
| Java 8 | ✅ |
20+
| Java 11+ | ❌ |
21+
22+
| Build tool | Support |
23+
| ---------- | ------- |
24+
| Gradle | ❌ |
25+
| Maven | ❌ |
26+
| Bazel | ❌ |
27+
| sbt | ✅ |
28+
29+
## Overview
30+
31+
This project is implemented as a
32+
[Java compiler plugin](https://docs.oracle.com/en/java/javase/11/docs/api/jdk.compiler/com/sun/source/util/Plugin.html)
33+
that generates one
34+
[SemanticDB](https://scalameta.org/docs/semanticdb/specification.html) file for
35+
every `*.java` source file. After compilation completes, the SemanticDB files
36+
are processed to produce LSIF.
37+
38+
![A three stage pipeline that starts with a list of Java sources, creates a list of SemanticDB files that then become a single LSIF index.](docs/img/semanticdb-javac-pipeline.svg)
39+
40+
### Why Java compiler plugin?
41+
42+
There are several benefits to implementing lsif-java as a compiler plugin:
43+
44+
- **Simple installation**: compiler plugins are enabled with the `-Xplugin`
45+
compiler option. All Java build tools support a way to customize compiler
46+
options, simplifying installation.
47+
- **Language fidelity**: by using the Java compiler to produce semantic
48+
information, we ensure that the produced LSIF data is accurate even as new
49+
Java language versions with new language features are released.
50+
- **Environment fidelity**: by hooking into the compilation process of the build
51+
tool, we minimize the risk of diverging from the CI build environment such as
52+
installed system dependencies, custom compiler options and custom annotation
53+
processors.
54+
55+
### Why SemanticDB?
56+
57+
SemanticDB is Protobuf schema for information about symbols and types in Java
58+
programs, Scala programs and other languages. There are several benefits to
59+
using SemanticDB as an intermediary representation for LSIF:
60+
61+
- **Simplicity**: It's easy to translate a single Java source file into a single
62+
SemanticDB file inside a compiler plugin. It's more complicated to produce
63+
LSIF because compiler plugins does not have access to a project-wide context,
64+
which is necessary to produce accurate definitions and hovers in multi-module
65+
projects with external library dependencies.
66+
- **Performance**: SemanticDB is fast to write and read. The compiler adds low
67+
overhead on compilation and the final conversion from SemanticDB to LSIF can
68+
be safely parallelized.
69+
- **Cross-language**: SemanticDB has a
70+
[spec](https://scalameta.org/docs/semanticdb/specification.html) for Java and
71+
Scala enabling cross-language navigation in hybrid Java/Scala codebases.
72+
- **Cross-repository**: Compiler plugins have access to both source code and the
73+
classpath (compiled bytecode of upstream dependencies). SemanticDB has been
74+
designed so that it's also possible to generate spec-compliant symbols from
75+
the classpath alone (no source code) and from the syntax tree of an individual
76+
source file (no classpath). This flexibility allows the
77+
[Metals](https://scalameta.org/metals/) language server to index codebases
78+
from a variety of different inputs, and will be helpful for lsif-java in the
79+
future to unblock cross-repository navigation.
80+
81+
## Contributing
82+
83+
The following sections provide tips on how to contribute to this codebase.
84+
85+
### Project structure
86+
87+
These are the main components of the project.
88+
89+
- `semanticdb-javac/src/main/java`: the Java compiler plugin that creates
90+
SemanticDB files.
91+
- `tests/minimized`: minimized Java source files that reproduce interesting test
92+
cases.
93+
- `tests/unit`: fast running unit tests that are helpful for local edit-and-test
94+
workflows.
95+
- `tests/snapshots`: slow running
96+
["snapshot tests"](https://jestjs.io/docs/en/snapshot-testing) that index a
97+
corpus of published Java libraries.
98+
- `build.sbt`: the sbt build definition.
99+
- `project/plugins.sbt`: plugins for the sbt build.
100+
101+
### Helpful commands
102+
103+
| Command | Where | Description |
104+
| ------------------------------------------------------------------ | -------- | ----------------------------------------------------------------------------------- |
105+
| `./sbt` | terminal | Start interactive sbt shell with Java 8. Takes a while to load on the first run. |
106+
| `unit/test` | sbt | Run fast unit tests. |
107+
| `~unit/test` | sbt | Start watch mode to run tests on file save, good for local edit-and-test workflows. |
108+
| `snapshot/testOnly tests.MinimizedSnapshotSuite` | sbt | Runs fast snapshot tests. Indexes a small set of files under `tests/minimized`. |
109+
| `snapshot/testOnly tests.MinimizedSnapshotSuite -- *InnerClasses*` | sbt | Runs only individual tests cases matching the name "InnerClasses". |
110+
| `snapshot/testOnly tests.LibrarySnapshotSuite` | sbt | Runs slow snapshot tests. Indexes a corpus of external Java libraries. |
111+
| `snapshot/test` | sbt | Runs all snapshot tests. |
112+
| `snapshot/run` | sbt | Update snapshot tests. Use this command after you have fixed a bug. |
113+
| `fixAll` | sbt | Run Scalafmt, Scalafix and Javafmt on all sources. Run this before opening a PR. |
114+
115+
### Import the project into IntelliJ
116+
117+
It's recommended to use IntelliJ when editing code in this codebase.
118+
119+
First, install the
120+
[IntelliJ Community Edition](https://www.jetbrains.com/idea/download/). The
121+
community edition is
122+
[open source](https://github.com/JetBrains/intellij-community) and free to use.
123+
124+
Next, install the IntelliJ Scala plugin.
125+
126+
Finally, run "File > Project From Existing Sources" to import the sbt build into
127+
IntelliJ. Select the "sbt" option if it asks you to choose between
128+
sbt/BSP/Bloop.
129+
130+
It's best to run tests from the sbt shell, not from the IntelliJ UI.
131+
132+
### Don't use VS Code/Vim/Sublime Text/Emacs
133+
134+
If you want to use completions and precise code navigation, it's not recommended
135+
to use other editors than IntelliJ. IntelliJ is the only IDE that properly
136+
supports hybrid Java/Scala codebases at the moment, although that may change
137+
soon thanks to lsif-java :)
138+
139+
### Tests are written in Scala
140+
141+
This codebases uses the Scala library [MUnit](https://scalameta.org/munit/) to
142+
write tests because:
143+
144+
- MUnit has built-in assertions that print readable multiline diffs in color.
145+
- MUnit makes it easy to implement
146+
[snapshot testing](https://jestjs.io/docs/en/snapshot-testing), which is a
147+
testing technique that's heavily used in this codebase.
148+
- Multiline literal strings in Scala make it easy to write unit tests for source
149+
code (which is always multiline). Modern versions of Java support multiline
150+
string literals, but they're not supported in Java 8, which is supported by
151+
lsif-java.

‎bin/coursier‎

35.4 KB
Binary file not shown.

‎docs/img/semanticdb-javac-pipeline.svg‎

Lines changed: 1 addition & 0 deletions
Loading

‎sbt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env bash
2+
set -eu
3+
./bin/coursier launch --jvm 8 sbt

0 commit comments

Comments
 (0)