Merge #3053

3053: Feature: downloading lsp server from GitHub r=matklad a=Veetaha

This is currently very WIP, I may need to change this and that, add "download language server command", logging stuff (for future bug reports), etc., but it already works.
Also didn't test this on windows yet and mac (don't have the latter)

The quirks:
* Downloaded binary doesn't have executable permissions by default, that's why we ~~`chmod 111`~~ (**[UPD]** `chmod 755` as per @lnicola [suggestion](#3053 (comment))) for it.
* To remove installed binary run `rm /${HOME}/.config/Code/User/globalStorage/matklad.rust-analyzer/ra_lsp_server-linux`, ~~note that `-f` flag is necessary, because of `111` permissions (I think this should be changed)~~ (**[UPD]** --force is no longer needed due to 755 permissions).

I also tried to keep things simple and not to use too many dependencies, all the ones added have 0 dependencies, (`ts-not-nil` is my personal npm package, that imitates `unwrap()` in TypeScript)

**[UPD]** I reduced throttle latency of progress indicator to 200ms for smoother UX


// TODO:
- [x] ~~Add `Rust Analyzer: Download latest language server` vscode command.~~ **[UPD]**: having reviewed the code and estimated available options I concluded that this feature requires too many code changes, I'd like to extract this into a separate PR after we merge this one.
- [x] Add some logging for future debugging
- [x] ~~Gracefully handle the case when language server is not available (e.g. no internet connection, user explicitly rejected the download, etc.)~~ **[UPD]** Decided to postpone better implementation of graceful degradation logic as per [conversation](https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/Deployment.20and.20installation/near/187758550).

Demo (**[UPD]** this is a bit outdated, but still mainly reflects the feature):
![ra-github-release-download-mvp](https://user-images.githubusercontent.com/36276403/74077961-4f248a80-4a2d-11ea-962f-27c650fd6c4c.gif)

Related issue: #2988 #3007 

Co-authored-by: Veetaha <[email protected]>
Co-authored-by: Veetaha <[email protected]>

Author: 3 people

docs/user/README.md

@@ -31,7 +31,38 @@ a minimum version of 10 installed. Please refer to
 You will also need the most recent version of VS Code: we don't try to
 maintain compatibility with older versions yet.
 
-The experimental VS Code plugin can then be built and installed by executing the
+### Installation from prebuilt binaries
+
+We ship prebuilt binaries for Linux, Mac and Windows via
+[GitHub releases](https://github.com/rust-analyzer/rust-analyzer/releases).
+In order to use them you need to install the client VSCode extension.
+
+Publishing to VSCode marketplace is currently WIP. Thus, you need to clone the repository and install **only** the client extension via
+```
+$ git clone https://github.com/rust-analyzer/rust-analyzer.git --depth 1
+$ cd rust-analyzer
+$ cargo xtask install --client-code
+```
+Then open VSCode (or reload the window if it was already running), open some Rust project and you should
+see an info message pop-up.
+
+
+<img height="140px" src="https://user-images.githubusercontent.com/36276403/74103174-a40df100-4b52-11ea-81f4-372c70797924.png" alt="Download now message"/>
+
+
+Click `Download now`, wait until the progress is 100% and you are ready to go.
+
+For updates you need to remove installed binary
+```
+rm -rf ${HOME}/.config/Code/User/globalStorage/matklad.rust-analyzer
+```
+
+`"Donwload latest language server"` command for VSCode and automatic updates detection is currently WIP.
+
+
+### Installation from sources
+
+The experimental VS Code plugin can be built and installed by executing the
 following commands:
 
 ```
@@ -46,6 +77,7 @@ doesn't, report bugs!
 **Note** [#1831](https://github.com/rust-analyzer/rust-analyzer/issues/1831): If you are using the popular
 [Vim emulation plugin](https://github.com/VSCodeVim/Vim), you will likely
 need to turn off the `rust-analyzer.enableEnhancedTyping` setting.
+(// TODO: This configuration is no longer available, enhanced typing shoud be disabled via removing Enter key binding, [see this issue](https://github.com/rust-analyzer/rust-analyzer/issues/3051))
 
 If you have an unusual setup (for example, `code` is not in the `PATH`), you
 should adapt these manual installation instructions:

editors/code/package-lock.json

@@ -25,18 +25,22 @@
     },
     "dependencies": {
         "jsonc-parser": "^2.1.0",
+        "node-fetch": "^2.6.0",
+        "throttle-debounce": "^2.1.0",
         "vscode-languageclient": "^6.1.0"
     },
     "devDependencies": {
         "@rollup/plugin-commonjs": "^11.0.2",
         "@rollup/plugin-node-resolve": "^7.1.1",
         "@types/node": "^12.12.25",
+        "@types/node-fetch": "^2.5.4",
+        "@types/throttle-debounce": "^2.1.0",
         "@types/vscode": "^1.41.0",
         "rollup": "^1.31.0",
         "tslib": "^1.10.0",
         "tslint": "^5.20.1",
-        "typescript-formatter": "^7.2.2",
         "typescript": "^3.7.5",
+        "typescript-formatter": "^7.2.2",
         "vsce": "^1.71.0"
     },
     "activationEvents": [
@@ -169,10 +173,11 @@
                 },
                 "rust-analyzer.raLspServerPath": {
                     "type": [
+                        "null",
                         "string"
                     ],
-                    "default": "ra_lsp_server",
-                    "description": "Path to ra_lsp_server executable"
+                    "default": null,
+                    "description": "Path to ra_lsp_server executable (points to bundled binary by default)"
                 },
                 "rust-analyzer.excludeGlobs": {
                     "type": "array",

editors/code/package.json

@@ -1,24 +1,18 @@
-import { homedir } from 'os';
 import * as lc from 'vscode-languageclient';
-import { spawnSync } from 'child_process';
 
 import { window, workspace } from 'vscode';
 import { Config } from './config';
+import { ensureLanguageServerBinary } from './installation/language_server';
 
-export function createClient(config: Config): lc.LanguageClient {
+export async function createClient(config: Config): Promise<null | lc.LanguageClient> {
     // '.' Is the fallback if no folder is open
     // TODO?: Workspace folders support Uri's (eg: file://test.txt).
     // It might be a good idea to test if the uri points to a file.
     const workspaceFolderPath = workspace.workspaceFolders?.[0]?.uri.fsPath ?? '.';
 
-    const raLspServerPath = expandPathResolving(config.raLspServerPath);
-    if (spawnSync(raLspServerPath, ["--version"]).status !== 0) {
-        window.showErrorMessage(
-            `Unable to execute '${raLspServerPath} --version'\n\n` +
-            `Perhaps it is not in $PATH?\n\n` +
-            `PATH=${process.env.PATH}\n`
-        );
-    }
+    const raLspServerPath = await ensureLanguageServerBinary(config.langServerSource);
+    if (!raLspServerPath) return null;
+
     const run: lc.Executable = {
         command: raLspServerPath,
         options: { cwd: workspaceFolderPath },
@@ -87,9 +81,3 @@ export function createClient(config: Config): lc.LanguageClient {
     res.registerProposedFeatures();
     return res;
 }
-function expandPathResolving(path: string) {
-    if (path.startsWith('~/')) {
-        return path.replace('~', homedir());
-    }
-    return path;
-}

editors/code/src/client.ts

@@ -1,4 +1,6 @@
+import * as os from "os";
 import * as vscode from 'vscode';
+import { BinarySource } from "./installation/interfaces";
 
 const RA_LSP_DEBUG = process.env.__RA_LSP_SERVER_DEBUG;
 
@@ -16,10 +18,11 @@ export interface CargoFeatures {
 }
 
 export class Config {
+    langServerSource!: null | BinarySource;
+
     highlightingOn = true;
     rainbowHighlightingOn = false;
     enableEnhancedTyping = true;
-    raLspServerPath = RA_LSP_DEBUG || 'ra_lsp_server';
     lruCapacity: null | number = null;
     displayInlayHints = true;
     maxInlayHintLength: null | number = null;
@@ -45,11 +48,72 @@ export class Config {
     private prevCargoWatchOptions: null | CargoWatchOptions = null;
 
     constructor(ctx: vscode.ExtensionContext) {
-        vscode.workspace.onDidChangeConfiguration(_ => this.refresh(), null, ctx.subscriptions);
-        this.refresh();
+        vscode.workspace.onDidChangeConfiguration(_ => this.refresh(ctx), null, ctx.subscriptions);
+        this.refresh(ctx);
+    }
+
+    private static expandPathResolving(path: string) {
+        if (path.startsWith('~/')) {
+            return path.replace('~', os.homedir());
+        }
+        return path;
+    }
+
+    /**
+     * Name of the binary artifact for `ra_lsp_server` that is published for
+     * `platform` on GitHub releases. (It is also stored under the same name when
+     * downloaded by the extension).
+     */
+    private static prebuiltLangServerFileName(platform: NodeJS.Platform): null | string {
+        switch (platform) {
+            case "linux":  return "ra_lsp_server-linux";
+            case "darwin": return "ra_lsp_server-mac";
+            case "win32":  return "ra_lsp_server-windows.exe";
+
+            // Users on these platforms yet need to manually build from sources
+            case "aix":
+            case "android":
+            case "freebsd":
+            case "openbsd":
+            case "sunos":
+            case "cygwin":
+            case "netbsd": return null;
+            // The list of platforms is exhaustive (see `NodeJS.Platform` type definition)
+        }
+    }
+
+    private static langServerBinarySource(
+        ctx: vscode.ExtensionContext,
+        config: vscode.WorkspaceConfiguration
+    ): null | BinarySource {
+        const langServerPath = RA_LSP_DEBUG ?? config.get<null | string>("raLspServerPath");
+
+        if (langServerPath) {
+            return {
+                type: BinarySource.Type.ExplicitPath,
+                path: Config.expandPathResolving(langServerPath)
+            };
+        }
+
+        const prebuiltBinaryName = Config.prebuiltLangServerFileName(process.platform);
+
+        if (!prebuiltBinaryName) return null;
+
+        return {
+            type: BinarySource.Type.GithubRelease,
+            dir: ctx.globalStoragePath,
+            file: prebuiltBinaryName,
+            repo: {
+                name: "rust-analyzer",
+                owner: "rust-analyzer",
+            }
+        };
     }
 
-    private refresh() {
+
+    // FIXME: revisit the logic for `if (.has(...)) config.get(...)` set default
+    // values only in one place (i.e. remove default values from non-readonly members declarations)
+    private refresh(ctx: vscode.ExtensionContext) {
         const config = vscode.workspace.getConfiguration('rust-analyzer');
 
         let requireReloadMessage = null;
@@ -82,10 +146,7 @@ export class Config {
             this.prevEnhancedTyping = this.enableEnhancedTyping;
         }
 
-        if (config.has('raLspServerPath')) {
-            this.raLspServerPath =
-                RA_LSP_DEBUG || (config.get('raLspServerPath') as string);
-        }
+        this.langServerSource = Config.langServerBinarySource(ctx, config);
 
         if (config.has('cargo-watch.enable')) {
             this.cargoWatchOptions.enable = config.get<boolean>(

editors/code/src/config.ts

@@ -11,6 +11,9 @@ export class Ctx {
     // deal with it.
     //
     // Ideally, this should be replaced with async getter though.
+    // FIXME: this actually needs syncronization of some kind (check how
+    // vscode deals with `deactivate()` call when extension has some work scheduled
+    // on the event loop to get a better picture of what we can do here)
     client: lc.LanguageClient | null = null;
     private extCtx: vscode.ExtensionContext;
     private onDidRestartHooks: Array<(client: lc.LanguageClient) => void> = [];
@@ -26,7 +29,14 @@ export class Ctx {
             await old.stop();
         }
         this.client = null;
-        const client = createClient(this.config);
+        const client = await createClient(this.config);
+        if (!client) {
+            throw new Error(
+                "Rust Analyzer Language Server is not available. " +
+                "Please, ensure its [proper installation](https://github.com/rust-analyzer/rust-analyzer/tree/master/docs/user#vs-code)."
+            );
+        }
+
         this.pushCleanup(client.start());
         await client.onReady();
 

editors/code/src/ctx.ts

@@ -0,0 +1,34 @@
+import fetch from "node-fetch";
+import * as fs from "fs";
+import { strict as assert } from "assert";
+
+/**
+ * Downloads file from `url` and stores it at `destFilePath`.
+ * `onProgress` callback is called on recieveing each chunk of bytes
+ * to track the progress of downloading, it gets the already read and total
+ * amount of bytes to read as its parameters.
+ */
+export async function downloadFile(
+    url: string,
+    destFilePath: fs.PathLike,
+    onProgress: (readBytes: number, totalBytes: number) => void
+): Promise<void> {
+    const response = await fetch(url);
+
+    const totalBytes = Number(response.headers.get('content-length'));
+    assert(!Number.isNaN(totalBytes), "Sanity check of content-length protocol");
+
+    let readBytes = 0;
+
+    console.log("Downloading file of", totalBytes, "bytes size from", url, "to", destFilePath);
+
+    return new Promise<void>((resolve, reject) => response.body
+        .on("data", (chunk: Buffer) => {
+            readBytes += chunk.length;
+            onProgress(readBytes, totalBytes);
+        })
+        .on("end", resolve)
+        .on("error", reject)
+        .pipe(fs.createWriteStream(destFilePath))
+    );
+}

editors/code/src/installation/download_file.ts

@@ -0,0 +1,46 @@
+import fetch from "node-fetch";
+import { GithubRepo, ArtifactMetadata } from "./interfaces";
+
+const GITHUB_API_ENDPOINT_URL = "https://api.github.com";
+
+/**
+ * Fetches the latest release from GitHub `repo` and returns metadata about
+ * `artifactFileName` shipped with this release or `null` if no such artifact was published.
+ */
+export async function fetchLatestArtifactMetadata(
+    repo: GithubRepo, artifactFileName: string
+): Promise<null | ArtifactMetadata> {
+
+    const repoOwner = encodeURIComponent(repo.owner);
+    const repoName  = encodeURIComponent(repo.name);
+
+    const apiEndpointPath = `/repos/${repoOwner}/${repoName}/releases/latest`;
+    const requestUrl = GITHUB_API_ENDPOINT_URL + apiEndpointPath;
+
+    // We skip runtime type checks for simplicity (here we cast from `any` to `GithubRelease`)
+
+    console.log("Issuing request for released artifacts metadata to", requestUrl);
+
+    const response: GithubRelease = await fetch(requestUrl, {
+            headers: { Accept: "application/vnd.github.v3+json" }
+        })
+        .then(res => res.json());
+
+    const artifact = response.assets.find(artifact => artifact.name === artifactFileName);
+
+    if (!artifact) return null;
+
+    return {
+        releaseName: response.name,
+        downloadUrl: artifact.browser_download_url
+    };
+
+    // We omit declaration of tremendous amount of fields that we are not using here
+    interface GithubRelease {
+        name: string;
+        assets: Array<{
+            name: string;
+            browser_download_url: string;
+        }>;
+    }
+}