Auto merge of #14307 - davidbarsky:davidbarsky/add-cargo-style-project-discovery-for-buck-and-bazel-sickos, r=Veykril

Add Cargo-style project discovery for Buck and Bazel Users

This feature requires the user to add a command that generates a `rust-project.json` from a set of files. Project discovery can be invoked in two ways:

1. At extension activation time, which includes the generated `rust-project.json` as part of the linkedProjects argument in `InitializeParams`.
2. Through a new command titled "rust-analyzer: Add current file to workspace", which makes use of a new, rust-analyzer-specific LSP request that adds the workspace without erasing any existing workspaces. Note that there is no mechanism to _remove_ workspaces other than "quit the rust-analyzer server".

Few notes:
-  I think that the command-running functionality _could_ merit being placed into its own extension (and expose it via extension contribution points) to provide build-system idiomatic progress reporting and status handling, but I haven't (yet) made an extension that does this nor does Buck expose this sort of functionality.
-  This approach would _just work_ for Bazel. I'll try and get the tool that's responsible for Buck integration open-sourced soon.
- On the testing side of things, I've used this in around my employer's Buck-powered monorepo and it's a nice experience. That being said, I can't think of an open-source repository where this can be tested in public, so you might need to trust me on this one.

I'd love to get feedback on:
- Naming of LSP extensions/new commands. I'm not too pleased with how "rust-analyzer: Add current file to workspace" is named, in that it's creating a _new_ workspace. I think that this command being added should be gated on `rust-analyzer.discoverProjectCommand` on being set, so I can add this in sequent commits.
- My Typescript. It's not particularly good.
- Suggestions on handling folders with _both_ Cargo and non-Cargo build systems and if I make activation a bit better.

(I previously tried to add this functionality entirely within rust-analyzer-the-LSP server itself, but matklad was right—an extension side approach is much, much easier.)

Author: bors

crates/rust-analyzer/src/config.rs

@@ -272,7 +272,6 @@ config_data! {
         /// The warnings will be indicated by a blue squiggly underline in code
         /// and a blue icon in the `Problems Panel`.
         diagnostics_warningsAsInfo: Vec<String> = "[]",
-
         /// These directories will be ignored by rust-analyzer. They are
         /// relative to the workspace root, and globs are not supported. You may
         /// also need to add the folders to Code's `files.watcherExclude`.
@@ -895,6 +894,15 @@ impl Config {
         }
     }
 
+    pub fn add_linked_projects(&mut self, linked_projects: Vec<ProjectJsonData>) {
+        let mut linked_projects = linked_projects
+            .into_iter()
+            .map(ManifestOrProjectJson::ProjectJson)
+            .collect::<Vec<ManifestOrProjectJson>>();
+
+        self.data.linkedProjects.append(&mut linked_projects);
+    }
+
     pub fn did_save_text_document_dynamic_registration(&self) -> bool {
         let caps = try_or_def!(self.caps.text_document.as_ref()?.synchronization.clone()?);
         caps.did_save == Some(true) && caps.dynamic_registration == Some(true)

crates/rust-analyzer/src/handlers.rs

@@ -46,6 +46,7 @@ use crate::{
 pub(crate) fn handle_workspace_reload(state: &mut GlobalState, _: ()) -> Result<()> {
     state.proc_macro_clients.clear();
     state.proc_macro_changed = false;
+
     state.fetch_workspaces_queue.request_op("reload workspace request".to_string());
     state.fetch_build_data_queue.request_op("reload workspace request".to_string());
     Ok(())

editors/code/package.json

@@ -199,6 +199,11 @@
                 "title": "Reload workspace",
                 "category": "rust-analyzer"
             },
+            {
+                "command": "rust-analyzer.addProject",
+                "title": "Add current file's crate to workspace",
+                "category": "rust-analyzer"
+            },
             {
                 "command": "rust-analyzer.reload",
                 "title": "Restart server",
@@ -428,6 +433,17 @@
                     "default": false,
                     "type": "boolean"
                 },
+                "rust-analyzer.discoverProjectCommand": {
+                    "markdownDescription": "Sets the command that rust-analyzer uses to generate `rust-project.json` files. This command should only be used\n if a build system like Buck or Bazel is also in use. The command must accept files as arguments and return \n a rust-project.json over stdout.",
+                    "default": null,
+                    "type": [
+                        "null",
+                        "array"
+                    ],
+                    "items": {
+                        "type": "string"
+                    }
+                },
                 "$generated-start": {},
                 "rust-analyzer.assist.emitMustUse": {
                     "markdownDescription": "Whether to insert #[must_use] when generating `as_` methods\nfor enum variants.",

editors/code/src/client.ts

@@ -6,7 +6,7 @@ import * as Is from "vscode-languageclient/lib/common/utils/is";
 import { assert } from "./util";
 import * as diagnostics from "./diagnostics";
 import { WorkspaceEdit } from "vscode";
-import { Config, substituteVSCodeVariables } from "./config";
+import { Config, prepareVSCodeConfig } from "./config";
 import { randomUUID } from "crypto";
 
 export interface Env {
@@ -95,7 +95,16 @@ export async function createClient(
                     const resp = await next(params, token);
                     if (resp && Array.isArray(resp)) {
                         return resp.map((val) => {
-                            return substituteVSCodeVariables(val);
+                            return prepareVSCodeConfig(val, (key, cfg) => {
+                                // we only want to set discovered workspaces on the right key
+                                // and if a workspace has been discovered.
+                                if (
+                                    key === "linkedProjects" &&
+                                    config.discoveredWorkspaces.length > 0
+                                ) {
+                                    cfg[key] = config.discoveredWorkspaces;
+                                }
+                            });
                         });
                     } else {
                         return resp;

editors/code/src/commands.ts

@@ -3,7 +3,7 @@ import * as lc from "vscode-languageclient";
 import * as ra from "./lsp_ext";
 import * as path from "path";
 
-import { Ctx, Cmd, CtxInit } from "./ctx";
+import { Ctx, Cmd, CtxInit, discoverWorkspace } from "./ctx";
 import { applySnippetWorkspaceEdit, applySnippetTextEdits } from "./snippets";
 import { spawnSync } from "child_process";
 import { RunnableQuickPick, selectRunnable, createTask, createArgs } from "./run";
@@ -749,6 +749,33 @@ export function reloadWorkspace(ctx: CtxInit): Cmd {
     return async () => ctx.client.sendRequest(ra.reloadWorkspace);
 }
 
+export function addProject(ctx: CtxInit): Cmd {
+    return async () => {
+        const discoverProjectCommand = ctx.config.discoverProjectCommand;
+        if (!discoverProjectCommand) {
+            return;
+        }
+
+        const workspaces: JsonProject[] = await Promise.all(
+            vscode.workspace.workspaceFolders!.map(async (folder): Promise<JsonProject> => {
+                const rustDocuments = vscode.workspace.textDocuments.filter(isRustDocument);
+                return discoverWorkspace(rustDocuments, discoverProjectCommand, {
+                    cwd: folder.uri.fsPath,
+                });
+            })
+        );
+
+        ctx.addToDiscoveredWorkspaces(workspaces);
+
+        // this is a workaround to avoid needing writing the `rust-project.json` into
+        // a workspace-level VS Code-specific settings folder. We'd like to keep the
+        // `rust-project.json` entirely in-memory.
+        await ctx.client?.sendNotification(lc.DidChangeConfigurationNotification.type, {
+            settings: "",
+        });
+    };
+}
+
 async function showReferencesImpl(
     client: LanguageClient | undefined,
     uri: string,

editors/code/src/config.ts

@@ -34,6 +34,7 @@ export class Config {
 
     constructor(ctx: vscode.ExtensionContext) {
         this.globalStorageUri = ctx.globalStorageUri;
+        this.discoveredWorkspaces = [];
         vscode.workspace.onDidChangeConfiguration(
             this.onDidChangeConfiguration,
             this,
@@ -55,6 +56,8 @@ export class Config {
         log.info("Using configuration", Object.fromEntries(cfg));
     }
 
+    public discoveredWorkspaces: JsonProject[];
+
     private async onDidChangeConfiguration(event: vscode.ConfigurationChangeEvent) {
         this.refreshLogging();
 
@@ -191,7 +194,7 @@ export class Config {
      * So this getter handles this quirk by not requiring the caller to use postfix `!`
      */
     private get<T>(path: string): T | undefined {
-        return substituteVSCodeVariables(this.cfg.get<T>(path));
+        return prepareVSCodeConfig(this.cfg.get<T>(path));
     }
 
     get serverPath() {
@@ -214,6 +217,10 @@ export class Config {
         return this.get<boolean>("trace.extension");
     }
 
+    get discoverProjectCommand() {
+        return this.get<string[] | undefined>("discoverProjectCommand");
+    }
+
     get cargoRunner() {
         return this.get<string | undefined>("cargoRunner");
     }
@@ -280,18 +287,32 @@ export class Config {
     }
 }
 
-export function substituteVSCodeVariables<T>(resp: T): T {
+// the optional `cb?` parameter is meant to be used to add additional
+// key/value pairs to the VS Code configuration. This needed for, e.g.,
+// including a `rust-project.json` into the `linkedProjects` key as part
+// of the configuration/InitializationParams _without_ causing VS Code
+// configuration to be written out to workspace-level settings. This is
+// undesirable behavior because rust-project.json files can be tens of
+// thousands of lines of JSON, most of which is not meant for humans
+// to interact with.
+export function prepareVSCodeConfig<T>(
+    resp: T,
+    cb?: (key: Extract<keyof T, string>, res: { [key: string]: any }) => void
+): T {
     if (Is.string(resp)) {
         return substituteVSCodeVariableInString(resp) as T;
     } else if (resp && Is.array<any>(resp)) {
         return resp.map((val) => {
-            return substituteVSCodeVariables(val);
+            return prepareVSCodeConfig(val);
         }) as T;
     } else if (resp && typeof resp === "object") {
         const res: { [key: string]: any } = {};
         for (const key in resp) {
             const val = resp[key];
-            res[key] = substituteVSCodeVariables(val);
+            res[key] = prepareVSCodeConfig(val);
+            if (cb) {
+                cb(key, res);
+            }
         }
         return res as T;
     }

editors/code/src/ctx.ts

@@ -2,12 +2,20 @@ import * as vscode from "vscode";
 import * as lc from "vscode-languageclient/node";
 import * as ra from "./lsp_ext";
 
-import { Config, substituteVSCodeVariables } from "./config";
+import { Config, prepareVSCodeConfig } from "./config";
 import { createClient } from "./client";
-import { isRustDocument, isRustEditor, LazyOutputChannel, log, RustEditor } from "./util";
+import {
+    executeDiscoverProject,
+    isRustDocument,
+    isRustEditor,
+    LazyOutputChannel,
+    log,
+    RustEditor,
+} from "./util";
 import { ServerStatusParams } from "./lsp_ext";
 import { PersistentState } from "./persistent_state";
 import { bootstrap } from "./bootstrap";
+import { ExecOptions } from "child_process";
 
 // We only support local folders, not eg. Live Share (`vlsl:` scheme), so don't activate if
 // only those are in use. We use "Empty" to represent these scenarios
@@ -41,6 +49,17 @@ export function fetchWorkspace(): Workspace {
         : { kind: "Workspace Folder" };
 }
 
+export async function discoverWorkspace(
+    files: readonly vscode.TextDocument[],
+    command: string[],
+    options: ExecOptions
+): Promise<JsonProject> {
+    const paths = files.map((f) => `"${f.uri.fsPath}"`).join(" ");
+    const joinedCommand = command.join(" ");
+    const data = await executeDiscoverProject(`${joinedCommand} ${paths}`, options);
+    return JSON.parse(data) as JsonProject;
+}
+
 export type CommandFactory = {
     enabled: (ctx: CtxInit) => Cmd;
     disabled?: (ctx: Ctx) => Cmd;
@@ -52,7 +71,7 @@ export type CtxInit = Ctx & {
 
 export class Ctx {
     readonly statusBar: vscode.StatusBarItem;
-    readonly config: Config;
+    config: Config;
     readonly workspace: Workspace;
 
     private _client: lc.LanguageClient | undefined;
@@ -169,7 +188,30 @@ export class Ctx {
                 };
             }
 
-            const initializationOptions = substituteVSCodeVariables(rawInitializationOptions);
+            const discoverProjectCommand = this.config.discoverProjectCommand;
+            if (discoverProjectCommand) {
+                const workspaces: JsonProject[] = await Promise.all(
+                    vscode.workspace.workspaceFolders!.map(async (folder): Promise<JsonProject> => {
+                        const rustDocuments = vscode.workspace.textDocuments.filter(isRustDocument);
+                        return discoverWorkspace(rustDocuments, discoverProjectCommand, {
+                            cwd: folder.uri.fsPath,
+                        });
+                    })
+                );
+
+                this.addToDiscoveredWorkspaces(workspaces);
+            }
+
+            const initializationOptions = prepareVSCodeConfig(
+                rawInitializationOptions,
+                (key, obj) => {
+                    // we only want to set discovered workspaces on the right key
+                    // and if a workspace has been discovered.
+                    if (key === "linkedProjects" && this.config.discoveredWorkspaces.length > 0) {
+                        obj["linkedProjects"] = this.config.discoveredWorkspaces;
+                    }
+                }
+            );
 
             this._client = await createClient(
                 this.traceOutputChannel,
@@ -251,6 +293,17 @@ export class Ctx {
         return this._serverPath;
     }
 
+    addToDiscoveredWorkspaces(workspaces: JsonProject[]) {
+        for (const workspace of workspaces) {
+            const index = this.config.discoveredWorkspaces.indexOf(workspace);
+            if (~index) {
+                this.config.discoveredWorkspaces[index] = workspace;
+            } else {
+                this.config.discoveredWorkspaces.push(workspace);
+            }
+        }
+    }
+
     private updateCommands(forceDisable?: "disable") {
         this.commandDisposables.forEach((disposable) => disposable.dispose());
         this.commandDisposables = [];

editors/code/src/lsp_ext.ts

@@ -43,6 +43,7 @@ export const relatedTests = new lc.RequestType<lc.TextDocumentPositionParams, Te
     "rust-analyzer/relatedTests"
 );
 export const reloadWorkspace = new lc.RequestType0<null, void>("rust-analyzer/reloadWorkspace");
+
 export const runFlycheck = new lc.NotificationType<{
     textDocument: lc.TextDocumentIdentifier | null;
 }>("rust-analyzer/runFlycheck");

editors/code/src/main.ts

@@ -153,6 +153,7 @@ function createCommands(): Record<string, CommandFactory> {
         memoryUsage: { enabled: commands.memoryUsage },
         shuffleCrateGraph: { enabled: commands.shuffleCrateGraph },
         reloadWorkspace: { enabled: commands.reloadWorkspace },
+        addProject: { enabled: commands.addProject },
         matchingBrace: { enabled: commands.matchingBrace },
         joinLines: { enabled: commands.joinLines },
         parentModule: { enabled: commands.parentModule },