rust-analyzer discoverConfig integration (#3073)

Adds a target that can be used for project auto-discovery by using the
`discoverConfig` settings as described in the `rust-analyzer` user
manual.

Unlike the `gen_rust_project` target, this can be used for dynamic
project discovery, and passing `{arg}` to `discoverConfig.command` can
split big repositories into multiple, smaller workspaces that
`rust-analyzer` switches between as needed. Large repositories can make
it OOM.

At amo, we've used a similar implementation for a while with great
success, which is why we figured we might upstream it. The changes also
include two additional output groups to ensure that proc-macros and
build script targets are built, as `rust-analyzer` depends on these to
provide complete IDE support.

Additionally, the PR makes use of the `output_base` value in `bazel`
invocations. We found it helpful to have tools such as `rust-analyzer`
and `clippy` run on a separate bazel server than the one used for
building. And a `config_group` argument was added to provide the ability
to provide a config group to `bazel` invocations.

An attempt to get codelens actions to work was done as well,
particularly around tests and binaries. They seem to work, but I'm not
100% sure whether the approach taken is the right one.

Closes #2755 .

---------

Co-authored-by: Krasimir Georgiev <[email protected]>

Author: bobozaur

.bazelci/presubmit.yml

@@ -885,7 +885,7 @@ tasks:
     build_flags:
       - "--compile_one_dependency"
     build_targets:
-      - "tools/rust_analyzer/main.rs"
+      - "tools/rust_analyzer/bin/gen_rust_project.rs"
   extensions_bindgen_linux:
     platform: ubuntu2004
     name: Extensions Bindgen

.gitignore

@@ -30,6 +30,9 @@ user.bazelrc
 .vscode
 *.code-workspace
 
+# zed
+.zed
+
 # JetBrains
 .idea
 .idea/**

MODULE.bazel

@@ -22,6 +22,7 @@ use_repo(
     internal_deps,
     "rrra",
     "rrra__anyhow-1.0.71",
+    "rrra__camino-1.1.9",
     "rrra__clap-4.3.11",
     "rrra__env_logger-0.10.0",
     "rrra__itertools-0.11.0",

docs/rust_analyzer.vm

@@ -2,10 +2,12 @@
 ## Overview
 
 For [non-Cargo projects](https://rust-analyzer.github.io/manual.html#non-cargo-based-projects),
-[rust-analyzer](https://rust-analyzer.github.io/) depends on a `rust-project.json` file at the
-root of the project that describes its structure. The `rust_analyzer` rule facilitates generating
-such a file.
+[rust-analyzer](https://rust-analyzer.github.io/) depends on either a `rust-project.json` file 
+at the root of the project that describes its structure or on build system specific 
+[project auto-discovery](https://rust-analyzer.github.io/manual.html#rust-analyzer.workspace.discoverConfig).
+The `rust_analyzer` rules facilitate both approaches.
 
+## rust-project.json approach
 ### Setup
 
 First, ensure `rules_rust` is setup in your workspace. By default, `rust_register_toolchains` will
@@ -84,4 +86,71 @@ build --@rules_rust//rust/settings:rustc_output_diagnostics=true --output_groups
 
 Then you can use a prototype [rust-analyzer plugin](https://marketplace.visualstudio.com/items?itemName=MattStark.bazel-rust-analyzer) that automatically collects the outputs whenever you recompile.
 
-]]#
+## Project auto-discovery
+### Setup
+
+Auto-discovery makes `rust-analyzer` behave in a Bazel project in a similar fashion to how it does
+in a Cargo project. This is achieved by generating a structure similar to what `rust-project.json` 
+contains but, instead of writing that to a file, the data gets piped to `rust-analyzer` directly 
+through `stdout`. To use auto-discovery the `rust-analyzer` IDE settings must be configured similar to:
+
+```json
+"rust-analyzer": {
+    "workspace": {
+        "discoverConfig": {
+            "command": ["discover_bazel_rust_project.sh"],
+            "progressLabel": "rust_analyzer",
+            "filesToWatch": ["BUILD", "BUILD.bazel", "MODULE.bazel"]
+        }
+    }
+}
+```
+
+The shell script passed to `discoverConfig.command` is typically meant to wrap the bazel rule invocation,
+primarily for muting `stderr` (because `rust-analyzer` will consider that an error has occurred if anything 
+is passed through `stderr`) and, additionally, for specifying rule arguments. E.g:
+
+```shell
+#!/usr/bin/bash
+
+bazel \
+    run \
+    @rules_rust//tools/rust_analyzer:discover_bazel_rust_project -- \
+    --bazel_startup_option=--output_base=~/ide_bazel \
+    --bazel_arg=--watchfs \
+    ${1:+"$1"} 2>/dev/null
+```
+
+The script above also handles an optional CLI argument which gets passed when workspace splitting is
+enabled. The script path should be either absolute or relative to the project root.
+
+### Workspace splitting
+
+The above configuration treats the entire project as a single workspace. However, large codebases might be
+too much to handle for `rust-analyzer` all at once. This can be addressed by splitting the codebase in 
+multiple workspaces by extending the `discoverConfig.command` setting:
+
+```json
+"rust-analyzer": {
+    "workspace": {
+        "discoverConfig": {
+            "command": ["discover_bazel_rust_project.sh", "{arg}"],
+            "progressLabel": "rust_analyzer",
+            "filesToWatch": ["BUILD", "BUILD.bazel", "MODULE.bazel"]
+        }
+    }
+}
+```
+
+`{arg}` acts as a placeholder that `rust-analyzer` replaces with the path of the source / build file
+that gets opened.
+
+The root of the workspace will, in this configuration, be the package the crate currently being worked on
+belongs to. This means that only that package and its dependencies get built and indexed by `rust-analyzer`, 
+thus allowing for a smaller footprint. 
+
+`rust-analyzer` will switch workspaces whenever an out-of-tree file gets opened, essentially indexing that
+crate and its dependencies separately. A caveat of this is that *dependents* of the crate currently being 
+worked on are not indexed and won't be tracked by `rust-analyzer`.
+
+]]#

extensions/prost/private/prost.bzl

@@ -308,14 +308,18 @@ def _rust_prost_aspect_impl(target, ctx):
     # https://github.com/rust-analyzer/rust-analyzer/blob/2021-11-15/crates/project_model/src/workspace.rs#L529-L531
     cfgs = ["test", "debug_assertions"]
 
+    build_info_out_dirs = [dep_variant_info.build_info.out_dir] if dep_variant_info.build_info != None and dep_variant_info.build_info.out_dir != None else None
+
     rust_analyzer_info = write_rust_analyzer_spec_file(ctx, ctx.rule.attr, ctx.label, RustAnalyzerInfo(
         aliases = {},
         crate = dep_variant_info.crate_info,
         cfgs = cfgs,
         env = dep_variant_info.crate_info.rustc_env,
         deps = rust_analyzer_deps,
         crate_specs = depset(transitive = [dep.crate_specs for dep in rust_analyzer_deps]),
-        proc_macro_dylib_path = None,
+        proc_macro_dylibs = depset(transitive = [dep.proc_macro_dylibs for dep in rust_analyzer_deps]),
+        build_info_out_dirs = depset(direct = build_info_out_dirs, transitive = [dep.build_info_out_dirs for dep in rust_analyzer_deps]),
+        proc_macro_dylib = None,
         build_info = dep_variant_info.build_info,
     ))
 

rust/private/providers.bzl

@@ -159,12 +159,14 @@ RustAnalyzerInfo = provider(
     fields = {
         "aliases": "Dict[RustAnalyzerInfo, String]: Replacement names these targets should be known as in Rust code",
         "build_info": "BuildInfo: build info for this crate if present",
+        "build_info_out_dirs": "Depset[File]: transitive closure of build script out dirs",
         "cfgs": "List[String]: features or other compilation `--cfg` settings",
         "crate": "CrateInfo: Crate information.",
-        "crate_specs": "Depset[File]: transitive closure of OutputGroupInfo files",
+        "crate_specs": "Depset[File]: transitive closure of crate spec files",
         "deps": "List[RustAnalyzerInfo]: direct dependencies",
         "env": "Dict[String: String]: Environment variables, used for the `env!` macro",
-        "proc_macro_dylib_path": "File: compiled shared library output of proc-macro rule",
+        "proc_macro_dylib": "File: if this is a proc-macro target, the shared library output",
+        "proc_macro_dylibs": "Depset[File]: transitive closure of proc-macro shared library files",
     },
 )
 

rust/private/rust_analyzer.bzl

@@ -54,7 +54,9 @@ def write_rust_analyzer_spec_file(ctx, attrs, owner, base_info):
         env = base_info.env,
         deps = base_info.deps,
         crate_specs = depset(direct = [crate_spec], transitive = [base_info.crate_specs]),
-        proc_macro_dylib_path = base_info.proc_macro_dylib_path,
+        proc_macro_dylibs = depset(transitive = [base_info.proc_macro_dylibs]),
+        build_info_out_dirs = depset(transitive = [base_info.build_info_out_dirs]),
+        proc_macro_dylib = base_info.proc_macro_dylib,
         build_info = base_info.build_info,
     )
 
@@ -135,30 +137,40 @@ def _rust_analyzer_aspect_impl(target, ctx):
         if aliased_target.label in labels_to_rais:
             aliases[labels_to_rais[aliased_target.label]] = aliased_name
 
+    proc_macro_dylib = find_proc_macro_dylib(toolchain, target)
+    proc_macro_dylibs = [proc_macro_dylib] if proc_macro_dylib else None
+    build_info_out_dirs = [build_info.out_dir] if build_info != None and build_info.out_dir != None else None
+
     rust_analyzer_info = write_rust_analyzer_spec_file(ctx, ctx.rule.attr, ctx.label, RustAnalyzerInfo(
         aliases = aliases,
         crate = crate_info,
         cfgs = cfgs,
         env = crate_info.rustc_env,
         deps = dep_infos,
         crate_specs = depset(transitive = [dep.crate_specs for dep in dep_infos]),
-        proc_macro_dylib_path = find_proc_macro_dylib_path(toolchain, target),
+        proc_macro_dylibs = depset(direct = proc_macro_dylibs, transitive = [dep.proc_macro_dylibs for dep in dep_infos]),
+        build_info_out_dirs = depset(direct = build_info_out_dirs, transitive = [dep.build_info_out_dirs for dep in dep_infos]),
+        proc_macro_dylib = proc_macro_dylib,
         build_info = build_info,
     ))
 
     return [
         rust_analyzer_info,
-        OutputGroupInfo(rust_analyzer_crate_spec = rust_analyzer_info.crate_specs),
+        OutputGroupInfo(
+            rust_analyzer_crate_spec = rust_analyzer_info.crate_specs,
+            rust_analyzer_proc_macro_dylib = rust_analyzer_info.proc_macro_dylibs,
+            rust_analyzer_src = rust_analyzer_info.build_info_out_dirs,
+        ),
     ]
 
-def find_proc_macro_dylib_path(toolchain, target):
-    """Find the proc_macro_dylib_path of target. Returns None if target crate is not type proc-macro.
+def find_proc_macro_dylib(toolchain, target):
+    """Find the proc_macro_dylib of target. Returns None if target crate is not type proc-macro.
 
     Args:
         toolchain: The current rust toolchain.
         target: The current target.
     Returns:
-        (path): The path to the proc macro dylib, or None if this crate is not a proc-macro.
+        (File): The path to the proc macro dylib, or None if this crate is not a proc-macro.
     """
     if rust_common.crate_info in target:
         crate_info = target[rust_common.crate_info]
@@ -174,7 +186,7 @@ def find_proc_macro_dylib_path(toolchain, target):
     for action in target.actions:
         for output in action.outputs.to_list():
             if output.extension == dylib_ext[1:]:
-                return output.path
+                return output
 
     # Failed to find the dylib path inside a proc-macro crate.
     # TODO: Should this be an error?
@@ -188,7 +200,7 @@ rust_analyzer_aspect = aspect(
 )
 
 # Paths in the generated JSON file begin with one of these placeholders.
-# The gen_rust_project driver will replace them with absolute paths.
+# The `rust-analyzer` driver will replace them with absolute paths.
 _WORKSPACE_TEMPLATE = "__WORKSPACE__/"
 _EXEC_ROOT_TEMPLATE = "__EXEC_ROOT__/"
 _OUTPUT_BASE_TEMPLATE = "__OUTPUT_BASE__/"
@@ -220,6 +232,7 @@ def _create_single_crate(ctx, attrs, info):
     crate["edition"] = info.crate.edition
     crate["env"] = {}
     crate["crate_type"] = info.crate.type
+    crate["is_test"] = info.crate.is_test
 
     # Switch on external/ to determine if crates are in the workspace or remote.
     # TODO: Some folks may want to override this for vendored dependencies.
@@ -230,6 +243,14 @@ def _create_single_crate(ctx, attrs, info):
     crate["root_module"] = path_prefix + info.crate.root.path
     crate["source"] = {"exclude_dirs": [], "include_dirs": []}
 
+    # We're only interested in the build info for local crates as these are the
+    # only ones for which we want build file watching and code lens runnables support.
+    if not is_external and not is_generated:
+        crate["build"] = {
+            "build_file": _WORKSPACE_TEMPLATE + ctx.build_file_path,
+            "label": ctx.label.package + ":" + ctx.label.name,
+        }
+
     if is_generated:
         srcs = getattr(ctx.rule.files, "srcs", [])
         src_map = {src.short_path: src for src in srcs if src.is_source}
@@ -268,8 +289,8 @@ def _create_single_crate(ctx, attrs, info):
     crate["cfg"] = info.cfgs
     toolchain = find_toolchain(ctx)
     crate["target"] = (_EXEC_ROOT_TEMPLATE + toolchain.target_json.path) if toolchain.target_json else toolchain.target_flag_value
-    if info.proc_macro_dylib_path != None:
-        crate["proc_macro_dylib_path"] = _EXEC_ROOT_TEMPLATE + info.proc_macro_dylib_path
+    if info.proc_macro_dylib != None:
+        crate["proc_macro_dylib_path"] = _EXEC_ROOT_TEMPLATE + info.proc_macro_dylib.path
     return crate
 
 def _rust_analyzer_toolchain_impl(ctx):

test/rust_analyzer/auto_discovery_static_and_shared_lib_test/BUILD.bazel

@@ -0,0 +1,42 @@
+load(
+    "@rules_rust//rust:defs.bzl",
+    "rust_shared_library",
+    "rust_static_library",
+    "rust_test",
+)
+
+rust_shared_library(
+    name = "greeter_cdylib",
+    srcs = [
+        "greeter.rs",
+        "shared_lib.rs",
+    ],
+    crate_root = "shared_lib.rs",
+    edition = "2018",
+)
+
+rust_static_library(
+    name = "greeter_staticlib",
+    srcs = [
+        "greeter.rs",
+        "static_lib.rs",
+    ],
+    crate_root = "static_lib.rs",
+    edition = "2018",
+)
+
+rust_test(
+    name = "auto_discovery_json_test",
+    srcs = ["auto_discovery_json_test.rs"],
+    data = [":auto-discovery.json"],
+    edition = "2018",
+    env = {"AUTO_DISCOVERY_JSON": "$(rootpath :auto-discovery.json)"},
+    # This target is tagged as manual since it's not expected to pass in
+    # contexts outside of `//test/rust_analyzer:rust_analyzer_test`. Run
+    # that target to execute this test.
+    tags = ["manual"],
+    deps = [
+        "//test/rust_analyzer/3rdparty/crates:serde",
+        "//test/rust_analyzer/3rdparty/crates:serde_json",
+    ],
+)

test/rust_analyzer/auto_discovery_static_and_shared_lib_test/auto_discovery_json_test.rs

@@ -0,0 +1,57 @@
+#[cfg(test)]
+mod tests {
+    use serde::Deserialize;
+    use std::env;
+    use std::path::PathBuf;
+
+    #[derive(Deserialize)]
+    #[serde(tag = "kind")]
+    #[serde(rename_all = "snake_case")]
+    enum DiscoverProject {
+        Finished { project: Project },
+        Progress {},
+    }
+
+    #[derive(Deserialize)]
+    struct Project {
+        crates: Vec<Crate>,
+    }
+
+    #[derive(Deserialize)]
+    struct Crate {
+        display_name: String,
+        root_module: String,
+    }
+
+    #[test]
+    fn test_static_and_shared_lib() {
+        let rust_project_path = PathBuf::from(env::var("AUTO_DISCOVERY_JSON").unwrap());
+        let content = std::fs::read_to_string(&rust_project_path)
+            .unwrap_or_else(|_| panic!("couldn't open {:?}", &rust_project_path));
+        println!("{}", content);
+
+        for line in content.lines() {
+            let discovery: DiscoverProject =
+                serde_json::from_str(line).expect("Failed to deserialize discovery JSON");
+
+            let project = match discovery {
+                DiscoverProject::Finished { project } => project,
+                DiscoverProject::Progress {} => continue,
+            };
+
+            let cdylib = project
+                .crates
+                .iter()
+                .find(|c| &c.display_name == "greeter_cdylib")
+                .unwrap();
+            assert!(cdylib.root_module.ends_with("/shared_lib.rs"));
+
+            let staticlib = project
+                .crates
+                .iter()
+                .find(|c| &c.display_name == "greeter_staticlib")
+                .unwrap();
+            assert!(staticlib.root_module.ends_with("/static_lib.rs"));
+        }
+    }
+}