Improved rust-analyzer tests (#3670)

This change adds `rust_analyzer` binaries to `rust_analyzer_toolchain`
so that it can be used with a new `//tools/rust_analyzer/validate`
target to ensure generated `rust-project.json` files are actually
correct.

Author: UebelAndre

.bazelci/presubmit.yml

@@ -430,12 +430,26 @@ tasks:
     platform: ubuntu2204
     run_targets:
       - "//test/rustfmt:rustfmt_failure_tester"
-  rust_analyzer_tests:
-    name: Rust-Analyzer Tests
+  rust_analyzer_tests_linux:
+    name: Rust-Analyzer Linux Tests
     platform: ubuntu2204
     run_targets:
       - "//tools/rust_analyzer:gen_rust_project"
+      - "//tools/rust_analyzer:discover_bazel_rust_project"
       - "//test/rust_analyzer:rust_analyzer_test"
+  rust_analyzer_tests_macos:
+    name: Rust-Analyzer Macos Tests
+    platform: macos_arm64
+    run_targets:
+      - "//tools/rust_analyzer:gen_rust_project"
+      - "//tools/rust_analyzer:discover_bazel_rust_project"
+      - "//test/rust_analyzer:rust_analyzer_test"
+  rust_analyzer_tests_windows:
+    name: Rust-Analyzer Windows Tests
+    platform: windows
+    run_targets:
+      - "//tools/rust_analyzer:gen_rust_project"
+      - "//tools/rust_analyzer:discover_bazel_rust_project"
   crate_universe_examples_ubuntu2204:
     name: Crate Universe Examples
     platform: ubuntu2204

docs/BUILD.bazel

@@ -101,18 +101,6 @@ stardoc(
     deps = [":all_docs"],
 )
 
-stardoc(
-    name = "rust_analyzer",
-    out = "src/rust_analyzer.md",
-    header_template = ":rust_analyzer.vm",
-    input = "@rules_rust//rust:defs.bzl",
-    symbol_names = [
-        "rust_analyzer_aspect",
-    ],
-    table_of_contents_template = "@stardoc//stardoc:templates/markdown_tables/table_of_contents.vm",
-    deps = [":all_docs"],
-)
-
 stardoc(
     name = "rust_clippy",
     out = "src/rust_clippy.md",
@@ -281,7 +269,6 @@ mdbook(
         ":crate_universe_workspace",
         ":providers",
         ":rust",
-        ":rust_analyzer",
         ":rust_bindgen",
         ":rust_bzlmod",
         ":rust_clippy",

docs/settings_table_of_contents.vm

@@ -1,13 +1,13 @@
-#[[
-## Overview
+# Rust Analyzer
 
 For [non-Cargo projects](https://rust-analyzer.github.io/manual.html#non-cargo-based-projects),
-[rust-analyzer](https://rust-analyzer.github.io/) depends on either a `rust-project.json` file 
-at the root of the project that describes its structure or on build system specific 
+[rust-analyzer](https://rust-analyzer.github.io/) depends on either a `rust-project.json` file
+at the root of the project that describes its structure or on build system specific
 [project auto-discovery](https://rust-analyzer.github.io/manual.html#rust-analyzer.workspace.discoverConfig).
 The `rust_analyzer` rules facilitate both approaches.
 
 ## rust-project.json approach
+
 ### Setup
 
 #### Bzlmod
@@ -95,26 +95,28 @@ to ensure a `rust-project.json` file is created and up to date when the editor i
             "runOptions": {
                 "runOn": "folderOpen"
             }
-        },
+        }
     ]
 }
 ```
 
 #### Alternative vscode option (prototype)
 
 Add the following to your bazelrc:
+
 ```
 build --@rules_rust//rust/settings:rustc_output_diagnostics=true --output_groups=+rust_lib_rustc_output,+rust_metadata_rustc_output
 ```
 
 Then you can use a prototype [rust-analyzer plugin](https://marketplace.visualstudio.com/items?itemName=MattStark.bazel-rust-analyzer) that automatically collects the outputs whenever you recompile.
 
 ## Project auto-discovery
+
 ### Setup
 
 Auto-discovery makes `rust-analyzer` behave in a Bazel project in a similar fashion to how it does
-in a Cargo project. This is achieved by generating a structure similar to what `rust-project.json` 
-contains but, instead of writing that to a file, the data gets piped to `rust-analyzer` directly 
+in a Cargo project. This is achieved by generating a structure similar to what `rust-project.json`
+contains but, instead of writing that to a file, the data gets piped to `rust-analyzer` directly
 through `stdout`. To use auto-discovery the `rust-analyzer` IDE settings must be configured similar to:
 
 ```json
@@ -130,7 +132,7 @@ through `stdout`. To use auto-discovery the `rust-analyzer` IDE settings must be
 ```
 
 The shell script passed to `discoverConfig.command` is typically meant to wrap the bazel rule invocation,
-primarily for muting `stderr` (because `rust-analyzer` will consider that an error has occurred if anything 
+primarily for muting `stderr` (because `rust-analyzer` will consider that an error has occurred if anything
 is passed through `stderr`) and, additionally, for specifying rule arguments. E.g:
 
 ```shell
@@ -150,7 +152,7 @@ enabled. The script path should be either absolute or relative to the project ro
 ### Workspace splitting
 
 The above configuration treats the entire project as a single workspace. However, large codebases might be
-too much to handle for `rust-analyzer` all at once. This can be addressed by splitting the codebase in 
+too much to handle for `rust-analyzer` all at once. This can be addressed by splitting the codebase in
 multiple workspaces by extending the `discoverConfig.command` setting:
 
 ```json
@@ -169,11 +171,9 @@ multiple workspaces by extending the `discoverConfig.command` setting:
 that gets opened.
 
 The root of the workspace will, in this configuration, be the package the crate currently being worked on
-belongs to. This means that only that package and its dependencies get built and indexed by `rust-analyzer`, 
-thus allowing for a smaller footprint. 
+belongs to. This means that only that package and its dependencies get built and indexed by `rust-analyzer`,
+thus allowing for a smaller footprint.
 
 `rust-analyzer` will switch workspaces whenever an out-of-tree file gets opened, essentially indexing that
-crate and its dependencies separately. A caveat of this is that *dependents* of the crate currently being 
+crate and its dependencies separately. A caveat of this is that _dependents_ of the crate currently being
 worked on are not indexed and won't be tracked by `rust-analyzer`.
-
-]]#

docs/rust_analyzer.vm renamed to docs/src/rust_analyzer.md

@@ -97,6 +97,27 @@ def BUILD_for_cargo(target_triple):
         binary_ext = system_to_binary_ext(target_triple.system),
     )
 
+_build_file_for_rust_analyzer_template = """\
+filegroup(
+    name = "rust_analyzer",
+    srcs = ["bin/rust-analyzer{binary_ext}"],
+    visibility = ["//visibility:public"],
+)
+"""
+
+def BUILD_for_rust_analyzer(target_triple):
+    """Emits a BUILD file for the rust-analyzer archive.
+
+    Args:
+        target_triple (str): The triple of the target platform
+
+    Returns:
+        str: The contents of a BUILD file
+    """
+    return _build_file_for_rust_analyzer_template.format(
+        binary_ext = system_to_binary_ext(target_triple.system),
+    )
+
 _build_file_for_rustfmt_template = """\
 load("@rules_shell//shell:sh_binary.bzl", "sh_binary")
 
@@ -396,6 +417,30 @@ def BUILD_for_toolchain(
         target_settings = target_settings_value,
     )
 
+def load_rust_analyzer(ctx, target_triple, version, iso_date):
+    """Loads a rust-analyzer binary and yields corresponding BUILD for it
+
+    Args:
+        ctx (repository_ctx): The repository rule's context object.
+        target_triple (struct): The platform triple to download rust-analyzer for.
+        version (str): The version or channel of rust-analyzer.
+        iso_date (str): The date of the tool (or None, if the version is a specific version).
+
+    Returns:
+        Tuple[str, Dict[str, str]]: The BUILD file contents for this rust-analyzer binary and sha256 of its artifact.
+    """
+
+    sha256 = load_arbitrary_tool(
+        ctx,
+        iso_date = iso_date,
+        target_triple = target_triple,
+        tool_name = "rust-analyzer",
+        tool_subdirectories = ["rust-analyzer-preview"],
+        version = version,
+    )
+
+    return BUILD_for_rust_analyzer(target_triple), sha256
+
 def load_rustfmt(ctx, target_triple, version, iso_date):
     """Loads a rustfmt binary and yields corresponding BUILD for it
 
@@ -566,17 +611,19 @@ load("@rules_rust//rust:toolchain.bzl", "rust_analyzer_toolchain")
 rust_analyzer_toolchain(
     name = "{name}",
     proc_macro_srv = {proc_macro_srv},
+    rust_analyzer = {rust_analyzer},
     rustc = "{rustc}",
     rustc_srcs = "//lib/rustlib/src:rustc_srcs",
     visibility = ["//visibility:public"],
 )
 """
 
-def BUILD_for_rust_analyzer_toolchain(name, rustc, proc_macro_srv):
+def BUILD_for_rust_analyzer_toolchain(name, rustc, proc_macro_srv, rust_analyzer = None):
     return _build_file_for_rust_analyzer_toolchain_template.format(
         name = name,
         rustc = rustc,
         proc_macro_srv = repr(proc_macro_srv),
+        rust_analyzer = repr(rust_analyzer) if rust_analyzer else "None",
     )
 
 _build_file_for_rustfmt_toolchain_template = """\

rust/private/repository_utils.bzl

@@ -293,15 +293,31 @@ def _create_single_crate(ctx, attrs, info):
         crate["proc_macro_dylib_path"] = _EXEC_ROOT_TEMPLATE + info.proc_macro_dylib.path
     return crate
 
+def _rlocationpath(file, workspace_name):
+    if file.short_path.startswith("../"):
+        return file.short_path[len("../"):]
+
+    return "{}/{}".format(workspace_name, file.short_path)
+
 def _rust_analyzer_toolchain_impl(ctx):
+    make_variable_info = platform_common.TemplateVariableInfo({
+        "RUST_ANALYZER": ctx.file.rust_analyzer.path,
+        "RUST_ANALYZER_RLOCATIONPATH": _rlocationpath(ctx.file.rust_analyzer, ctx.workspace_name),
+    })
+
     toolchain = platform_common.ToolchainInfo(
         proc_macro_srv = ctx.executable.proc_macro_srv,
+        rust_analyzer = ctx.executable.rust_analyzer,
         rustc = ctx.executable.rustc,
         rustc_srcs = ctx.attr.rustc_srcs,
         rustc_srcs_path = ctx.attr.rustc_srcs_path,
+        make_variables = make_variable_info,
     )
 
-    return [toolchain]
+    return [
+        make_variable_info,
+        toolchain,
+    ]
 
 rust_analyzer_toolchain = rule(
     implementation = _rust_analyzer_toolchain_impl,
@@ -313,6 +329,12 @@ rust_analyzer_toolchain = rule(
             executable = True,
             allow_single_file = True,
         ),
+        "rust_analyzer": attr.label(
+            doc = "The path to a `rust-analyzer` binary.",
+            cfg = "exec",
+            executable = True,
+            allow_single_file = True,
+        ),
         "rustc": attr.label(
             doc = "The path to a `rustc` binary.",
             cfg = "exec",
@@ -331,6 +353,33 @@ rust_analyzer_toolchain = rule(
     },
 )
 
+def _current_rust_analyzer_toolchain_impl(ctx):
+    toolchain = ctx.toolchains[str(Label("@rules_rust//rust/rust_analyzer:toolchain_type"))]
+
+    files = []
+    if toolchain.rust_analyzer:
+        files.append(toolchain.rust_analyzer)
+    if toolchain.proc_macro_srv:
+        files.append(toolchain.proc_macro_srv)
+    if toolchain.rustc:
+        files.append(toolchain.rustc)
+
+    return [
+        toolchain,
+        toolchain.make_variables,
+        DefaultInfo(
+            files = depset(files),
+        ),
+    ]
+
+current_rust_analyzer_toolchain = rule(
+    doc = "A rule for exposing the current registered `rust_analyzer_toolchain`.",
+    implementation = _current_rust_analyzer_toolchain_impl,
+    toolchains = [
+        str(Label("@rules_rust//rust/rust_analyzer:toolchain_type")),
+    ],
+)
+
 def _rust_analyzer_detect_sysroot_impl(ctx):
     rust_analyzer_toolchain = ctx.toolchains[Label("@rules_rust//rust/rust_analyzer:toolchain_type")]
 

rust/private/rust_analyzer.bzl

@@ -25,6 +25,7 @@ load(
     "load_cargo",
     "load_clippy",
     "load_llvm_tools",
+    "load_rust_analyzer",
     "load_rust_compiler",
     "load_rust_src",
     "load_rust_stdlib",
@@ -786,10 +787,24 @@ def _rust_analyzer_toolchain_tools_repository_impl(repository_ctx):
         build_contents.append(BUILD_for_rust_analyzer_proc_macro_srv(host_triple))
         proc_macro_srv = "//:rust_analyzer_proc_macro_srv"
 
+    # Load rust-analyzer binary from official Rust distribution
+    rust_analyzer = None
+    rust_analyzer_content, rust_analyzer_sha256 = load_rust_analyzer(
+        ctx = repository_ctx,
+        target_triple = host_triple,
+        version = version,
+        iso_date = iso_date,
+    )
+    if rust_analyzer_content:
+        build_contents.append(rust_analyzer_content)
+        sha256s.update(rust_analyzer_sha256)
+        rust_analyzer = "//:rust_analyzer"
+
     build_contents.append(BUILD_for_rust_analyzer_toolchain(
         name = "rust_analyzer_toolchain",
         rustc = rustc,
         proc_macro_srv = proc_macro_srv,
+        rust_analyzer = rust_analyzer,
     ))
 
     repository_ctx.file("BUILD.bazel", "\n".join(build_contents))

rust/repositories.bzl

@@ -9,7 +9,11 @@ load("@rules_cc//cc/common:cc_info.bzl", "CcInfo")
 load("//rust/platform:triple.bzl", "triple")
 load("//rust/private:common.bzl", "rust_common")
 load("//rust/private:lto.bzl", "RustLtoInfo")
-load("//rust/private:rust_analyzer.bzl", _rust_analyzer_toolchain = "rust_analyzer_toolchain")
+load(
+    "//rust/private:rust_analyzer.bzl",
+    _current_rust_analyzer_toolchain = "current_rust_analyzer_toolchain",
+    _rust_analyzer_toolchain = "rust_analyzer_toolchain",
+)
 load(
     "//rust/private:rustfmt.bzl",
     _current_rustfmt_toolchain = "current_rustfmt_toolchain",
@@ -27,6 +31,7 @@ load(
 load("//rust/settings:incompatible.bzl", "IncompatibleFlagInfo")
 
 rust_analyzer_toolchain = _rust_analyzer_toolchain
+current_rust_analyzer_toolchain = _current_rust_analyzer_toolchain
 rustfmt_toolchain = _rustfmt_toolchain
 current_rustfmt_toolchain = _current_rustfmt_toolchain
 

rust/toolchain.bzl

@@ -1,3 +1,4 @@
+load("//rust/private:rust_analyzer.bzl", "current_rust_analyzer_toolchain")
 load("//rust/private:rustfmt.bzl", "current_rustfmt_toolchain")
 load("//rust/private:toolchain_utils.bzl", "current_rust_toolchain", "toolchain_files", "toolchain_files_for_target")
 
@@ -46,6 +47,11 @@ current_rustfmt_toolchain(
     name = "current_rustfmt_toolchain",
 )
 
+current_rust_analyzer_toolchain(
+    name = "current_rust_analyzer_toolchain",
+    tags = ["manual"],
+)
+
 toolchain_files_for_target(
     name = "current_rustfmt_toolchain_for_target",
     toolchain_files = ":current_rustfmt_toolchain",

rust/toolchain/BUILD.bazel

@@ -103,6 +103,9 @@ function rust_analyzer_test() {
         RUST_LOG="${rust_log}" bazel run "@rules_rust//tools/rust_analyzer:gen_rust_project"
     fi
 
+    echo "Validating rust-project.json..."
+    bazel run "@rules_rust//tools/rust_analyzer:validate" -- rust-project.json
+    
     echo "Generating auto-discovery.json..."
     RUST_LOG="${rust_log}" bazel run "@rules_rust//tools/rust_analyzer:discover_bazel_rust_project" > auto-discovery.json