Auto merge of rust-lang#134767 - Bryanskiy:dylibs-3, r=petrochenkov

Initial support for dynamically linked crates

This PR is an initial implementation of [rust-lang/rfcs#3435](rust-lang/rfcs#3435) proposal.
### component 1: interface generator

Interface generator - a tool for generating a stripped version of crate source code. The interface is like a C header, where all function bodies are omitted. For example, initial crate:

```rust
#[export]
#[repr(C)]
pub struct S {
   pub x: i32
}
#[export]
pub extern "C" fn foo(x: S) {
   m1::bar(x);
}

pub fn bar(x: crate::S) {
    // some computations
}
```

generated interface:

```rust
#[export]
#[repr(C)]
pub struct S {
    pub x: i32,
}

#[export]
pub extern "C" fn foo(x: S);

pub fn bar(x: crate::S);
```

The interface generator was implemented as part of the pretty-printer. Ideally interface should only contain exportable items, but here is the first problem:
-  pass for determining exportable items relies on privacy information, which is totally available only in HIR
- HIR pretty-printer uses pseudo-code(at least for attributes)

So, the interface generator was implemented in AST. This has led to the fact that non-exportable items cannot be filtered out, but I don't think this is a major issue at the moment.

To emit an interface use a new `sdylib` crate type which is basically the same as `dylib`, but it doesn't contain metadata, and also produces the interface as a second artifact. The current interface name is `lib{crate_name}.rs`.
#### Why was it decided to use a design with an auto-generated interface?

One of the main objectives of this proposal is to allow building the library and the application with different compiler versions. This requires either a metadata format compatible across rustc versions or some form of a source code. The option with a stable metadata format has not been investigated in detail, but it is not part of RFC either. Here is the the related discussion: rust-lang/rfcs#3435 (comment)

Original proposal suggests using the source code for the dynamic library and all its dependencies. Metadata is obtained from `cargo check`. I decided to use interface files since it is more or less compatible with the original proposal, but also allows users to hide the source code.
##### Regarding the design with interfaces

in Rust, files generally do not have a special meaning, unlike C++. A translation unit i.e. a crate is not a single file, it consists of modules. Modules, in turn, can be declared either in one file or divided into several. That's why the "interface file" isn't a very coherent concept in Rust. I would like to avoid adding an additional level of complexity for users until it is proven necessary. Therefore, the initial plan was to make the interfaces completely invisible to users i. e. make them auto-generated. I also planned to put them in the dylib, but this has not been done yet. (since the PR is already big enough, I decided to postpone it)

There is one concern, though, which has not yet been investigated(rust-lang#134767 (comment)):

> Compiling the interface as pretty-printed source code doesn't use correct macro hygiene (mostly relevant to macros 2.0, stable macros do not affect item hygiene).  I don't have much hope for encoding hygiene data in any stable way, we should rather support a way for the interface file to be provided manually, instead of being auto-generated, if there are any non-trivial requirements.
### component 2: crate loader

When building dynamic dependencies, the crate loader searches for the interface in the file system, builds the interface without codegen and loads it's metadata. Routing rules for interface files are almost the same as for `rlibs` and `dylibs`. Firstly, the compiler checks `extern` options and then tries to deduce the path himself.

Here are the code and commands that corresponds to the compilation process:

```rust
// simple-lib.rs
#![crate_type = "sdylib"]

#[extern]
pub extern "C" fn foo() -> i32 {
    42
}
```

```rust
// app.rs
extern crate simple_lib;

fn main() {
    assert!(simple_lib::foo(), 42);
}
```

```
// Generate interface, build library.
rustc +toolchain1 lib.rs

// Build app. Perhaps with a different compiler version.
rustc +toolchain2 app.rs -L.
```

P.S. The interface name/format and rules for file system routing can be changed further.
### component 3: exportable items collector

Query for collecting exportable items. Which items are exportable is defined [here](https://github.com/m-ou-se/rfcs/blob/export/text/0000-export.md#the-export-attribute) .
### component 4: "stable" mangling scheme

The mangling scheme proposed in the RFC consists of two parts: a mangled item path and a hash of the signature.
#### mangled item path

For the first part of the symbol it has been decided to reuse the `v0` mangling scheme as it much less dependent on compiler internals compared to the `legacy` scheme.

The exception is disambiguators (https://doc.rust-lang.org/rustc/symbol-mangling/v0.html#disambiguator):

For example, during symbol mangling rustc uses a special index to distinguish between two impls of the same type in the same module(See `DisambiguatedDefPathData`). The calculation of this index may depend on private items, but private items should not affect the ABI. Example:

```rust
#[export]
#[repr(C)]
pub struct S<T>(pub T);

struct S1;
pub struct S2;

impl S<S1> {
    extern "C" fn foo() -> i32 {
        1
    }
}

#[export]
impl S<S2> {
    // Different symbol names can be generated for this item
    // when compiling the interface and source code.
    pub extern "C" fn foo() -> i32 {
        2
    }
}
```

In order to make disambiguation independent of the compiler version we can assign an id to each impl according to their relative order in the source code.

The second example is `StableCrateId` which is used to disambiguate different crates. `StableCrateId` consists of crate name, `-Cmetadata` arguments and compiler version. At the moment, I have decided to keep only the crate name, but a more consistent approach to crate disambiguation could be added in the future.

Actually, there are more cases where such disambiguation can be used. For instance, when mangling internal rustc symbols, but it also hasn't been investigated in detail yet.
#### hash of the signature

Exportable functions from stable dylibs can be called from safe code. In order to provide type safety, 128 bit hash with relevant type information is appended to the symbol ([description from RFC](https://github.com/m-ou-se/rfcs/blob/export/text/0000-export.md#name-mangling-and-safety)). For now, it includes:

- hash of the type name for primitive types
- for ADT types with public fields the implementation follows [this](https://github.com/m-ou-se/rfcs/blob/export/text/0000-export.md#types-with-public-fields) rules

`#[export(unsafe_stable_abi = "hash")]` syntax for ADT types with private fields is not yet implemented.

Type safety is a subtle thing here. I used the approach from RFC, but there is the ongoing research project about it. [https://rust-lang.github.io/rust-project-goals/2025h1/safe-linking.html](https://rust-lang.github.io/rust-project-goals/2025h1/safe-linking.html)

### Unresolved questions

Interfaces:
1. Move the interface generator to HIR and add an exportable items filter.
2. Compatibility of auto-generated interfaces and macro hygiene.
3. There is an open issue with interface files compilation: rust-lang#134767 (comment)
4. Put an interface into a dylib.

Mangling scheme:
1. Which information is required to ensure type safety and how should it be encoded? ([https://rust-lang.github.io/rust-project-goals/2025h1/safe-linking.html](https://rust-lang.github.io/rust-project-goals/2025h1/safe-linking.html))
2. Determine all other possible cases, where path disambiguation is used. Make it compiler independent.

We also need a semi-stable API to represent types. For example, the order of fields in the `VariantDef` must be stable. Or a semi-stable representation for AST, which ensures that the order of the items in the code is preserved.

There are some others, mentioned in the proposal.

Author: bors

Cargo.lock

@@ -4036,6 +4036,7 @@ dependencies = [
  "rustc_session",
  "rustc_span",
  "rustc_target",
+ "tempfile",
  "tracing",
 ]
 

compiler/rustc_ast_lowering/src/item.rs

@@ -1310,7 +1310,9 @@ impl<'hir> LoweringContext<'_, 'hir> {
             // create a fake body so that the entire rest of the compiler doesn't have to deal with
             // this as a special case.
             return self.lower_fn_body(decl, contract, |this| {
-                if attrs.iter().any(|a| a.has_name(sym::rustc_intrinsic)) {
+                if attrs.iter().any(|a| a.has_name(sym::rustc_intrinsic))
+                    || this.tcx.is_sdylib_interface_build()
+                {
                     let span = this.lower_span(span);
                     let empty_block = hir::Block {
                         hir_id: this.next_id(),

compiler/rustc_ast_passes/src/ast_validation.rs

@@ -84,6 +84,8 @@ struct AstValidator<'a> {
 
     lint_node_id: NodeId,
 
+    is_sdylib_interface: bool,
+
     lint_buffer: &'a mut LintBuffer,
 }
 
@@ -952,7 +954,7 @@ impl<'a> Visitor<'a> for AstValidator<'a> {
                 self.check_defaultness(item.span, *defaultness);
 
                 let is_intrinsic = item.attrs.iter().any(|a| a.has_name(sym::rustc_intrinsic));
-                if body.is_none() && !is_intrinsic {
+                if body.is_none() && !is_intrinsic && !self.is_sdylib_interface {
                     self.dcx().emit_err(errors::FnWithoutBody {
                         span: item.span,
                         replace_span: self.ending_semi_or_hi(item.span),
@@ -1441,7 +1443,7 @@ impl<'a> Visitor<'a> for AstValidator<'a> {
                     });
                 }
                 AssocItemKind::Fn(box Fn { body, .. }) => {
-                    if body.is_none() {
+                    if body.is_none() && !self.is_sdylib_interface {
                         self.dcx().emit_err(errors::AssocFnWithoutBody {
                             span: item.span,
                             replace_span: self.ending_semi_or_hi(item.span),
@@ -1689,6 +1691,7 @@ pub fn check_crate(
     sess: &Session,
     features: &Features,
     krate: &Crate,
+    is_sdylib_interface: bool,
     lints: &mut LintBuffer,
 ) -> bool {
     let mut validator = AstValidator {
@@ -1701,6 +1704,7 @@ pub fn check_crate(
         disallow_tilde_const: Some(TildeConstReason::Item),
         extern_mod_safety: None,
         lint_node_id: CRATE_NODE_ID,
+        is_sdylib_interface,
         lint_buffer: lints,
     };
     visit::walk_crate(&mut validator, krate);

compiler/rustc_ast_pretty/src/pprust/mod.rs

@@ -7,7 +7,9 @@ use std::borrow::Cow;
 use rustc_ast as ast;
 use rustc_ast::token::{Token, TokenKind};
 use rustc_ast::tokenstream::{TokenStream, TokenTree};
-pub use state::{AnnNode, Comments, PpAnn, PrintState, State, print_crate};
+pub use state::{
+    AnnNode, Comments, PpAnn, PrintState, State, print_crate, print_crate_as_interface,
+};
 
 /// Print the token kind precisely, without converting `$crate` into its respective crate name.
 pub fn token_kind_to_string(tok: &TokenKind) -> Cow<'static, str> {

compiler/rustc_ast_pretty/src/pprust/state.rs

@@ -221,6 +221,7 @@ pub struct State<'a> {
     pub s: pp::Printer,
     comments: Option<Comments<'a>>,
     ann: &'a (dyn PpAnn + 'a),
+    is_sdylib_interface: bool,
 }
 
 const INDENT_UNIT: isize = 4;
@@ -236,10 +237,37 @@ pub fn print_crate<'a>(
     is_expanded: bool,
     edition: Edition,
     g: &AttrIdGenerator,
+) -> String {
+    let mut s = State {
+        s: pp::Printer::new(),
+        comments: Some(Comments::new(sm, filename, input)),
+        ann,
+        is_sdylib_interface: false,
+    };
+
+    print_crate_inner(&mut s, krate, is_expanded, edition, g);
+    s.s.eof()
+}
+
+pub fn print_crate_as_interface(
+    krate: &ast::Crate,
+    edition: Edition,
+    g: &AttrIdGenerator,
 ) -> String {
     let mut s =
-        State { s: pp::Printer::new(), comments: Some(Comments::new(sm, filename, input)), ann };
+        State { s: pp::Printer::new(), comments: None, ann: &NoAnn, is_sdylib_interface: true };
 
+    print_crate_inner(&mut s, krate, false, edition, g);
+    s.s.eof()
+}
+
+fn print_crate_inner<'a>(
+    s: &mut State<'a>,
+    krate: &ast::Crate,
+    is_expanded: bool,
+    edition: Edition,
+    g: &AttrIdGenerator,
+) {
     // We need to print shebang before anything else
     // otherwise the resulting code will not compile
     // and shebang will be useless.
@@ -282,8 +310,7 @@ pub fn print_crate<'a>(
         s.print_item(item);
     }
     s.print_remaining_comments();
-    s.ann.post(&mut s, AnnNode::Crate(krate));
-    s.s.eof()
+    s.ann.post(s, AnnNode::Crate(krate));
 }
 
 /// Should two consecutive tokens be printed with a space between them?
@@ -1111,7 +1138,7 @@ impl<'a> PrintState<'a> for State<'a> {
 
 impl<'a> State<'a> {
     pub fn new() -> State<'a> {
-        State { s: pp::Printer::new(), comments: None, ann: &NoAnn }
+        State { s: pp::Printer::new(), comments: None, ann: &NoAnn, is_sdylib_interface: false }
     }
 
     fn commasep_cmnt<T, F, G>(&mut self, b: Breaks, elts: &[T], mut op: F, mut get_span: G)

compiler/rustc_ast_pretty/src/pprust/state/item.rs

@@ -160,6 +160,10 @@ impl<'a> State<'a> {
 
     /// Pretty-prints an item.
     pub(crate) fn print_item(&mut self, item: &ast::Item) {
+        if self.is_sdylib_interface && item.span.is_dummy() {
+            // Do not print prelude for interface files.
+            return;
+        }
         self.hardbreak_if_not_bol();
         self.maybe_print_comment(item.span.lo());
         self.print_outer_attributes(&item.attrs);
@@ -682,6 +686,13 @@ impl<'a> State<'a> {
             self.print_contract(contract);
         }
         if let Some((body, (cb, ib))) = body_cb_ib {
+            if self.is_sdylib_interface {
+                self.word(";");
+                self.end(ib); // end inner head-block
+                self.end(cb); // end outer head-block
+                return;
+            }
+
             self.nbsp();
             self.print_block_with_attrs(body, attrs, cb, ib);
         } else {

compiler/rustc_codegen_gcc/src/back/lto.rs

@@ -44,7 +44,11 @@ use crate::{GccCodegenBackend, GccContext, SyncContext, to_gcc_opt_level};
 
 pub fn crate_type_allows_lto(crate_type: CrateType) -> bool {
     match crate_type {
-        CrateType::Executable | CrateType::Dylib | CrateType::Staticlib | CrateType::Cdylib => true,
+        CrateType::Executable
+        | CrateType::Dylib
+        | CrateType::Staticlib
+        | CrateType::Cdylib
+        | CrateType::Sdylib => true,
         CrateType::Rlib | CrateType::ProcMacro => false,
     }
 }

compiler/rustc_codegen_llvm/src/back/lto.rs

@@ -42,7 +42,8 @@ fn crate_type_allows_lto(crate_type: CrateType) -> bool {
         | CrateType::Dylib
         | CrateType::Staticlib
         | CrateType::Cdylib
-        | CrateType::ProcMacro => true,
+        | CrateType::ProcMacro
+        | CrateType::Sdylib => true,
         CrateType::Rlib => false,
     }
 }

compiler/rustc_codegen_llvm/src/debuginfo/gdb.rs

@@ -95,7 +95,11 @@ pub(crate) fn needs_gdb_debug_scripts_section(cx: &CodegenCx<'_, '_>) -> bool {
     // in the `.debug_gdb_scripts` section. For that reason, we make sure that the
     // section is only emitted for leaf crates.
     let embed_visualizers = cx.tcx.crate_types().iter().any(|&crate_type| match crate_type {
-        CrateType::Executable | CrateType::Dylib | CrateType::Cdylib | CrateType::Staticlib => {
+        CrateType::Executable
+        | CrateType::Dylib
+        | CrateType::Cdylib
+        | CrateType::Staticlib
+        | CrateType::Sdylib => {
             // These are crate types for which we will embed pretty printers since they
             // are treated as leaf crates.
             true

compiler/rustc_codegen_ssa/src/back/link.rs

@@ -1053,9 +1053,10 @@ fn link_natively(
                 strip_with_external_utility(sess, stripcmd, out_filename, &["--strip-debug"])
             }
             // Per the manpage, `-x` is the maximum safe strip level for dynamic libraries. (#93988)
-            (Strip::Symbols, CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro) => {
-                strip_with_external_utility(sess, stripcmd, out_filename, &["-x"])
-            }
+            (
+                Strip::Symbols,
+                CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro | CrateType::Sdylib,
+            ) => strip_with_external_utility(sess, stripcmd, out_filename, &["-x"]),
             (Strip::Symbols, _) => {
                 strip_with_external_utility(sess, stripcmd, out_filename, &["--strip-all"])
             }
@@ -1243,8 +1244,10 @@ fn add_sanitizer_libraries(
     // which should be linked to both executables and dynamic libraries.
     // Everywhere else the runtimes are currently distributed as static
     // libraries which should be linked to executables only.
-    if matches!(crate_type, CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro)
-        && !(sess.target.is_like_darwin || sess.target.is_like_msvc)
+    if matches!(
+        crate_type,
+        CrateType::Dylib | CrateType::Cdylib | CrateType::ProcMacro | CrateType::Sdylib
+    ) && !(sess.target.is_like_darwin || sess.target.is_like_msvc)
     {
         return;
     }
@@ -1938,6 +1941,7 @@ fn add_late_link_args(
     codegen_results: &CodegenResults,
 ) {
     let any_dynamic_crate = crate_type == CrateType::Dylib
+        || crate_type == CrateType::Sdylib
         || codegen_results.crate_info.dependency_formats.iter().any(|(ty, list)| {
             *ty == crate_type && list.iter().any(|&linkage| linkage == Linkage::Dynamic)
         });