Auto merge of #84278 - Aaron1011:feature/new-proc-macro-meta-span, r=estebank

Implement span quoting for proc-macros

This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:

```
error[E0412]: cannot find type `MissingType` in this scope
  --> $DIR/auxiliary/span-from-proc-macro.rs:37:20
   |
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
   | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL |             field: MissingType
   |                    ^^^^^^^^^^^ not found in this scope
   |
  ::: $DIR/span-from-proc-macro.rs:8:1
   |
LL | #[error_from_attribute]
   | ----------------------- in this macro invocation
```

Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`

This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.

This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
  macro to get run. This saves all of the sapns in the input to `quote!`
  into the metadata of *the proc-macro-crate* (which we are currently
  compiling). The `quote!` macro then expands to a call to
  `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
  and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.

The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.

This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`

Custom quoting currently has a few limitations:

In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.

Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.

Author: bors

compiler/rustc_builtin_macros/src/lib.rs

@@ -20,6 +20,7 @@ use crate::deriving::*;
 
 use rustc_expand::base::{MacroExpanderFn, ResolverExpand, SyntaxExtensionKind};
 use rustc_expand::proc_macro::BangProcMacro;
+use rustc_span::def_id::LOCAL_CRATE;
 use rustc_span::symbol::sym;
 
 mod asm;
@@ -114,5 +115,8 @@ pub fn register_builtin_macros(resolver: &mut dyn ResolverExpand) {
     }
 
     let client = proc_macro::bridge::client::Client::expand1(proc_macro::quote);
-    register(sym::quote, SyntaxExtensionKind::Bang(Box::new(BangProcMacro { client })));
+    register(
+        sym::quote,
+        SyntaxExtensionKind::Bang(Box::new(BangProcMacro { client, krate: LOCAL_CRATE })),
+    );
 }

compiler/rustc_errors/src/emitter.rs

@@ -309,7 +309,9 @@ pub trait Emitter {
                     // are some which do actually involve macros.
                     ExpnKind::Inlined | ExpnKind::Desugaring(..) | ExpnKind::AstPass(..) => None,
 
-                    ExpnKind::Macro(macro_kind, _) => Some(macro_kind),
+                    ExpnKind::Macro { kind: macro_kind, name: _, proc_macro: _ } => {
+                        Some(macro_kind)
+                    }
                 }
             });
 
@@ -371,10 +373,19 @@ pub trait Emitter {
                     new_labels
                         .push((trace.call_site, "in the inlined copy of this code".to_string()));
                 } else if always_backtrace {
+                    let proc_macro = if let ExpnKind::Macro { kind: _, name: _, proc_macro: true } =
+                        trace.kind
+                    {
+                        "procedural macro "
+                    } else {
+                        ""
+                    };
+
                     new_labels.push((
                         trace.def_site,
                         format!(
-                            "in this expansion of `{}`{}",
+                            "in this expansion of {}`{}`{}",
+                            proc_macro,
                             trace.kind.descr(),
                             if macro_backtrace.len() > 1 {
                                 // if macro_backtrace.len() == 1 it'll be
@@ -400,7 +411,11 @@ pub trait Emitter {
                 // and it needs an "in this macro invocation" label to match that.
                 let redundant_span = trace.call_site.contains(sp);
 
-                if !redundant_span && matches!(trace.kind, ExpnKind::Macro(MacroKind::Bang, _))
+                if !redundant_span
+                    && matches!(
+                        trace.kind,
+                        ExpnKind::Macro { kind: MacroKind::Bang, name: _, proc_macro: _ }
+                    )
                     || always_backtrace
                 {
                     new_labels.push((

compiler/rustc_expand/src/base.rs

@@ -14,7 +14,7 @@ use rustc_lint_defs::builtin::PROC_MACRO_BACK_COMPAT;
 use rustc_lint_defs::BuiltinLintDiagnostics;
 use rustc_parse::{self, nt_to_tokenstream, parser, MACRO_ARGUMENTS};
 use rustc_session::{parse::ParseSess, Limit, Session};
-use rustc_span::def_id::DefId;
+use rustc_span::def_id::{CrateNum, DefId};
 use rustc_span::edition::Edition;
 use rustc_span::hygiene::{AstPass, ExpnData, ExpnId, ExpnKind};
 use rustc_span::source_map::SourceMap;
@@ -810,8 +810,16 @@ impl SyntaxExtension {
         descr: Symbol,
         macro_def_id: Option<DefId>,
     ) -> ExpnData {
+        use SyntaxExtensionKind::*;
+        let proc_macro = match self.kind {
+            // User-defined proc macro
+            Bang(..) | Attr(..) | Derive(..) => true,
+            // Consider everthing else to be not a proc
+            // macro for diagnostic purposes
+            LegacyBang(..) | LegacyAttr(..) | NonMacroAttr { .. } | LegacyDerive(..) => false,
+        };
         ExpnData::new(
-            ExpnKind::Macro(self.macro_kind(), descr),
+            ExpnKind::Macro { kind: self.macro_kind(), name: descr, proc_macro },
             parent,
             call_site,
             self.span,
@@ -873,6 +881,10 @@ pub trait ResolverExpand {
     fn take_derive_resolutions(&mut self, expn_id: ExpnId) -> Option<DeriveResolutions>;
     /// Path resolution logic for `#[cfg_accessible(path)]`.
     fn cfg_accessible(&mut self, expn_id: ExpnId, path: &ast::Path) -> Result<bool, Indeterminate>;
+
+    /// Decodes the proc-macro quoted span in the specified crate, with the specified id.
+    /// No caching is performed.
+    fn get_proc_macro_quoted_span(&self, krate: CrateNum, id: usize) -> Span;
 }
 
 #[derive(Clone, Default)]

compiler/rustc_expand/src/proc_macro.rs

@@ -9,12 +9,14 @@ use rustc_data_structures::sync::Lrc;
 use rustc_errors::ErrorReported;
 use rustc_parse::nt_to_tokenstream;
 use rustc_parse::parser::ForceCollect;
+use rustc_span::def_id::CrateNum;
 use rustc_span::{Span, DUMMY_SP};
 
 const EXEC_STRATEGY: pm::bridge::server::SameThread = pm::bridge::server::SameThread;
 
 pub struct BangProcMacro {
     pub client: pm::bridge::client::Client<fn(pm::TokenStream) -> pm::TokenStream>,
+    pub krate: CrateNum,
 }
 
 impl base::ProcMacro for BangProcMacro {
@@ -24,7 +26,7 @@ impl base::ProcMacro for BangProcMacro {
         span: Span,
         input: TokenStream,
     ) -> Result<TokenStream, ErrorReported> {
-        let server = proc_macro_server::Rustc::new(ecx);
+        let server = proc_macro_server::Rustc::new(ecx, self.krate);
         self.client.run(&EXEC_STRATEGY, server, input, ecx.ecfg.proc_macro_backtrace).map_err(|e| {
             let mut err = ecx.struct_span_err(span, "proc macro panicked");
             if let Some(s) = e.as_str() {
@@ -38,6 +40,7 @@ impl base::ProcMacro for BangProcMacro {
 
 pub struct AttrProcMacro {
     pub client: pm::bridge::client::Client<fn(pm::TokenStream, pm::TokenStream) -> pm::TokenStream>,
+    pub krate: CrateNum,
 }
 
 impl base::AttrProcMacro for AttrProcMacro {
@@ -48,7 +51,7 @@ impl base::AttrProcMacro for AttrProcMacro {
         annotation: TokenStream,
         annotated: TokenStream,
     ) -> Result<TokenStream, ErrorReported> {
-        let server = proc_macro_server::Rustc::new(ecx);
+        let server = proc_macro_server::Rustc::new(ecx, self.krate);
         self.client
             .run(&EXEC_STRATEGY, server, annotation, annotated, ecx.ecfg.proc_macro_backtrace)
             .map_err(|e| {
@@ -64,6 +67,7 @@ impl base::AttrProcMacro for AttrProcMacro {
 
 pub struct ProcMacroDerive {
     pub client: pm::bridge::client::Client<fn(pm::TokenStream) -> pm::TokenStream>,
+    pub krate: CrateNum,
 }
 
 impl MultiItemModifier for ProcMacroDerive {
@@ -97,7 +101,7 @@ impl MultiItemModifier for ProcMacroDerive {
             nt_to_tokenstream(&item, &ecx.sess.parse_sess, CanSynthesizeMissingTokens::No)
         };
 
-        let server = proc_macro_server::Rustc::new(ecx);
+        let server = proc_macro_server::Rustc::new(ecx, self.krate);
         let stream =
             match self.client.run(&EXEC_STRATEGY, server, input, ecx.ecfg.proc_macro_backtrace) {
                 Ok(stream) => stream,

compiler/rustc_expand/src/proc_macro_server.rs

@@ -1,4 +1,4 @@
-use crate::base::ExtCtxt;
+use crate::base::{ExtCtxt, ResolverExpand};
 
 use rustc_ast as ast;
 use rustc_ast::token;
@@ -7,13 +7,16 @@ use rustc_ast::token::NtIdent;
 use rustc_ast::tokenstream::{self, CanSynthesizeMissingTokens};
 use rustc_ast::tokenstream::{DelimSpan, Spacing::*, TokenStream, TreeAndSpacing};
 use rustc_ast_pretty::pprust;
+use rustc_data_structures::fx::FxHashMap;
 use rustc_data_structures::sync::Lrc;
 use rustc_errors::Diagnostic;
 use rustc_lint_defs::builtin::PROC_MACRO_BACK_COMPAT;
 use rustc_lint_defs::BuiltinLintDiagnostics;
 use rustc_parse::lexer::nfc_normalize;
 use rustc_parse::{nt_to_tokenstream, parse_stream_from_source_str};
 use rustc_session::parse::ParseSess;
+use rustc_span::def_id::CrateNum;
+use rustc_span::hygiene::ExpnId;
 use rustc_span::hygiene::ExpnKind;
 use rustc_span::symbol::{self, kw, sym, Symbol};
 use rustc_span::{BytePos, FileName, MultiSpan, Pos, RealFileName, SourceFile, Span};
@@ -355,22 +358,34 @@ pub struct Literal {
 }
 
 pub(crate) struct Rustc<'a> {
+    resolver: &'a dyn ResolverExpand,
     sess: &'a ParseSess,
     def_site: Span,
     call_site: Span,
     mixed_site: Span,
     span_debug: bool,
+    krate: CrateNum,
+    expn_id: ExpnId,
+    rebased_spans: FxHashMap<usize, Span>,
 }
 
 impl<'a> Rustc<'a> {
-    pub fn new(cx: &'a ExtCtxt<'_>) -> Self {
+    pub fn new(cx: &'a ExtCtxt<'_>, krate: CrateNum) -> Self {
         let expn_data = cx.current_expansion.id.expn_data();
+        let def_site = cx.with_def_site_ctxt(expn_data.def_site);
+        let call_site = cx.with_call_site_ctxt(expn_data.call_site);
+        let mixed_site = cx.with_mixed_site_ctxt(expn_data.call_site);
+        let sess = cx.parse_sess();
         Rustc {
-            sess: &cx.sess.parse_sess,
-            def_site: cx.with_def_site_ctxt(expn_data.def_site),
-            call_site: cx.with_call_site_ctxt(expn_data.call_site),
-            mixed_site: cx.with_mixed_site_ctxt(expn_data.call_site),
+            resolver: cx.resolver,
+            sess,
+            def_site,
+            call_site,
+            mixed_site,
             span_debug: cx.ecfg.span_debug,
+            krate,
+            expn_id: cx.current_expansion.id,
+            rebased_spans: FxHashMap::default(),
         }
     }
 
@@ -713,6 +728,51 @@ impl server::Span for Rustc<'_> {
     fn source_text(&mut self, span: Self::Span) -> Option<String> {
         self.sess.source_map().span_to_snippet(span).ok()
     }
+    /// Saves the provided span into the metadata of
+    /// *the crate we are currently compiling*, which must
+    /// be a proc-macro crate. This id can be passed to
+    /// `recover_proc_macro_span` when our current crate
+    /// is *run* as a proc-macro.
+    ///
+    /// Let's suppose that we have two crates - `my_client`
+    /// and `my_proc_macro`. The `my_proc_macro` crate
+    /// contains a procedural macro `my_macro`, which
+    /// is implemented as: `quote! { "hello" }`
+    ///
+    /// When we *compile* `my_proc_macro`, we will execute
+    /// the `quote` proc-macro. This will save the span of
+    /// "hello" into the metadata of `my_proc_macro`. As a result,
+    /// the body of `my_proc_macro` (after expansion) will end
+    /// up containg a call that looks like this:
+    /// `proc_macro::Ident::new("hello", proc_macro::Span::recover_proc_macro_span(0))`
+    ///
+    /// where `0` is the id returned by this function.
+    /// When `my_proc_macro` *executes* (during the compilation of `my_client`),
+    /// the call to `recover_proc_macro_span` will load the corresponding
+    /// span from the metadata of `my_proc_macro` (which we have access to,
+    /// since we've loaded `my_proc_macro` from disk in order to execute it).
+    /// In this way, we have obtained a span pointing into `my_proc_macro`
+    fn save_span(&mut self, mut span: Self::Span) -> usize {
+        // Throw away the `SyntaxContext`, since we currently
+        // skip serializing `SyntaxContext`s for proc-macro crates
+        span = span.with_ctxt(rustc_span::SyntaxContext::root());
+        self.sess.save_proc_macro_span(span)
+    }
+    fn recover_proc_macro_span(&mut self, id: usize) -> Self::Span {
+        let resolver = self.resolver;
+        let krate = self.krate;
+        let expn_id = self.expn_id;
+        *self.rebased_spans.entry(id).or_insert_with(|| {
+            let raw_span = resolver.get_proc_macro_quoted_span(krate, id);
+            // Ignore the deserialized `SyntaxContext` entirely.
+            // FIXME: Preserve the macro backtrace from the serialized span
+            // For example, if a proc-macro crate has code like
+            // `macro_one!() -> macro_two!() -> quote!()`, we might
+            // want to 'concatenate' this backtrace with the backtrace from
+            // our current call site.
+            raw_span.with_def_site_ctxt(expn_id)
+        })
+    }
 }
 
 // See issue #74616 for details
@@ -722,7 +782,7 @@ fn ident_name_compatibility_hack(
     rustc: &mut Rustc<'_>,
 ) -> Option<(rustc_span::symbol::Ident, bool)> {
     if let NtIdent(ident, is_raw) = nt {
-        if let ExpnKind::Macro(_, macro_name) = orig_span.ctxt().outer_expn_data().kind {
+        if let ExpnKind::Macro { name: macro_name, .. } = orig_span.ctxt().outer_expn_data().kind {
             let source_map = rustc.sess.source_map();
             let filename = source_map.span_to_filename(orig_span);
             if let FileName::Real(RealFileName::Named(path)) = filename {

compiler/rustc_lint/src/internal.rs

@@ -248,10 +248,21 @@ impl EarlyLintPass for LintPassImpl {
                 if last.ident.name == sym::LintPass {
                     let expn_data = lint_pass.path.span.ctxt().outer_expn_data();
                     let call_site = expn_data.call_site;
-                    if expn_data.kind != ExpnKind::Macro(MacroKind::Bang, sym::impl_lint_pass)
-                        && call_site.ctxt().outer_expn_data().kind
-                            != ExpnKind::Macro(MacroKind::Bang, sym::declare_lint_pass)
-                    {
+                    if !matches!(
+                        expn_data.kind,
+                        ExpnKind::Macro {
+                            kind: MacroKind::Bang,
+                            name: sym::impl_lint_pass,
+                            proc_macro: _
+                        }
+                    ) && !matches!(
+                        call_site.ctxt().outer_expn_data().kind,
+                        ExpnKind::Macro {
+                            kind: MacroKind::Bang,
+                            name: sym::declare_lint_pass,
+                            proc_macro: _
+                        }
+                    ) {
                         cx.struct_span_lint(
                             LINT_PASS_IMPL_WITHOUT_MACRO,
                             lint_pass.path.span,

compiler/rustc_lint/src/non_fmt_panic.rs

@@ -248,10 +248,11 @@ fn panic_call<'tcx>(cx: &LateContext<'tcx>, f: &'tcx hir::Expr<'tcx>) -> (Span,
         }
     }
 
-    let macro_symbol = if let hygiene::ExpnKind::Macro(_, symbol) = expn.kind {
-        symbol
-    } else {
-        Symbol::intern("panic")
-    };
+    let macro_symbol =
+        if let hygiene::ExpnKind::Macro { kind: _, name: symbol, proc_macro: _ } = expn.kind {
+            symbol
+        } else {
+            Symbol::intern("panic")
+        };
     (expn.call_site, panic_macro, macro_symbol.as_str())
 }

compiler/rustc_metadata/src/rmeta/decoder.rs

@@ -716,30 +716,37 @@ impl<'a, 'tcx> CrateMetadataRef<'a> {
             .decode((self, sess))
     }
 
-    fn load_proc_macro(&self, id: DefIndex, sess: &Session) -> SyntaxExtension {
-        let (name, kind, helper_attrs) = match *self.raw_proc_macro(id) {
+    fn load_proc_macro(&self, def_id: DefId, sess: &Session) -> SyntaxExtension {
+        let (name, kind, helper_attrs) = match *self.raw_proc_macro(def_id.index) {
             ProcMacro::CustomDerive { trait_name, attributes, client } => {
                 let helper_attrs =
                     attributes.iter().cloned().map(Symbol::intern).collect::<Vec<_>>();
                 (
                     trait_name,
-                    SyntaxExtensionKind::Derive(Box::new(ProcMacroDerive { client })),
+                    SyntaxExtensionKind::Derive(Box::new(ProcMacroDerive {
+                        client,
+                        krate: def_id.krate,
+                    })),
                     helper_attrs,
                 )
             }
-            ProcMacro::Attr { name, client } => {
-                (name, SyntaxExtensionKind::Attr(Box::new(AttrProcMacro { client })), Vec::new())
-            }
-            ProcMacro::Bang { name, client } => {
-                (name, SyntaxExtensionKind::Bang(Box::new(BangProcMacro { client })), Vec::new())
-            }
+            ProcMacro::Attr { name, client } => (
+                name,
+                SyntaxExtensionKind::Attr(Box::new(AttrProcMacro { client, krate: def_id.krate })),
+                Vec::new(),
+            ),
+            ProcMacro::Bang { name, client } => (
+                name,
+                SyntaxExtensionKind::Bang(Box::new(BangProcMacro { client, krate: def_id.krate })),
+                Vec::new(),
+            ),
         };
 
-        let attrs: Vec<_> = self.get_item_attrs(id, sess).collect();
+        let attrs: Vec<_> = self.get_item_attrs(def_id.index, sess).collect();
         SyntaxExtension::new(
             sess,
             kind,
-            self.get_span(id, sess),
+            self.get_span(def_id.index, sess),
             helper_attrs,
             self.root.edition,
             Symbol::intern(name),
@@ -1379,6 +1386,15 @@ impl<'a, 'tcx> CrateMetadataRef<'a> {
         }
     }
 
+    fn get_proc_macro_quoted_span(&self, index: usize, sess: &Session) -> Span {
+        self.root
+            .tables
+            .proc_macro_quoted_spans
+            .get(self, index)
+            .unwrap_or_else(|| panic!("Missing proc macro quoted span: {:?}", index))
+            .decode((self, sess))
+    }
+
     fn get_foreign_modules(&self, tcx: TyCtxt<'tcx>) -> Lrc<FxHashMap<DefId, ForeignModule>> {
         if self.root.is_proc_macro_crate() {
             // Proc macro crates do not have any *target* foreign modules.