build-std: second iteration

[davidtwco]

Supercedes #1. Intended for discussion with invited reviewers until we're happy enough to post publicly. Feel free to discuss on #project-goals/2025h1/build-std.

Please don't leave feedback on this unless you've been invited to. We'll invite more project members and teams to participate over time and eventually post this publicly for community feedback. With such a large proposal, this sort of staged rollout is the only way we can manage the feedback and evolve the document. If you aren't invited and post anyway, I reserve the right to just delete comments.

Rendered

[adamgemmell]

Bootstrap just defaults to panic-unwind for std targets all the time it seems. If std is made an optional dependency of sysroot so the sysroot crate can be built for all targets then perhaps panic_unwind should just be a default feature to avoid cargo needing to pass it.

A future better solution involving feature flags once they're stable would allow for not building the panic runtime that won't be used, and would avoid libs needing to specially stabilise the panic-unwind feature now.

[davidtwco]

..and would avoid libs needing to specially stabilise the panic-unwind feature now.

Our proposal doesn't require this. Cargo will enable the feature sometimes but that doesn't mean it has to be stable. It will need to be part of the contract between the standard library and Cargo, if the feature changes, both need updated, but that doesn't make it stable.

[adamgemmell]

This might be up to the Cargo team to decide - Cargo does not currently use any unstable interfaces for stable features as far as I'm aware.

On our part we can ensure test coverage of this area on both stable/nightly tests in Cargo, but then we hit the problem of needing to update both trees at the same time. We can of course also show a plan to eventually stabilise libs features.

[ehuss]

Just a minor point: I'm not sure how you intend to publish this as an RFC (that is, if you want to push all of this as a single RFC). If that's the plan, the directory structure will need to be a little different. The top-level document has to be text/0000-build-std.md, and then additional chapters have to be in a subdirectory. You can look at 3392 as an example.

[davidtwco]

Just a minor point: I'm not sure how you intend to publish this as an RFC (that is, if you want to push all of this as a single RFC). If that's the plan, the directory structure will need to be a little different. The top-level document has to be text/0000-build-std.md, and then additional chapters have to be in a subdirectory. You can look at 3392 as an example.

We can definitely change it to match that. I expect we'll end up changing the structure again prior to publishing anyway.

[ehuss]

These are all very minor points for stage 1a.

[ehuss]

Avoiding the depinfo may be difficult due to the way cargo works. Is there any particular reason say that it is avoided? AFAIK, the cost is negligible.

[davidtwco]

This was just based on what I understood the current implementation to be doing since rust-lang/cargo#8177:

• Cargo's dep-info fingerprint tracking will not track the std crate sources, these are tracked via other means.
• Cargo's .d dep-info file does not include std crate sources.

Maybe I've phrased it incorrectly here?

[ehuss]

Have you gone with this "default to std" to avoid needing to make sure the compiler knows if std is supported?

We previously had some discussions around rust-lang/cargo#14183 where we added something similar to this (but using target-spec-json). I think we discussed possibly adding a dedicated --print option instead of using spec-json, but I can't remember if there were some concerns around that?

(Not important, automatic detection could always be added back later, but I'm wondering what the thinking is here.)

[davidtwco]

I went with "std" as a default assuming that the most common case would be that the user would want to rebuild the entire standard library, and if their target didn't support std, with this being a manual opt-in mechanism, it's okay if that doesn't work and they need to change this option to compile fewer crates (just alloc/core or core).

It's not ideal for targets where std will never be supported and we might want to just prevent that from happening entirely with a nice message, where we could introduce a --print std-support-type of option again to give us the ability to prevent those builds. I'm certainly open doing that.

[adamgemmell]

I think I'll work on an edit in favour of effectively keeping the behaviour of current -Zbuild-std from rust-lang/cargo#14899 where the default is based on target-spec. We could also prevent building std on no_std targets unless the unstable option that relaxes build-std-crates is present.

Having the default option for half the targets to be something that never works seems sub-optimal. Even worse is that default happening to build and then breaking all the time. It's fairly feasible for a user to get lucky and not even realise their target is no_std and end up publishing a crate that uses a std API.

[ehuss]

Just FYI, I'm not sure how easy this will be. It may require some ugly special casing in cargo.

There is rust-lang/wg-cargo-std-aware#88 tracking that cargo doc is mostly broken right now.

rustdoc has special handling where it will auto-link to https://doc.rust-lang.org/, but I'm not sure why it is broken with build-std. An alternative here would be to investigate if that is an easy fix, and just link to the online docs for now.

I'm also not sure if you are aware of -Z rustdoc-map which provides a way to control how std documentation links work.

[davidtwco]

I would have assumed that linking to local standard library documentation would have been the default and "just worked" (that it would just be "do the same thing as any other crate for these crates too), and that linking to the upstream documentation would be harder (but more desirable if possible), which is why I propose the former at the moment.

I'm aware of rust-lang/wg-cargo-std-aware#88 which is why in the rationale for this part I mention we'd need to be able to pass -Zcrate-attr="doc(html_root_url=...)" with the correct url and it isn't clear what that would be.

Either way, linking to local docs initially seems sufficient for this initial proposal and we could improve that later.

See also my reply to your other comment - #2 (comment)

[ehuss]

-Zcrate-attr may not be required. See the comment above about how rustdoc automatically determines if it can link to the online docs.

[davidtwco]

I think we should leave the RFC as-written, that we'll link to local docs, and explore this in implementation - it might be trivial to link to the hosted docs if -Zrustdoc-map can be used and there's just a small issue somewhere, but its hard to know without trying, and what we propose is sufficient if it ends up not being possible.

[Kobzol]

I went through the introduction and "stage 1a" and left some comments. I think that the individual stages should explicitly list which subset of the main motivation of the RFC is enabled by the given stage. For example, I just finished 1a, and realized that it doesn't solve the use-cases of rebuilding with a custom profile, so it "only" enables the motivation 4) and 5) (building std for tier-3 targets). That's quite important piece of information, IMO, and it should be made more explicit.

Btw, I have to say, this is a momumental amount of work, great job everyone involved! Not just the research itself, but also the structuring of the document, the question mark links are really great to avoid adding tons of explanations and rationale into the "main" text.

[Kobzol]

What about PGO? The sysroot crate has an optional (and non-default) feature called profiler (https://github.com/rust-lang/rust/blob/21120e297c3d2db6a1f28e1a8798777a52f4fee2/library/sysroot/Cargo.toml#L35), which has to be enabled for PGO to be supported. If users pass -Cprofile-generate in RUSTFLAGS (which is opaque to Cargo), this feature would need to be enabled. That would probably require Cargo to scan rustflags (?).

Or maybe we could just build it unconditionally (or vice-versa: forbid PGO with build-std).

Similar question about the backtrace feature, which is currently not enabled by default, but there I suppose it could just be enabled unconditionally.

Edit: I saw In line with Cargo's stance on not parsing the RUSTFLAGS environment variable, it will not be checked for compilation flags that would require additional crates to be built for compilation to succeed. mentioned below. Which suggests that we either build the profiler runtime always or never.

[epage]

(which is opaque to Cargo), this feature would need to be enabled. That would probably require Cargo to scan rustflags (?).

We generally try to steer solutions away from cargo parsing rustflags as it effectively requires us to duplicate rustc's entire CLI parser as-is to ensure correctness as CLIs are context sensitive, dependent on the flags specified, and can support multiple syntaxes for the same thing.

[davidtwco]

I think I'd like us to be able to compile profiler-builtins unconditionally if we can, and if Cargo eventually has a mechanism for enabling/disabling profiling through Cargo configuration, then we can start compiling that crate only when necessary.

In practice, that's tricky because building profiler-builtins requires a C toolchain. If we are able to ship the C-compiled part as a rustup component, then that might be an option. Otherwise, I'd be happy not building it initially, leaving the ability to do that as a follow-up, and if you set -Cprofile-generate in RUSTFLAGS before that follow-up then you're out of luck.

It's similar to sanitizers w/ needing some native component, except profiling is stable. I don't know if that changes the calculus here and we'd need this to work via RUSTFLAGS w/ build-std. Obviously we can't scan RUSTFLAGS to prevent this combination.

We'll look into whether it'll be possible to ship in a self-contained component the native parts that profiler-builtins needs so that always building it is possible. Definitely some details to work out here.

[Kobzol]

Some brief stats on x64 Linux:

• Current size of rust-src when built locally with xz and best compression profile: 3.5 MiB archived, 44 MiB extracted.
• With stdlib vendored dependencies added: 9.1 MiB archived, 131 MiB extracted

[Mark-Simulacrum]

It looks like the majority of the extra space from vendoring is from windows-sys (and friends):

   16.9 MiB [ 20.0%] /windows-sys
   12.6 MiB [ 14.9%] /windows_i686_gnu
   12.1 MiB [ 14.4%] /windows_x86_64_gnu
    5.3 MiB [  6.3%] /windows_i686_msvc
    5.0 MiB [  5.9%] /windows_aarch64_msvc
    5.0 MiB [  5.9%] /windows_x86_64_msvc
    4.6 MiB [  5.4%] /libc
    3.9 MiB [  4.7%] /windows_i686_gnullvm
    3.7 MiB [  4.4%] /windows_aarch64_gnullvm
    3.7 MiB [  4.4%] /windows_x86_64_gnullvm
    2.3 MiB [  2.7%] /unicode-width
    2.1 MiB [  2.4%] /object
    1.8 MiB [  2.2%] /gimli
  920.0 KiB [  1.1%] /hashbrown

It does feel unfortunate for users to pay the extra ~90MB of toolchain space for (probably?) unused code in most local build-std use cases... IIUC, the argument that people download that anyways only applies when targeting the windows targets (and even then, you'd probably not be targeting i686, x86_64, and aarch64 windows...).

I think it's worth asking whether we should split rust-src to be optionally distributed for each target, similar to how this RFC was proposing we'd do for the CRT components. Or whether that change could be added later backwards-compatibly? (Even without the vendoring, having the stdarch intrinsics for x86_64 when you don't build for that target feels unfortunate. Disk space is pretty cheap for most people, but still).

[davidtwco]

Maybe we could do something like rust-lang/compiler-team#738 to ship a MIR-only rlib that that just "finish" the compilation of, and depend on that instead of rust-src. If I recall with that though, there were challenges with cfgs that probably still make that impossible, even if we did a per-target partial rlib.

Either way, I think this is more of an optimisation for later rather than an immediate concern. It's a relatively small disk space cost that enables build-std for everyone, and I think that's reasonable, esp. when it can be improved later. Splitting up std per-target might have greater net disk space usage for those compiling to multiple targets regularly. Trade-offs!

[Kobzol]

I wonder why is this needed in general. build-std implements and stabilizes a mechanism for building the standard library (using Cargo). But if the library can be built on a given system or not is always partly a responsibility of the system's user. There are many Rust crates that cannot be built without a C compiler, and I think that's fine; is it necessarily bad that if people want to use build-std, they will need a C compiler?

[Kobzol]

I think that this will become a necessity if build-std moves forward.

[Kobzol]

I think that this will break LTO, because the stdlib will be compiled with -Cembed-bitcode=no (bootstrap explicitly overrides this with -Cembed-bitcode=yes to ship libstd that is LTO-compatible). But even without merging the stdlib and user's workspace profiles, this should be relatively easy to go around; cargo would just hardcode -Cembed-bitcode=yes if the user's profile has LTO enabled (or it would just use it always, but that would waste disk space and possibly some compilation time).

[Mark-Simulacrum]

I'm also a bit confused by this, are we ruling out any customization of that profile other than (maybe) via RUSTFLAGS? E.g., opt-level=s or enabling/disabling debuginfo?

[davidtwco]

I think that this will break LTO, because the stdlib will be compiled with -Cembed-bitcode=no (bootstrap explicitly overrides this with -Cembed-bitcode=yes to ship libstd that is LTO-compatible).

We could move -Cembed-bitcode=yes into the release profile of the standard library (if we permit nightly-only profile.rustflags to be respected for std even if doing an otherwise stable build).

I'm also a bit confused by this, are we ruling out any customization of that profile other than (maybe) via RUSTFLAGS? E.g., opt-level=s or enabling/disabling debuginfo?

In Stage 1a, all we're enabling is that you can build a vanilla release profile std for a target. In the later stages, we want to enable automatically building a std that has compatible target modifiers (configured through Cargo profiles, somehow), and we want to enable building std with the user's profile.

[Mark-Simulacrum]

Is libunwind still required? I'm a little confused by "in the past" here (do we care about the past for some reason?).

Is it accurate to say that the rust-src component is a fully vendored set of sources (across both C and Rust) needed to build the standard library, modulo the configuration to do so (which lives in bootstrap and elsewhere)? (E.g., crates.io dependencies are vendored in when creating it)?

Editing after reading further... I see that stage 1a actually proposes extending rust-src to include vendored sources for all Rust dependencies. I think that's confusing to have this section in the background if we're proposing changes for it. I wonder if we could "just" make the vendoring happen as a separate proposal so we can decouple those changes from this wider RFC?

[adamgemmell]

unwind now links to primarily the system libunwind, or the sysroot binary. The unwind sources in rust-src haven't been used since rust-lang/rust#85600. rust-src at the moment is just a copy of certain paths in the rust repo. You're right in that mentioning libunwind sources here is a bit of a nothingburger, I might just fix it and remove it from the background.

A few people have tried so far to have a vendored rust-src (most recently bjorn3) but the Cargo changes currently required are quite tricky. It would make sense to do other large changes proposed around the resolver in this RFC first before handling vendored dependencies. Vendoring could come in a later MCP, what do you think the benefits would be?

[Mark-Simulacrum]

I'm not super attached to any particular order. It just seems like if we think this is worth doing, it's something that IMO doesn't actually require an RFC or even MCP to do, just bootstrap/cargo PR(s). And landing it cuts the need for the RFC to be proposing anything in this direction, just relying on existing things.

(Effectively I see this as a technique for shortening the RFC by doing non-controversial stuff without it).

[davidtwco]

We could definitely explore vendoring deps into rust-src while we're discussing this draft so we can cut some complexity out. If it isn't straightforward then it's probably not worth spending too much time on separately IMO.

[Mark-Simulacrum]

From experience with bootstrap, we have had complaints that this kind of "eager" build behavior can cause confusion and annoyance -- the standard library shouldn't take that long to build (1-2 minutes) but it might still be confusing to people why tweaking a line in their toml added extra dependencies (e.g., rust-src component) and incurred this cost. Maybe worth clarifying whether we can find some way to teach users about the cause of the build.

[davidtwco]

At the very least it makes sense to mention in the documentation for target modifiers exposed through Cargo that they'll necessarily trigger rebuilds of the standard library. I think eager behaviour is the right choice here as long as we can make sure the rebuilds are only when they are actually necessary.

[Mark-Simulacrum]

Not really this particular section, but: do we have some consideration of how we'd vary the defaults for target modifiers across versions, if they're exposed in Cargo's profiles? It seems more plausible for customized RUSTFLAGS to break than Cargo options).

I'm thinking us e.g. enabling some security mitigation that's an ABI change and so is an (implicit) target modifier...

[davidtwco]

I expect if we changed the default for a target modifier in rustc, we'd want to also change the default in std and any Cargo option that tweaks that flag, so as to avoid the default configuration triggering rebuilds of the standard library (which I'm sure we'd notice quite quickly). I don't know how Cargo deals with defaults today (if it defines them or relies on rustc's default) and if we've had a case before where rustc has changed its default but Cargo hasn't.

[Mark-Simulacrum]

How does this work for custom profile names? Do we used the inherited-from profile?

Does this stabilize the meaning of e.g. the debug profile in library/Cargo.toml? Currently it's arguable that none of the profiles in library/ are in any way exposed to users, only the actual configuration used.

(I think this applies to earlier stages... I am still fuzzy on exactly what options/profiles are passed to rustc by Cargo or by external non-Cargo build systems for building their own standard library crates).

[davidtwco]

I hadn't considered custom profile names. I think we'd use the inherited from profile, yeah. I'll update the RFC to mention this.

I'd want our stability guarantees to say that we don't guarantee that any given profile will always build with a given option set, and that if you want to guarantee that, you need to set it yourself. I think we say something like this already.

[Mark-Simulacrum]

How will we do that? If, for example, we look at today's build it's something like "various options, both stable and unstable, set across a dozen or so files, including target-specific and runner-specific files"... or are we envisioning new CI jobs that are dedicated to testing the dist output with particular configurations (probably makes sense)?

Do we expect to guarantee that users can, on stable, reproduce the build that we distributed binaries for? Or can that require extra flags that aren't stable?

[davidtwco]

This paragraph assumes that we can get the release profile to match the pre-built standard library, in which case we would document that the default release configuration, built locally or pre-built, is tested. We don't necessarily need to go into detail on all of the options that this configuration is comprised of. We could similarly chose to say "the release profile with the addition of one of these sanitisers enabled is tested", or something like that.

I'm envisioning something high-level but that gives users a greater level of confidence for some of the common use cases for build-std, like enabling sanitisers.

[Mark-Simulacrum]

Will we unset or ignore any other flags that affect the build? e.g. -Zallow-features or future extensions to RUSTC_BOOTSTRAP (I think we already support per-crate =1 but at the all hands there was discussion of allowing allow-features via env variables, too).

[davidtwco]

I wasn't intending to - expecting that we're never going to only build the standard library, it'll always be paired with the user's project. So if you have RUSTFLAGS=-Zsome-unstable-flag then the implied RUSTC_BOOTSTRAP=1 of the sysroot dependencies might mean that those builds succeed using the nightly flag where they might not otherwise, but it'll still fail when it gets to compiling the user's code. That seems okay to me.

[Kobzol]

I did some experiments with PGO:

1. You need the profiler_builtins std crate if you want to use PGO
2. If you don't compile it with -Zbuild-std, the prebuilt version from the sysroot will be used instead. This seems like a more general "issue" - if Cargo only recompiles a subset of libstd, but then the rest of the crates can still be looked up from the sysroot, we would combine crates from the prebuilt std and the local built std (which seems to happen today). But the --extern magic described in this RFC would hopefully solve that.
3. If you want to compile profiler_builtins with build-std, then you'll need LLVM sources to be available.

[thejpster]

I've left some pretty minor comments. In general, this was well structured and highly legible given the very technical nature. Excellent work!

[adamgemmell]

This precedence means that target triple/cfg options in low-precedence config locations can override users explicitly asking for build.build-std=always in high precedence locations, like through --config, which is unintuitive at best. It's really hard to override those settings in the global config.

[adamgemmell]

Per the meeting, we should make a more explicit plan here so we ensure we're not closing anything off

[ehuss]

Just some minor comments on stage-1b.

[ehuss]

I haven't seen a discussion about prelude behavior here. I can imagine various different behaviors here:

• Builtin dependencies always use noprelude. This relies on rustc to add implicit extern crate for std/core (conditional on #![no_std] attribute), which would also implicitly add it to the prelude.
  • This causes an inconsistency with other dependency declarations. This means that you still have to specify extern crate alloc; in all cases, something we've wanted to avoid.
• Explicit builtin dependencies do not use noprelude. This is a partial backwards incompatibility (see the whole reason noprelude exists), but might be acceptable if this is limited to explicit dependencies, since the user has to manually opt-in to using them (and thus can fix any problems they encounter, possibly making std optional).
  • That means that implicit builtin dependencies (that is, when no builtins are specified) continue to use noprelude, which means that hello_world is not equivalent to what is listed above.

I'm not sure what the answer is here, but would be good to think about the compatibility and consistency issues, and come up with a plan.

[ehuss]

We should also probably specify the behavior with [replace] and paths.

I would be fine with just permanently disallowing the use of either with the standard library, though I don't know how easy that is.

[ehuss]

Does this optionality check also check other dependency kinds like target-specific dependencies? For example:

[target.x86_64-pc-windows-gnu.dependencies]
alloc.builtin = true

Does this mean that it uses an explicit alloc on x86_64-pc-windows-gnu, and an implicit std/alloc/core on all other targets? Or is this an error, requiring a builtin to be specified in [dependencies]?