Skip to content

Enable outline-atomics by default on more AArch64 platforms #144938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Enable outline-atomics by default on more AArch64 platforms #144938

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
57f88d4
compiler-builtins: Change gating for outline atomic symbols
tgross35 Aug 5, 2025
9cf6ece
Make the `RUST_LSE_INIT` constructor cross-platform
tgross35 Aug 5, 2025
dafcd25
Enable `outline-atomics` by default on AArch64 Apple platforms
tgross35 Sep 5, 2025
f6d958f
Enable `outline-atomics` by default on AArch64 Windows platforms
tgross35 Sep 5, 2025
d49a46a
Enable `outline-atomics` by default on AArch64 Android
tgross35 Sep 5, 2025
d8eb386
Enable `outline-atomics` by default on AArch64 FreeBSD
tgross35 Sep 5, 2025
0f135f7
Enable `outline-atomics` by default on AArch64 Fuchsia
tgross35 Sep 5, 2025
b18a6c6
Enable `outline-atomics` by default on AArch64 OpenBSD
tgross35 Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_apple_ios.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pub(crate) fn target() -> Target {
.into(),
arch,
options: TargetOptions {
features: "+neon,+fp-armv8,+apple-a7".into(),
features: "+neon,+fp-armv8,+apple-a7,+outline-atomics".into(),
max_atomic_width: Some(128),
supported_sanitizers: SanitizerSet::ADDRESS | SanitizerSet::THREAD,
..opts
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_apple_tvos.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pub(crate) fn target() -> Target {
.into(),
arch,
options: TargetOptions {
features: "+neon,+fp-armv8,+apple-a7".into(),
features: "+neon,+fp-armv8,+apple-a7,+outline-atomics".into(),
max_atomic_width: Some(128),
..opts
},
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_apple_visionos.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pub(crate) fn target() -> Target {
.into(),
arch,
options: TargetOptions {
features: "+neon,+fp-armv8,+apple-a16".into(),
features: "+neon,+fp-armv8,+apple-a16,+outline-atomics".into(),
max_atomic_width: Some(128),
supported_sanitizers: SanitizerSet::ADDRESS | SanitizerSet::THREAD,
..opts
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_apple_watchos.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pub(crate) fn target() -> Target {
.into(),
arch,
options: TargetOptions {
features: "+v8a,+neon,+fp-armv8,+apple-a7".into(),
features: "+v8a,+neon,+fp-armv8,+apple-a7,+outline-atomics".into(),
max_atomic_width: Some(128),
dynamic_linking: false,
position_independent_executables: true,
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_linux_android.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ pub(crate) fn target() -> Target {
max_atomic_width: Some(128),
// As documented in https://developer.android.com/ndk/guides/cpu-features.html
// the neon (ASIMD) and FP must exist on all android aarch64 targets.
features: "+v8a,+neon,+fp-armv8".into(),
features: "+v8a,+neon,+fp-armv8,+outline-atomics".into(),
// the AAPCS64 expects use of non-leaf frame pointers per
// https://github.com/ARM-software/abi-aa/blob/4492d1570eb70c8fd146623e0db65b2d241f12e7/aapcs64/aapcs64.rst#the-frame-pointer
// and we tend to encounter interesting bugs in AArch64 unwinding code if we do not
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_pc_windows_gnullvm.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ use crate::spec::{FramePointer, Target, TargetMetadata, base};
pub(crate) fn target() -> Target {
let mut base = base::windows_gnullvm::opts();
base.max_atomic_width = Some(128);
base.features = "+v8a,+neon,+fp-armv8".into();
base.features = "+v8a,+neon,+fp-armv8,+outline-atomics".into();
base.linker = Some("aarch64-w64-mingw32-clang".into());

// Microsoft recommends enabling frame pointers on Arm64 Windows.
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_pc_windows_msvc.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ use crate::spec::{FramePointer, Target, TargetMetadata, base};
pub(crate) fn target() -> Target {
let mut base = base::windows_msvc::opts();
base.max_atomic_width = Some(128);
base.features = "+v8a,+neon,+fp-armv8".into();
base.features = "+v8a,+neon,+fp-armv8,+outline-atomics".into();

// Microsoft recommends enabling frame pointers on Arm64 Windows.
// From https://learn.microsoft.com/en-us/cpp/build/arm64-windows-abi-conventions?view=msvc-170#integer-registers
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_unknown_freebsd.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ pub(crate) fn target() -> Target {
data_layout: "e-m:e-p270:32:32-p271:32:32-p272:64:64-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128-Fn32".into(),
arch: "aarch64".into(),
options: TargetOptions {
features: "+v8a".into(),
features: "+v8a,+outline-atomics".into(),
max_atomic_width: Some(128),
stack_probes: StackProbeType::Inline,
supported_sanitizers: SanitizerSet::ADDRESS
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_unknown_fuchsia.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use crate::spec::{
pub(crate) fn target() -> Target {
let mut base = base::fuchsia::opts();
base.cpu = "generic".into();
base.features = "+v8a,+crc,+aes,+sha2,+neon".into();
base.features = "+v8a,+crc,+aes,+sha2,+neon,+outline-atomics".into();
base.max_atomic_width = Some(128);
base.stack_probes = StackProbeType::Inline;
base.supported_sanitizers = SanitizerSet::ADDRESS
Expand Down
2 changes: 1 addition & 1 deletion compiler/rustc_target/src/spec/targets/aarch64_unknown_openbsd.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ pub(crate) fn target() -> Target {
data_layout: "e-m:e-p270:32:32-p271:32:32-p272:64:64-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128-Fn32".into(),
arch: "aarch64".into(),
options: TargetOptions {
features: "+v8a".into(),
features: "+v8a,+outline-atomics".into(),
max_atomic_width: Some(128),
stack_probes: StackProbeType::Inline,
..base::openbsd::opts()
Expand Down
8 changes: 4 additions & 4 deletions ...ns/compiler-builtins/src/aarch64_linux.rs → ...r-builtins/src/aarch64_outline_atomics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ macro_rules! compare_and_swap {
"cbnz w17, 0b",
"1:",
"ret",
have_lse = sym crate::aarch64_linux::HAVE_LSE_ATOMICS,
have_lse = sym crate::aarch64_outline_atomics::HAVE_LSE_ATOMICS,
}
}
}
Expand Down Expand Up @@ -228,7 +228,7 @@ macro_rules! compare_and_swap_i128 {
"cbnz w15, 0b",
"1:",
"ret",
have_lse = sym crate::aarch64_linux::HAVE_LSE_ATOMICS,
have_lse = sym crate::aarch64_outline_atomics::HAVE_LSE_ATOMICS,
}
}
}
Expand Down Expand Up @@ -256,7 +256,7 @@ macro_rules! swap {
concat!(stxr!($ordering, $bytes), " w17, ", reg!($bytes, 16), ", [x1]"),
"cbnz w17, 0b",
"ret",
have_lse = sym crate::aarch64_linux::HAVE_LSE_ATOMICS,
have_lse = sym crate::aarch64_outline_atomics::HAVE_LSE_ATOMICS,
}
}
}
Expand Down Expand Up @@ -286,7 +286,7 @@ macro_rules! fetch_op {
concat!(stxr!($ordering, $bytes), " w15, ", reg!($bytes, 17), ", [x1]"),
"cbnz w15, 0b",
"ret",
have_lse = sym crate::aarch64_linux::HAVE_LSE_ATOMICS,
have_lse = sym crate::aarch64_outline_atomics::HAVE_LSE_ATOMICS,
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions library/compiler-builtins/compiler-builtins/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ pub mod arm;
#[cfg(any(target_arch = "aarch64", target_arch = "arm64ec"))]
pub mod aarch64;

#[cfg(all(target_arch = "aarch64", target_os = "linux"))]
pub mod aarch64_linux;
#[cfg(all(target_arch = "aarch64", target_feature = "outline-atomics"))]
pub mod aarch64_outline_atomics;

#[cfg(all(
kernel_user_helpers,
Expand Down
35 changes: 31 additions & 4 deletions library/std/src/sys/configure_builtins.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,35 @@
/// Hook into .init_array to enable LSE atomic operations at startup, if
/// supported.
#[cfg(all(target_arch = "aarch64", target_os = "linux", not(feature = "compiler-builtins-c")))]
/// Enable LSE atomic operations at startup, if supported.
///
/// Linker sections are based on what [`ctor`] does, with priorities to run slightly before user
/// code:
///
/// - Apple uses the section `__mod_init_func`, `mod_init_funcs` is needed to set
/// `S_MOD_INIT_FUNC_POINTERS`. There doesn't seem to be a way to indicate priorities.
/// - Windows uses `.CRT$XCT`, which is run before user constructors (these should use `.CRT$XCU`).
/// - ELF uses `.init_array` with a priority of 90, which runs before our `ARGV_INIT_ARRAY`
/// initializer (priority 99). Both are within the 0-100 implementation-reserved range, per docs
/// for the [`prio-ctor-dtor`] warning, and this matches compiler-rt's `CONSTRUCTOR_PRIORITY`.
///
/// To save startup time, the initializer is only run if outline atomic routines from
/// compiler-builtins may be used. If LSE is known to be available then the calls are never
/// emitted, and if we build the C intrinsics then it has its own initializer using the symbol
/// `__aarch64_have_lse_atomics`.
///
/// [`ctor`]: https://github.com/mmastrac/rust-ctor/blob/63382b833ddcbfb8b064f4e86bfa1ed4026ff356/shared/src/macros/mod.rs#L522-L534
/// [`prio-ctor-dtor`]: https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
Comment on lines +1 to +19
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reflecting on it, I think it'd be useful with a comment here why we do this in a static initialiser, and aren't doing it in std's init function in rt.rs?

I can think of an answer myself: because this affects atomics, which are in core, and because it'd be confusing if #![no_main] negatively affected atomics performance - but in that case, why aren't we doing the initialisation in core to start with, to not affect #![no_std]? The answer here is probably because we need runtime feature detection, which is only available in std.

But again, it'd be nice to have a discussion about this in the docs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would make sense as an addition.

#[cfg(all(
target_arch = "aarch64",
target_feature = "outline-atomics",
not(target_feature = "lse"),
not(feature = "compiler-builtins-c"),
))]
#[used]
#[unsafe(link_section = ".init_array.90")]
#[cfg_attr(target_vendor = "apple", unsafe(link_section = "__DATA,__mod_init_func,mod_init_funcs"))]
#[cfg_attr(target_os = "windows", unsafe(link_section = ".CRT$XCT"))]
#[cfg_attr(
not(any(target_vendor = "apple", target_os = "windows")),
unsafe(link_section = ".init_array.90")
)]
static RUST_LSE_INIT: extern "C" fn() = {
extern "C" fn init_lse() {
use crate::arch;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is actually for line 37 below. Shouldn't the comment be updated to refer to the renamed file?

-// This is provided by compiler-builtins::aarch64_linux.
+// This is provided by compiler-builtins::aarch64_outline_atomics.

Expand Down
Loading