Implement profiling for compiler-generated move/copy operations

[jsgf]

As part of Rust's move semantics, the compiler will generate memory copy operations to move objects about. These are generally pretty small, and the backend is good at optimizing them. But sometimes, if the type is large, they can end up being surprisingly expensive. In such cases, you might want to pass them by reference, or Box them up.

However, these moves are also invisible to profiling. At best they appear as a memcpy, but one memcpy is basically indistinguishable from another, and its very hard to know that 1) it's actually a compiler-generated copy, and 2) what type it pertains to.

This PR adds two new pseudo-intrinsic functions in core::intrinsics:

pub fn compiler_move<T, const SIZE: usize>(_src: *const T, _dst: *mut T);
pub fn compiler_copy<T, const SIZE: usize>(_src: *const T, _dst: *mut T);

These functions are never actually called however. A MIR transform pass -- instrument_moves.rs -- will locate all Operand::Move/Copy operations, and modify their source location to make them appear as if they had been inlined from compiler_move/_copy.

These functions have two generic parameters: the type being copied, and its size in bytes. This should make it very easy to identify which types are being expensive in your program (both in aggregate, and at specific hotspots). The size isn't strictly necessary since you can derive it from the type, but it's small and it makes it easier to understand what you're looking at.

This functionality is only enabled if you have debug info generation enabled, and also set the -Zinstrument-moves option.

It does not instrument all moves. By default it will only annotate ones for types over 64 bytes. The -Zinstrument-moves-size-limit specifies the size in bytes to start instrumenting for.

This has minimal impact on the size of debugging info. For rustc itself, the overall increase in librustc_driver*.so size is around .05% for 65 byte limit, 0.004% for 1025 byte limit, and a worst case of 0.6% for an 8 byte limit.

There's no effect on generated code, it only adds debug info.

As an example of a backtrace:

Breakpoint 1.3, __memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:255
255	ENTRY_P2ALIGN (MEMMOVE_SYMBOL (__memmove, unaligned_erms), 6)
(gdb) bt
 # 0  __memcpy_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:255
 # 1  0x0000555555590e7e in core::intrinsics::compiler_copy<[u64; 1000], 8000> () at library/core/src/intrinsics/mod.rs:10
 # 2  t::main () at t.rs:10

[rustbot]

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred to the intrinsics. Make sure the CTFE / Miri interpreter
gets adapted for the changes, if necessary.

cc @rust-lang/miri, @RalfJung, @oli-obk, @lcnr

[rustbot]

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[jsgf]

I'm not really sure whether Statement::Assign's rvalues covers all the interesting cases or not. I'd like to make sure it covers:

• parameter passing I was missing TerminatorKind::Call
• returns
• assignment
• initialization
• ...anything else?

[RalfJung]

Interesting idea!

Is it really worth distinguishing moves and copies? That doesn't make much of a difference for the runtime code, it's mostly a type system thing.

I'm not really sure whether Statement::Assign's rvalues covers all the interesting cases or not. I'd like to make sure it covers:

In MIR these will be spread across various places. The codegen backend would have an easier time centralizing all the ways in which operand uses get codegen'd as memcpy. But I am not sure if there's still a good way to adjust debuginfo there...

Isn't there a mutating MIR visitor you can use that traverses all operands?

[RalfJung]

These aren't intrinsics so I don't think this is the best place for them. The file is already too big anyway.^^

[jsgf]

Yeah I wasn't sure exactly where to put them. Originally I had the idea of actually making them real functions implementing copy & move in terms of calls to them, but that seemed more fiddly than it's worth.

[jsgf]

Is it really worth distinguishing moves and copies? That doesn't make much of a difference for the runtime code, it's mostly a type system thing.

I think in practice it's useful - I've seen very large structures being made Copy just because all their fields allow it and then being copied around unexpectedly. It would be nice to be able to see that, and distinguish it from regular Move.

The codegen backend would have an easier time centralizing all the ways in which operand uses get codegen'd as memcpy. But I am not sure if there's still a good way to adjust debuginfo there...

That was actually my first attempt but I ended up with a stream of mysterious crashes/assertion failures from within the guts of llvm. Doing the manipulations at the MIR level turned out to be much more straightforward.

Isn't there a mutating MIR visitor you can use that traverses all operands?

I'll take another look.

[jsgf]

@RalfJung

• I created core::profiling to hold compiler_copy/compiler_move
• I looked at the visit_operand visitor; it's appealing but it doesn't give access to change the containing source info.

[jsgf]

This has minimal impact on the size of debugging info. For rustc itself, the overall increase in librustc_driver*.so size is around .05% for 65 byte limit, 0.004% for 1025 byte limit, and a worst case of 0.6% for an 8 byte limit.

I was missing parameter passing moves the first time around, so it's a little larger now: about 0.2% for 65 byte, about (almost nothing) for 1024 and closer to 1% for 8 byte.

[rust-log-analyzer]

The job x86_64-gnu-llvm-20 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

test [mir-opt] tests/mir-opt/unusual_item_types.rs ... ok

failures:

---- [mir-opt] tests/mir-opt/instrument-moves/iter.rs stdout ----
7         debug iter => _1;
8         scope 2 (inlined test_impl_trait_arg::<std::iter::Chain<Map<std::array::IntoIter<u64, 5>, fn(u64) -> u64>, Map<std::array::IntoIter<u64, 5>, fn(u64) -> u64>>>) {
9             debug iter => _1;
-             scope 3 (inlined core::profiling::compiler_move::<std::iter::Chain<Map<std::array::IntoIter<u64, 5>, fn(u64) -> u64>, Map<std::array::IntoIter<u64, 5>, fn(u64) -> u64>>, 128>) {
+             scope 3 (inlined core::profiling::compiler_move::<std::iter::Chain<Map<std::array::IntoIter<u64, 5>, fn(u64) -> u64>, Map<std::array::IntoIter<u64, 5>, fn(u64) -> u64>>, 104>) {
11             }
12         }
13     }