don't apply temporary lifetime extension rules to non-extended super let

[dianne]

Reference PR: rust-lang/reference#1980

This changes the semantics for super let (and macros implemented in terms of it, such as pin!, format_args!, write!, and println!) as suggested by @theemathas in #145784 (comment), making super let initializers only count as extending expressions when the super let itself is within an extending block. Since super let initializers aren't temporary drop scopes, their temporaries outside of inner temporary scopes are effectively always extended, even when not in extending positions; this only affects two cases as far as I can tell:

• Block tail expressions in Rust 2024. This PR makes f(pin!({ &temp() })) drop temp() at the end of the block in Rust 2024, whereas previously it would live until after the call to f because syntactically the temp() was in an extending position as a result of super let in pin!'s expansion.
• super let nested within a non-extended super let is no longer extended. i.e. a normal let is required to treat super lets as extending (in which case nested super lets will also be extending).

Closes #145784

This is a breaking change. Both static and dynamic semantics are affected. The most likely breakage is for programs to stop compiling, but it's technically possible for drop order to silently change as well (as in #145784). Since this affects stable macros, it probably would need a crater run.

Nominating for discussion alongside #145784: @rustbot label +I-lang-nominated +I-libs-api-nominated

Tracking issue for super let: #139076

[rustbot]

r? @jackh726

rustbot has assigned @jackh726.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[dianne]

Copied from #145784 (comment), since I think this is a notable caveat of this PR and worth considering before approving it:

This comes with a bit of a gotcha in terms of temporary lifetimes: it might be strange that the temp() would live longer in

non_extending({ let x = { &temp() }; f(x) }); // ok

than in

non_extending({ super let x = { &temp() }; f(x) }); // error

Though the case for if let is similar: its scrutinee isn't a temporary scope and it doesn't have lifetime extension rules that can make block tail expressions' temporaries live longer.

[dianne]

Also copied since it motivates this PR: I think something like this may be necessary for the identity

&EXPR === { super let x = &EXPR; x }

to hold in both extending and non-extending contexts without changing how block tail expression scopes work. Substituting, e.g. { &temp() } in for EXPR, the identity only currently holds in extending contexts in Rust 2024. In non-extending contexts,

&{ &temp() }

drops temp() when leaving the block, but

{ super let x = &{ &temp() }; x }

would extend it to outlive x. This PR shortens the lifetime of temp() such that it's dropped at the end of the block in both cases. I haven't done any rigorous proof or extensive testing that this PR together with #145342 makes the identity always hold, however.

[traviscross]

To confirm, with this PR, does this behavior hold (in Rust 2024)?:

fn f<T>(_: LogDrop<'_>, x: T) -> T { x }

// These two should be the same.
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(2), &{ &raw const *&e.log(1) });
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(2), {
        super let v = &{ &raw const *&e.log(1) };
        v
    });
    drop(e.log(3));
});
// These two should be the same.
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(1), &&raw const *&e.log(2));
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| {
    let _v = f(e.log(1), {
        super let v = &&raw const *&e.log(2);
        v
    });
    drop(e.log(3));
});
// These two should be the same.
assert_drop_order(1..=2, |e| {
    let _v = &{ &raw const *&e.log(2) };
    drop(e.log(1));
});
assert_drop_order(1..=2, |e| {
    let _v = {
        super let v = &{ &raw const *&e.log(2) };
        v
    };
    drop(e.log(1));
});

Playground link

(If any of these are missing, please add them as tests.)

[theemathas]

super let nested within a non-extended super let is no longer extended

Does this PR affect any edition 2021 code?

[dianne]

To confirm, with this PR, does this behavior hold (in Rust 2024)?:

Those all hold under this PR, yes. I've added them all as tests (with some additional versioning to account for the Edition-dependent drop order in the first one); thanks!

super let nested within a non-extended super let is no longer extended

Does this PR affect any edition 2021 code?

Not that I'm aware of. That detail matters in Edition 2024, since the nested super let could have an extending borrow operator within a block tail expression in its initializer, which would previously extend the borrowed temporary's lifetime to outlive the outer super let's binding. In Edition 2021, that temporary outlives the outer super let's binding regardless, since block tail expressions (and super let initializers) aren't temporary scopes.

[traviscross]

Suggested change

	// We have extending borrow expressions within the initializer
	// expression.
	// We have extending borrow expressions within a non-extending
	// expression within the initializer expression.

(Revising my earlier text here.)

[traviscross]

This is correct, I believe. Let's propose to do it.

@rfcbot fcp merge

@dianne, let's document this in the Reference.

cc @rust-lang/lang-docs

[rfcbot]

Team member @traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[traviscross]

cc @m-ou-se @rust-lang/libs-api

[traviscross]

Interesting point. Our current order here is rather challenging to explain because one can't work from the inside out.

Relatedly, can you explain the behavior of "3"?:

assert_drop_order(1..=3, |e| { //~ 1.
    &({ &raw const *&e.log(1) }, drop(e.log(2)));
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| { //~ 2.
    { let _x; _x = &({ &raw const *&e.log(1) }, drop(e.log(2))); }
    drop(e.log(3));
});
assert_drop_order(1..=3, |e| { //~ 3.
    _ = &({ &raw const *&e.log(2) }, drop(e.log(1)));
    drop(e.log(3));
});

Playground link

[dianne]

I believe that's due to destructuring assignment expression desugaring. Since a pattern is being assigned to in an assignment expression, it actually turns into a let statement in the HIR to perform the destructuring, then an assignment expression is generated for each binding present. In this case, there's no bindings, so no assignment expressions are present in the final HIR.

assert_drop_order(#[lang = "range_inclusive_new"](1, 3), //~ 3.
    |e|
        {
            { let _ = &({ &raw const *&e.log(2) }, drop(e.log(1))); };
            drop(e.log(3));
        });

It's wrapped in a block, as above, which prevents temporaries from living past the end of the desugared destructuring assignment, but it does extend e.log(2) to live past the initializer's evaluation. Thus drop(e.log(1)) happens first.

[traviscross]

Interesting, makes sense. E.g., it does the same thing with:

assert_drop_order(1..=3, |e| { //~ 5.
    let _x; let _y;
    (_x, _y) = ({ &raw const *&e.log(2) }, drop(e.log(1)));
    drop(e.log(3));
});

We do document the desugaring of destructing assignments in the Reference, but we don't note anywhere the effect this has on drop order. Probably we should do that.

cc @rust-lang/lang-docs

This is, I think, an argument in favor of the semantic that you're proposing, as that would make the drop order between these cases more similar (as I understand your proposal). What do you think, would it make the drop order of temporaries in _ = $EXPR; and drop($EXPR); exactly the same, or would there be remaining differences?

[dianne]

I think they'd be the exactly the same. The same rules would be used to determine the temporaries that live to the end of the block in { let _ = $EXPR; } and those that live to the end of the statement in drop($EXPR);.

[jieyouxu]

Retrying the prepared-fails.
@craterbot run mode=build-and-test crates=https://crater-reports.s3.amazonaws.com/pr-145838/retry-regressed-list.txt p=1

[craterbot]

👌 Experiment pr-145838-1 created and queued.
🤖 Automatically detected try build b83b707
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[traviscross]

We'd be breaking, e.g.:

fn f(x: *const u8, y: *mut u8) -> u8 {
    unsafe { y.write(42); x.read() }
}

fn main() {
    let x = core::cell::RefCell::new((0u8, 0u8));
    // Sound under TB.
    f({ &raw const x.borrow().0 }, &raw mut x.borrow_mut().1);
}

...not that anyone would defend that code, I hope. I wonder if it'd break anything more defensible.

[jieyouxu]

Edit: Looking at the test failures, it seems like there's a lot of tests for nondeterministic behavior (lots in particular relying on timing or random numbers). I don't have time to go through them all at the moment, but I can try opening a PR against the denylist later (or a few if breaking them up would help in reviewing). I imagine these are the sorts of things that should be skip-tests = true on there?

Yes. Please do feel free to submit skip-tests for crates with non-det tests, you can cc me on the crater PR. Smaller PRs would make it easier, but not too fussed in any case.

[traviscross]

I'd suggest to go ahead and still submit PRs to the affected projects. Even if we might want to consider this new semantic, we may in the meanwhile not want to be accumulating code in the ecosystem that relies on this accidental divergence between the intended and the current behavior of super let. Even if we like the new semantic, it could always happen that we find some reason many releases from now that prevents us from stabilizing it, and then it'd be harder to close the divergence in the way this PR does.

[dianne]

We'd be breaking, e.g.:

Oh, true. I'll have to think on that. I'm not that familiar with what sorts of things are realistic in unsafe code though, so I may not be cut out to evaluate it entirely on my own.

---

Speaking of breakages due to lifetime changes, I don't think it's been mentioned yet, but I think this PR can silently introduce UB in cases where raw pointers' pointee lifespans are shortened. e.g. if anyone wrote

// Implicitly coerce to sneakily create a dangling pointer with `&`:
non_extending(pin!(if cond { raw_ptr } else { &temp() }));

or

non_extending(pin!({ Struct { raw_ptr: &temp() } }));

temp() wouldn't live past the block under this PR, so the pointer would be dangling. Currently (and under the alternative semantics I proposed), temp() is extended to outlive the result of pin!. One might expect it to be, too, since it is extended to live long enough under this PR if the pin! is extending:

let pin1 = pin!(if cond { raw_ptr } else { &temp() });
let pin2 = pin!({ Struct { raw_ptr: &temp() } });

We've talked about the case where this shortening causes compilation failures (due to borrow-checking), but it totally slipped my mind that people could be relying on subtle lifetime extension behavior in unsafe code like that.

[theemathas]

I've reviewed 13 of the test failures in #145838 (comment) (top-to-bottom from Arxoto.sudoku.78ccf4297d22f37e62e8bf07ff19f7f2e7843139 to LegendarySaiyan.structures.f375d1f242c501951f7a3a6c49a09a7895f57326). All of them are flaky tests. Made a crater PR.

[theemathas]

Saethlin told me that I should wait for the retry before checking the test failures.

[craterbot]

🚧 Experiment pr-145838-1 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-145838-1 is completed!
📊 697 regressed and 670 fixed (187804 total)
📊 580 spurious results on the retry-regessed-list.txt, consider a retry¹ if this is a significant amount.
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

Footnotes

1. re-run the experiment with crates=https://crater-reports.s3.amazonaws.com/pr-145838-1/retry-regressed-list.txt ↩

[dianne]

At a glance, the "build compiler error" failures look real, following the pattern of using &format!(...) in a block of a conditional expression inside a format argument (alongside at least one other format argument). The rest I've peeked at look spurious. The "fixed" crates are almost certainly all noise, so I'm expecting a comparable number of spurious regressions. I'll try and review them more comprehensively later.

---

In the mean time, I've implemented my lifetime extension suggestion for block tails. It's not ready for review yet, but it lives at #146098 now for testing/discussion purposes.

[jieyouxu]

Retrying since there's a lot of "cannot find directory"s.
@craterbot run mode=build-and-test crates=https://crater-reports.s3.amazonaws.com/pr-145838-1/retry-regressed-list.txt p=1

[craterbot]

👌 Experiment pr-145838-2 created and queued.
🤖 Automatically detected try build b83b707
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more