Skip to content

[InstCombine] Push freeze through non-recurrence PHIs #157678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
+166 −25
Open

[InstCombine] Push freeze through non-recurrence PHIs #157678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3ece5e0
[InstCombine] Push freeze through non-recurrence PHIs
c-rhodes Sep 8, 2025
83986a4
address comments
c-rhodes Sep 10, 2025
73e6bc6
replace isa<InvokeInst> with generic check for terminator
c-rhodes Sep 10, 2025
c0d7cde
address comments
c-rhodes Sep 22, 2025
df6755d
address comments
c-rhodes Sep 23, 2025
8b9e1ec
Revert "address comments"
c-rhodes Sep 23, 2025
05d8a51
fix compile-time issue in spec2017 527.cam4_r
c-rhodes Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 32 additions & 3 deletions llvm/lib/Transforms/InstCombine/InstructionCombining.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4976,10 +4976,33 @@ InstCombinerImpl::pushFreezeToPreventPoisonFromPropagating(FreezeInst &OrigFI) {
// Op1.fr = Freeze(Op1)
// ... = Inst(Op1.fr, NonPoisonOps...)

auto CanPushFreeze = [](Value *V) {
if (!isa<Instruction>(V) || isa<PHINode>(V))
auto CanPushFreeze = [this](Value *V) {
if (!isa<Instruction>(V))
return false;

if (auto *PN = dyn_cast<PHINode>(V)) {
BasicBlock *BB = PN->getParent();
SmallPtrSet<BasicBlock *, 8> VisitedBBs;
for (Use &U : PN->incoming_values()) {
BasicBlock *InBB = PN->getIncomingBlock(U);
// We can't move freeze if the start value is the result of a
// terminator (e.g. an invoke).
if (auto *OpI = dyn_cast<Instruction>(U)) {
if (OpI->isTerminator())
return false;
}

// If there's multiple incoming edges from the same predecessor we must
// ensure the freeze isn't pushed to a single one of the uses,
// invalidating the iterator. We simply don't support this case, but it
// could be handled if there's a use case.
if (isBackEdge(InBB, BB) || !VisitedBBs.insert(InBB).second ||
match(U.get(), m_Undef()))
return false;
VisitedBBs.insert(InBB);
}
}

// We can't push the freeze through an instruction which can itself create
// poison. If the only source of new poison is flags, we can simply
// strip them (since we know the only use is the freeze and nothing can
Expand All @@ -5003,8 +5026,14 @@ InstCombinerImpl::pushFreezeToPreventPoisonFromPropagating(FreezeInst &OrigFI) {
if (U == OrigUse)
return nullptr;

if (isa<PoisonValue>(V))
continue;

auto *UserI = cast<Instruction>(U->getUser());
Builder.SetInsertPoint(UserI);
if (auto *PN = dyn_cast<PHINode>(UserI))
Builder.SetInsertPoint(PN->getIncomingBlock(*U)->getTerminator());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure why this change is needed. Before this PR CanPushFreeze always returned false for PHI nodes and the existing Builder.SetInsertPoint(UserI); seemed to work perfectly fine. All that happens now is that we sometimes return false if it's a PHI node.

Is this fixing an existing bug?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the existing Builder.SetInsertPoint(UserI); hasn't been removed

Copy link
Contributor

@david-arm david-arm Sep 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps I'm missing something, but before your patch CanPushFreeze always returned false for PHI nodes which meant we were calling Builder.SetInsertPoint(UserI); for PHI nodes.

However, even in places where we still return false for PHI nodes same as before we're now calling Builder.SetInsertPoint(PN->getIncomingBlock(*U)->getTerminator());. For example, it looks like you've changed the insertion point for PHI nodes with undef as incoming value.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that CanPushFreeze() is called on the use, while this is checking for the user being a phi node. Previously we could not have ended up with a phi user, as that would imply that we have pushed through a phi.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok I think I understand now. So basically if CanPushFreeze returns true for a PHI node we fall through to code below that adds the operands of the PHI to the worklist. Then if CanPushFreeze returns false for one of those operands we'll end up asking for the user, which will be the PHI.

else
Builder.SetInsertPoint(UserI);
Value *Frozen = Builder.CreateFreeze(V, V->getName() + ".fr");
U->set(Frozen);
continue;
Expand Down
5 changes: 2 additions & 3 deletions llvm/test/Transforms/InstCombine/freeze-landingpad.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,13 @@ define i32 @propagate_freeze_in_landingpad() personality ptr null {
; CHECK-NEXT: [[RES1:%.*]] = invoke i32 @foo()
; CHECK-NEXT: to label [[NORMAL_RETURN]] unwind label [[EXCEPTIONAL_RETURN]]
; CHECK: normal_return:
; CHECK-NEXT: [[INC]] = add nuw nsw i32 [[X]], 1
; CHECK-NEXT: [[INC]] = add i32 [[X]], 1
; CHECK-NEXT: br label [[INVOKE_BB1]]
; CHECK: exceptional_return:
; CHECK-NEXT: [[PHI:%.*]] = phi i32 [ [[X]], [[INVOKE_BB1]] ], [ 0, [[INVOKE_BB2]] ]
; CHECK-NEXT: [[LANDING_PAD:%.*]] = landingpad { ptr, i32 }
; CHECK-NEXT: cleanup
; CHECK-NEXT: [[FR:%.*]] = freeze i32 [[PHI]]
; CHECK-NEXT: [[RES:%.*]] = shl i32 [[FR]], 1
; CHECK-NEXT: [[RES:%.*]] = shl i32 [[PHI]], 1
; CHECK-NEXT: ret i32 [[RES]]
;
entry:
Expand Down
7 changes: 4 additions & 3 deletions llvm/test/Transforms/InstCombine/freeze-phi.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,13 +94,14 @@ define i32 @two(i1 %cond, i32 %x, i32 %x2) {
; CHECK-LABEL: @two(
; CHECK-NEXT: br i1 [[COND:%.*]], label [[A:%.*]], label [[B:%.*]]
; CHECK: A:
; CHECK-NEXT: [[X_FR:%.*]] = freeze i32 [[X:%.*]]
; CHECK-NEXT: br label [[C:%.*]]
; CHECK: B:
; CHECK-NEXT: [[X2_FR:%.*]] = freeze i32 [[X2:%.*]]
; CHECK-NEXT: br label [[C]]
; CHECK: C:
; CHECK-NEXT: [[Y:%.*]] = phi i32 [ [[X:%.*]], [[A]] ], [ [[X2:%.*]], [[B]] ]
; CHECK-NEXT: [[Y_FR:%.*]] = freeze i32 [[Y]]
; CHECK-NEXT: ret i32 [[Y_FR]]
; CHECK-NEXT: [[Y:%.*]] = phi i32 [ [[X_FR]], [[A]] ], [ [[X2_FR]], [[B]] ]
; CHECK-NEXT: ret i32 [[Y]]
;
br i1 %cond, label %A, label %B
A:
Expand Down
144 changes: 128 additions & 16 deletions llvm/test/Transforms/InstCombine/freeze.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,15 +269,15 @@ define void @freeze_dominated_uses_catchswitch(i1 %c, i32 %x) personality ptr @_
; CHECK-NEXT: invoke void @use_i32(i32 0)
; CHECK-NEXT: to label %[[CLEANUP:.*]] unwind label %[[CATCH_DISPATCH:.*]]
; CHECK: [[IF_ELSE]]:
; CHECK-NEXT: [[X_FR:%.*]] = freeze i32 [[X]]
; CHECK-NEXT: invoke void @use_i32(i32 1)
; CHECK-NEXT: to label %[[CLEANUP]] unwind label %[[CATCH_DISPATCH]]
; CHECK: [[CATCH_DISPATCH]]:
; CHECK-NEXT: [[PHI:%.*]] = phi i32 [ 0, %[[IF_THEN]] ], [ [[X]], %[[IF_ELSE]] ]
; CHECK-NEXT: [[PHI:%.*]] = phi i32 [ 0, %[[IF_THEN]] ], [ [[X_FR]], %[[IF_ELSE]] ]
; CHECK-NEXT: [[CS:%.*]] = catchswitch within none [label %[[CATCH:.*]], label %catch2] unwind to caller
; CHECK: [[CATCH]]:
; CHECK-NEXT: [[CP:%.*]] = catchpad within [[CS]] [ptr null, i32 64, ptr null]
; CHECK-NEXT: [[PHI_FREEZE:%.*]] = freeze i32 [[PHI]]
; CHECK-NEXT: call void @use_i32(i32 [[PHI_FREEZE]]) [ "funclet"(token [[CP]]) ]
; CHECK-NEXT: call void @use_i32(i32 [[PHI]]) [ "funclet"(token [[CP]]) ]
; CHECK-NEXT: unreachable
; CHECK: [[CATCH2:.*:]]
; CHECK-NEXT: [[CP2:%.*]] = catchpad within [[CS]] [ptr null, i32 64, ptr null]
Expand Down Expand Up @@ -384,15 +384,16 @@ define i32 @freeze_phi_followed_by_phi(i1 %c, i32 %y, i32 %z) {
; CHECK-LABEL: define i32 @freeze_phi_followed_by_phi(
; CHECK-SAME: i1 [[C:%.*]], i32 [[Y:%.*]], i32 [[Z:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: [[Z_FR:%.*]] = freeze i32 [[Z]]
; CHECK-NEXT: [[Y_FR:%.*]] = freeze i32 [[Y]]
; CHECK-NEXT: br i1 [[C]], label %[[IF:.*]], label %[[JOIN:.*]]
; CHECK: [[IF]]:
; CHECK-NEXT: br label %[[JOIN]]
; CHECK: [[JOIN]]:
; CHECK-NEXT: [[X:%.*]] = phi i32 [ [[Y]], %[[IF]] ], [ [[Z]], %[[ENTRY]] ]
; CHECK-NEXT: [[PHI:%.*]] = phi i32 [ [[Z]], %[[IF]] ], [ [[Y]], %[[ENTRY]] ]
; CHECK-NEXT: [[FR:%.*]] = freeze i32 [[X]]
; CHECK-NEXT: call void @use_i32(i32 [[FR]])
; CHECK-NEXT: call void @use_i32(i32 [[FR]])
; CHECK-NEXT: [[X:%.*]] = phi i32 [ [[Y_FR]], %[[IF]] ], [ [[Z_FR]], %[[ENTRY]] ]
; CHECK-NEXT: [[PHI:%.*]] = phi i32 [ [[Z_FR]], %[[IF]] ], [ [[Y_FR]], %[[ENTRY]] ]
; CHECK-NEXT: call void @use_i32(i32 [[X]])
; CHECK-NEXT: call void @use_i32(i32 [[X]])
; CHECK-NEXT: ret i32 [[PHI]]
;
entry:
Expand Down Expand Up @@ -1141,11 +1142,10 @@ exit:
}

; We can remove this freeze as the incoming values to the PHI have the same
; well-defined start value and the GEP can't produce poison, but this is
; currently unsupported.
define void @fold_phi_noundef_start_value(ptr noundef %init, i1 %cond.0, i1 %cond.1, i1 %cond.2) {
; well-defined start value and the GEP can't produce poison.
define void @fold_phi_noundef_start_value(ptr noundef %init, i1 %cond.0, i1 %cond.1) {
; CHECK-LABEL: define void @fold_phi_noundef_start_value(
; CHECK-SAME: ptr noundef [[INIT:%.*]], i1 [[COND_0:%.*]], i1 [[COND_1:%.*]], i1 [[COND_2:%.*]]) {
; CHECK-SAME: ptr noundef [[INIT:%.*]], i1 [[COND_0:%.*]], i1 [[COND_1:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: br label %[[LOOP:.*]]
; CHECK: [[LOOP]]:
Expand All @@ -1156,12 +1156,11 @@ define void @fold_phi_noundef_start_value(ptr noundef %init, i1 %cond.0, i1 %con
; CHECK-NEXT: br label %[[LOOP_LATCH]]
; CHECK: [[LOOP_LATCH]]:
; CHECK-NEXT: [[IV_2:%.*]] = phi ptr [ [[IV_0]], %[[LOOP]] ], [ [[IV_1]], %[[IF_ELSE]] ]
; CHECK-NEXT: [[IV_2_FR:%.*]] = freeze ptr [[IV_2]]
; CHECK-NEXT: [[IV_2_FR_INT:%.*]] = ptrtoint ptr [[IV_2_FR]] to i64
; CHECK-NEXT: [[IV_2_FR_INT:%.*]] = ptrtoint ptr [[IV_2]] to i64
; CHECK-NEXT: [[IV_0_INT:%.*]] = ptrtoint ptr [[IV_0]] to i64
; CHECK-NEXT: [[IDX:%.*]] = sub i64 [[IV_0_INT]], [[IV_2_FR_INT]]
; CHECK-NEXT: [[IV_0_NEXT]] = getelementptr i8, ptr [[IV_0]], i64 [[IDX]]
; CHECK-NEXT: br i1 [[COND_2]], label %[[EXIT:.*]], label %[[LOOP]]
; CHECK-NEXT: br i1 [[COND_1]], label %[[EXIT:.*]], label %[[LOOP]]
; CHECK: [[EXIT]]:
; CHECK-NEXT: ret void
;
Expand All @@ -1183,7 +1182,120 @@ loop.latch:
%iv.0.int = ptrtoint ptr %iv.0 to i64
%idx = sub i64 %iv.0.int, %iv.2.fr.int
%iv.0.next = getelementptr i8, ptr %iv.0, i64 %idx
br i1 %cond.2, label %exit, label %loop
br i1 %cond.1, label %exit, label %loop

exit:
ret void
}

declare ptr @get_ptr()

; When the phi isn't a simple recurrence and has multiple inputs from the same
; predecessor, we need to be careful to avoid iterator invalidation. The phi
; must have identical values for the predecessor and at no point should the
; freeze be pushed to a single one of the uses, e.g.
;
; %iv.2 = phi ptr [ %iv.0, %loop ], [ %iv.1.fr, %if.else ], [ %iv.1, %if.else ]
;
; We simply don't support this case, although it could be handled if there's a
; use case.
define void @fold_phi_non_simple_recurrence_multiple_forward_edges(ptr noundef %init, i1 %cond.0, i1 %cond.1) {
; CHECK-LABEL: define void @fold_phi_non_simple_recurrence_multiple_forward_edges(
; CHECK-SAME: ptr noundef [[INIT:%.*]], i1 [[COND_0:%.*]], i1 [[COND_1:%.*]]) {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: br label %[[LOOP:.*]]
; CHECK: [[LOOP]]:
; CHECK-NEXT: [[IV_0:%.*]] = phi ptr [ [[INIT]], %[[ENTRY]] ], [ [[IV_0_NEXT:%.*]], %[[LOOP_LATCH:.*]] ]
; CHECK-NEXT: br i1 [[COND_0]], label %[[LOOP_LATCH]], label %[[IF_ELSE:.*]]
; CHECK: [[IF_ELSE]]:
; CHECK-NEXT: [[IV_1:%.*]] = call ptr @get_ptr()
; CHECK-NEXT: br i1 false, label %[[LOOP_LATCH]], label %[[LOOP_LATCH]]
; CHECK: [[LOOP_LATCH]]:
; CHECK-NEXT: [[IV_2:%.*]] = phi ptr [ [[IV_0]], %[[LOOP]] ], [ [[IV_1]], %[[IF_ELSE]] ], [ [[IV_1]], %[[IF_ELSE]] ]
; CHECK-NEXT: [[IV_2_FR:%.*]] = freeze ptr [[IV_2]]
; CHECK-NEXT: [[IV_2_FR_INT:%.*]] = ptrtoint ptr [[IV_2_FR]] to i64
; CHECK-NEXT: [[IV_0_INT:%.*]] = ptrtoint ptr [[IV_0]] to i64
; CHECK-NEXT: [[IDX:%.*]] = sub i64 [[IV_0_INT]], [[IV_2_FR_INT]]
; CHECK-NEXT: [[IV_0_NEXT]] = getelementptr i8, ptr [[IV_0]], i64 [[IDX]]
; CHECK-NEXT: br i1 [[COND_1]], label %[[EXIT:.*]], label %[[LOOP]]
; CHECK: [[EXIT]]:
; CHECK-NEXT: ret void
;
entry:
br label %loop

loop:
%iv.0 = phi ptr [ %init, %entry ], [ %iv.0.next, %loop.latch ]
br i1 %cond.0, label %loop.latch, label %if.else

if.else:
%iv.1 = call ptr @get_ptr()
br i1 %cond.0, label %loop.latch, label %loop.latch

loop.latch:
%iv.2 = phi ptr [ %iv.0, %loop ], [ %iv.1, %if.else ], [ %iv.1, %if.else ]
%iv.2.fr = freeze ptr %iv.2
%iv.2.fr.int = ptrtoint ptr %iv.2.fr to i64
%iv.0.int = ptrtoint ptr %iv.0 to i64
%idx = sub i64 %iv.0.int, %iv.2.fr.int
%iv.0.next = getelementptr i8, ptr %iv.0, i64 %idx
br i1 %cond.1, label %exit, label %loop

exit:
ret void
}

; When the phi input comes from an invoke, we need to be careful the freeze
; isn't pushed after the invoke.
define void @fold_phi_noundef_start_value_with_invoke(ptr noundef %init, i1 %cond.0, i1 %cond.1) personality ptr undef {
; CHECK-LABEL: define void @fold_phi_noundef_start_value_with_invoke(
; CHECK-SAME: ptr noundef [[INIT:%.*]], i1 [[COND_0:%.*]], i1 [[COND_1:%.*]]) personality ptr undef {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: br label %[[LOOP:.*]]
; CHECK: [[LOOP]]:
; CHECK-NEXT: [[IV_0:%.*]] = phi ptr [ [[INIT]], %[[ENTRY]] ], [ [[IV_0_NEXT:%.*]], %[[LOOP_LATCH:.*]] ]
; CHECK-NEXT: br i1 [[COND_0]], label %[[LOOP_LATCH]], label %[[IF_ELSE:.*]]
; CHECK: [[IF_ELSE]]:
; CHECK-NEXT: [[IV_1:%.*]] = invoke ptr @get_ptr()
; CHECK-NEXT: to label %[[LOOP_LATCH]] unwind label %[[UNWIND:.*]]
; CHECK: [[LOOP_LATCH]]:
; CHECK-NEXT: [[IV_2:%.*]] = phi ptr [ [[IV_0]], %[[LOOP]] ], [ [[IV_1]], %[[IF_ELSE]] ]
; CHECK-NEXT: [[IV_2_FR:%.*]] = freeze ptr [[IV_2]]
; CHECK-NEXT: [[IV_2_FR_INT:%.*]] = ptrtoint ptr [[IV_2_FR]] to i64
; CHECK-NEXT: [[IV_0_INT:%.*]] = ptrtoint ptr [[IV_0]] to i64
; CHECK-NEXT: [[IDX:%.*]] = sub i64 [[IV_0_INT]], [[IV_2_FR_INT]]
; CHECK-NEXT: [[IV_0_NEXT]] = getelementptr i8, ptr [[IV_0]], i64 [[IDX]]
; CHECK-NEXT: br i1 [[COND_1]], label %[[EXIT:.*]], label %[[LOOP]]
; CHECK: [[UNWIND]]:
; CHECK-NEXT: [[TMP0:%.*]] = landingpad i8
; CHECK-NEXT: cleanup
; CHECK-NEXT: unreachable
; CHECK: [[EXIT]]:
; CHECK-NEXT: ret void
;
entry:
br label %loop

loop:
%iv.0 = phi ptr [ %init, %entry ], [ %iv.0.next, %loop.latch ]
br i1 %cond.0, label %loop.latch, label %if.else

if.else:
%iv.1 = invoke ptr @get_ptr()
to label %loop.latch unwind label %unwind

loop.latch:
%iv.2 = phi ptr [ %iv.0, %loop ], [ %iv.1, %if.else ]
%iv.2.fr = freeze ptr %iv.2
%iv.2.fr.int = ptrtoint ptr %iv.2.fr to i64
%iv.0.int = ptrtoint ptr %iv.0 to i64
%idx = sub i64 %iv.0.int, %iv.2.fr.int
%iv.0.next = getelementptr i8, ptr %iv.0, i64 %idx
br i1 %cond.1, label %exit, label %loop

unwind:
landingpad i8 cleanup
unreachable

exit:
ret void
Expand Down
Loading