[SimplifyCFG] Probabilities associated with same condition are constant #155734
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mtrofin
requested review from
alanzhao1,
david-xl,
jdenny-ornl,
nikic and
rotateright
|
@llvm/pr-subscribers-llvm-transforms Author: Mircea Trofin (mtrofin) ChangesThe branch weights capture probability. The probability has everything to do with the (SSA) value the condition is predicated on, and nothing to do with the position in the CFG. Full diff: https://github.com/llvm/llvm-project/pull/155734.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
index ef110a6922f05..2ef44c6d2849c 100644
--- a/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
+++ b/llvm/lib/Transforms/Utils/SimplifyCFG.cpp
@@ -4808,23 +4808,12 @@ static bool SimplifyCondBranchToCondBranch(BranchInst *PBI, BranchInst *BI,
SelectInst *NV = cast<SelectInst>(
Builder.CreateSelect(PBICond, PBIV, BIV, PBIV->getName() + ".mux"));
PN.setIncomingValue(PBBIdx, NV);
- // Although the select has the same condition as PBI, the original branch
- // weights for PBI do not apply to the new select because the select's
- // 'logical' edges are incoming edges of the phi that is eliminated, not
- // the outgoing edges of PBI.
+ // The select has the same condition as PBI. The probabilities don't
+ // change.
if (HasWeights) {
- uint64_t PredCommon = PBIOp ? PredFalseWeight : PredTrueWeight;
- uint64_t PredOther = PBIOp ? PredTrueWeight : PredFalseWeight;
- uint64_t SuccCommon = BIOp ? SuccFalseWeight : SuccTrueWeight;
- uint64_t SuccOther = BIOp ? SuccTrueWeight : SuccFalseWeight;
- // The weight to PredCommonDest should be PredCommon * SuccTotal.
- // The weight to PredOtherDest should be PredOther * SuccCommon.
- uint64_t NewWeights[2] = {PredCommon * (SuccCommon + SuccOther),
- PredOther * SuccCommon};
-
- fitWeights(NewWeights);
-
- setBranchWeights(NV, NewWeights[0], NewWeights[1],
+ uint64_t TrueWeight = PBIOp ? PredFalseWeight : PredTrueWeight;
+ uint64_t FalseWeight = PBIOp ? PredTrueWeight : PredFalseWeight;
+ setBranchWeights(NV, TrueWeight, FalseWeight,
/*IsExpected=*/false);
}
}
diff --git a/llvm/test/Transforms/SimplifyCFG/preserve-branchweights.ll b/llvm/test/Transforms/SimplifyCFG/preserve-branchweights.ll
index 0f78e236b4248..f200de7503816 100644
--- a/llvm/test/Transforms/SimplifyCFG/preserve-branchweights.ll
+++ b/llvm/test/Transforms/SimplifyCFG/preserve-branchweights.ll
@@ -610,8 +610,8 @@ define i32 @SimplifyCondBranchToCondBranchSwapMissingWeight(i1 %cmpa, i1 %cmpb)
; CHECK-NEXT: [[CMPA_NOT:%.*]] = xor i1 [[CMPA:%.*]], true
; CHECK-NEXT: [[CMPB_NOT:%.*]] = xor i1 [[CMPB:%.*]], true
; CHECK-NEXT: [[BRMERGE:%.*]] = select i1 [[CMPA_NOT]], i1 true, i1 [[CMPB_NOT]]
-; CHECK-NEXT: [[DOTMUX:%.*]] = select i1 [[CMPA_NOT]], i32 0, i32 2, !prof [[PROF17:![0-9]+]]
-; CHECK-NEXT: [[OUTVAL:%.*]] = select i1 [[BRMERGE]], i32 [[DOTMUX]], i32 1, !prof [[PROF18:![0-9]+]]
+; CHECK-NEXT: [[DOTMUX:%.*]] = select i1 [[CMPA_NOT]], i32 0, i32 2, !prof [[PROF15]]
+; CHECK-NEXT: [[OUTVAL:%.*]] = select i1 [[BRMERGE]], i32 [[DOTMUX]], i32 1, !prof [[PROF17:![0-9]+]]
; CHECK-NEXT: ret i32 [[OUTVAL]]
;
block1:
@@ -636,7 +636,7 @@ define void @or_icmps_harmful(i32 %x, i32 %y, ptr %p) {
; CHECK-LABEL: @or_icmps_harmful(
; CHECK-NEXT: entry:
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sgt i32 [[X:%.*]], -1
-; CHECK-NEXT: br i1 [[EXPECTED_TRUE]], label [[EXIT:%.*]], label [[RARE:%.*]], !prof [[PROF19:![0-9]+]]
+; CHECK-NEXT: br i1 [[EXPECTED_TRUE]], label [[EXIT:%.*]], label [[RARE:%.*]], !prof [[PROF18:![0-9]+]]
; CHECK: rare:
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: br i1 [[EXPENSIVE]], label [[EXIT]], label [[FALSE:%.*]]
@@ -669,7 +669,7 @@ define void @or_icmps_harmful_inverted(i32 %x, i32 %y, ptr %p) {
; CHECK-LABEL: @or_icmps_harmful_inverted(
; CHECK-NEXT: entry:
; CHECK-NEXT: [[EXPECTED_FALSE:%.*]] = icmp sgt i32 [[X:%.*]], -1
-; CHECK-NEXT: br i1 [[EXPECTED_FALSE]], label [[RARE:%.*]], label [[EXIT:%.*]], !prof [[PROF20:![0-9]+]]
+; CHECK-NEXT: br i1 [[EXPECTED_FALSE]], label [[RARE:%.*]], label [[EXIT:%.*]], !prof [[PROF19:![0-9]+]]
; CHECK: rare:
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: br i1 [[EXPENSIVE]], label [[EXIT]], label [[FALSE:%.*]]
@@ -702,7 +702,7 @@ define void @or_icmps_probably_not_harmful(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sgt i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_TRUE]], i1 true, i1 [[EXPENSIVE]]
-; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF21:![0-9]+]], !unpredictable [[META22:![0-9]+]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF20:![0-9]+]], !unpredictable [[META21:![0-9]+]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -734,7 +734,7 @@ define void @or_icmps_not_that_harmful(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sgt i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_TRUE]], i1 true, i1 [[EXPENSIVE]]
-; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF23:![0-9]+]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF22:![0-9]+]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -766,7 +766,7 @@ define void @or_icmps_not_that_harmful_inverted(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sgt i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_TRUE]], i1 true, i1 [[EXPENSIVE]]
-; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF24:![0-9]+]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF23:![0-9]+]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -797,7 +797,7 @@ define void @or_icmps_useful(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sle i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_TRUE]], i1 true, i1 [[EXPENSIVE]]
-; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF25:![0-9]+]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF24:![0-9]+]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -828,7 +828,7 @@ define void @or_icmps_useful_inverted(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_FALSE:%.*]] = icmp sgt i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_FALSE]], i1 true, i1 [[EXPENSIVE]]
-; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF25]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[EXIT:%.*]], label [[FALSE:%.*]], !prof [[PROF24]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -889,7 +889,7 @@ define void @and_icmps_harmful(i32 %x, i32 %y, ptr %p) {
; CHECK-LABEL: @and_icmps_harmful(
; CHECK-NEXT: entry:
; CHECK-NEXT: [[EXPECTED_FALSE:%.*]] = icmp sgt i32 [[X:%.*]], -1
-; CHECK-NEXT: br i1 [[EXPECTED_FALSE]], label [[RARE:%.*]], label [[EXIT:%.*]], !prof [[PROF20]]
+; CHECK-NEXT: br i1 [[EXPECTED_FALSE]], label [[RARE:%.*]], label [[EXIT:%.*]], !prof [[PROF19]]
; CHECK: rare:
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: br i1 [[EXPENSIVE]], label [[FALSE:%.*]], label [[EXIT]]
@@ -922,7 +922,7 @@ define void @and_icmps_harmful_inverted(i32 %x, i32 %y, ptr %p) {
; CHECK-LABEL: @and_icmps_harmful_inverted(
; CHECK-NEXT: entry:
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sgt i32 [[X:%.*]], -1
-; CHECK-NEXT: br i1 [[EXPECTED_TRUE]], label [[EXIT:%.*]], label [[RARE:%.*]], !prof [[PROF19]]
+; CHECK-NEXT: br i1 [[EXPECTED_TRUE]], label [[EXIT:%.*]], label [[RARE:%.*]], !prof [[PROF18]]
; CHECK: rare:
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: br i1 [[EXPENSIVE]], label [[FALSE:%.*]], label [[EXIT]]
@@ -957,7 +957,7 @@ define void @and_icmps_not_that_harmful(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_FALSE:%.*]] = icmp sgt i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_FALSE]], i1 [[EXPENSIVE]], i1 false
-; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF26:![0-9]+]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF25:![0-9]+]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -989,7 +989,7 @@ define void @and_icmps_not_that_harmful_inverted(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sle i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_TRUE]], i1 [[EXPENSIVE]], i1 false
-; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF26]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF25]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -1020,7 +1020,7 @@ define void @and_icmps_useful(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_TRUE:%.*]] = icmp sgt i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_TRUE]], i1 [[EXPENSIVE]], i1 false
-; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF27:![0-9]+]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF26:![0-9]+]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -1051,7 +1051,7 @@ define void @and_icmps_useful_inverted(i32 %x, i32 %y, ptr %p) {
; CHECK-NEXT: [[EXPECTED_FALSE:%.*]] = icmp sle i32 [[X:%.*]], -1
; CHECK-NEXT: [[EXPENSIVE:%.*]] = icmp eq i32 [[Y:%.*]], 0
; CHECK-NEXT: [[OR_COND:%.*]] = select i1 [[EXPECTED_FALSE]], i1 [[EXPENSIVE]], i1 false
-; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF27]]
+; CHECK-NEXT: br i1 [[OR_COND]], label [[FALSE:%.*]], label [[EXIT:%.*]], !prof [[PROF26]]
; CHECK: false:
; CHECK-NEXT: store i8 42, ptr [[P:%.*]], align 1
; CHECK-NEXT: br label [[EXIT]]
@@ -1114,19 +1114,18 @@ exit:
; CHECK: [[PROF10]] = !{!"branch_weights", i32 8, i32 33}
; CHECK: [[PROF11]] = !{!"branch_weights", i32 112017436, i32 -735157296}
; CHECK: [[PROF12]] = !{!"branch_weights", i32 3, i32 5}
-; CHECK: [[PROF13]] = !{!"branch_weights", i32 22, i32 12}
+; CHECK: [[PROF13]] = !{!"branch_weights", i32 2, i32 3}
; CHECK: [[PROF14]] = !{!"branch_weights", i32 34, i32 21}
-; CHECK: [[PROF15]] = !{!"branch_weights", i32 33, i32 14}
+; CHECK: [[PROF15]] = !{!"branch_weights", i32 3, i32 2}
; CHECK: [[PROF16]] = !{!"branch_weights", i32 47, i32 8}
-; CHECK: [[PROF17]] = !{!"branch_weights", i32 6, i32 2}
-; CHECK: [[PROF18]] = !{!"branch_weights", i32 8, i32 2}
-; CHECK: [[PROF19]] = !{!"branch_weights", i32 99, i32 1}
-; CHECK: [[PROF20]] = !{!"branch_weights", i32 1, i32 99}
-; CHECK: [[PROF21]] = !{!"branch_weights", i32 199, i32 1}
-; CHECK: [[META22]] = !{}
-; CHECK: [[PROF23]] = !{!"branch_weights", i32 197, i32 1}
-; CHECK: [[PROF24]] = !{!"branch_weights", i32 100, i32 98}
-; CHECK: [[PROF25]] = !{!"branch_weights", i32 101, i32 99}
-; CHECK: [[PROF26]] = !{!"branch_weights", i32 1, i32 197}
-; CHECK: [[PROF27]] = !{!"branch_weights", i32 99, i32 101}
+; CHECK: [[PROF17]] = !{!"branch_weights", i32 8, i32 2}
+; CHECK: [[PROF18]] = !{!"branch_weights", i32 99, i32 1}
+; CHECK: [[PROF19]] = !{!"branch_weights", i32 1, i32 99}
+; CHECK: [[PROF20]] = !{!"branch_weights", i32 199, i32 1}
+; CHECK: [[META21]] = !{}
+; CHECK: [[PROF22]] = !{!"branch_weights", i32 197, i32 1}
+; CHECK: [[PROF23]] = !{!"branch_weights", i32 100, i32 98}
+; CHECK: [[PROF24]] = !{!"branch_weights", i32 101, i32 99}
+; CHECK: [[PROF25]] = !{!"branch_weights", i32 1, i32 197}
+; CHECK: [[PROF26]] = !{!"branch_weights", i32 99, i32 101}
;.
|
david-xl
reviewed
Sep 2, 2025
| // weights for PBI do not apply to the new select because the select's | ||
| // 'logical' edges are incoming edges of the phi that is eliminated, not | ||
| // the outgoing edges of PBI. | ||
| // The select has the same condition as PBI. The probabilities don't |
There was a problem hiding this comment.
It helps to add a brief description of the transformation (with a simple ascii art) to reason about weight update strategy.
There was a problem hiding this comment.
Here, the point is that it's irrespective of what's happening in the transformation, because the probabilities have to do with the value distribution of the condition of the conditional branch.
There was a problem hiding this comment.
I looked at the original change that introduced this. I am not convinced it is completely wrong though. The reason is that the branch probability is control flow dependent. For instance :
if (a) {
if (a) {
...
}
}
These two branches have the same condition, but the second branch is 100% biased.
The select case is tricky -- the 'select cmpa, , ' is result is intended to be used under the condition if (cmpa || cmpb), so the branch probability should change logically, however the select statement is no longer control dependent, so it should not use the control dependent branch prob. The dillema is that 1) the current impl is wrong if select is lowered to branch (but not the 'or statement', 2) however if some cfg lowering that pushes the select under the guarding control flow, then we will lose information (control dependent branch prob).
I am leaning toward making it consistent (i.e. the way your patch does).
There was a problem hiding this comment.
the branch probability is control flow dependent
Ah, true. Got it.
The dillema is that[...]
Here, the select itself is created in the same BB as the condition it governs it (see Builder, lines 4747, 4808). But ack on the overall issue. I'm thinking that ultimately, the general case here becomes a question for the next phase - where we look for profile value correctness. Potentially we could track the probabilities of values associated with branches, at least for validation, if not for derivation, too.
mtrofin
force-pushed
the
users/mtrofin/08-28-_simplifycfg_probabilities_associated_with_same_condition_are_constant
branch
from
c61eb49 to
9d2d78e
Compare
david-xl
approved these changes
Sep 4, 2025
mtrofin
force-pushed
the
users/mtrofin/08-28-_simplifycfg_probabilities_associated_with_same_condition_are_constant
branch
from
9d2d78e to
7b84436
Compare
mtrofin
deleted the
users/mtrofin/08-28-_simplifycfg_probabilities_associated_with_same_condition_are_constant
branch
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/138/builds/18585 Here is the relevant piece of the build log for the reference |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The branch weights capture probability. The probability has everything to do with the (SSA) value the condition is predicated on, and nothing to do with the position in the CFG.