Skip to content

[AMDGPU] Fold into uses of splat REG_SEQUENCEs through COPYs. #145691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 4, 2025
Merged

[AMDGPU] Fold into uses of splat REG_SEQUENCEs through COPYs. #145691

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9300f4e
[AMDGPU] Fold into uses of splat REG_SEQUENCEs through COPYs.
kosarev Jun 25, 2025
5b81404
Add a MIR test.
kosarev Jul 20, 2025
081d3ab
Rename the registers and remove an unused register.
kosarev Aug 4, 2025
354354f
Merge branch 'main' into splat-copies
kosarev Aug 4, 2025
b8309ee
Merge branch 'main' into splat-copies
kosarev Aug 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion llvm/lib/Target/AMDGPU/SIFoldOperands.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1169,11 +1169,18 @@ void SIFoldOperandsImpl::foldOperand(
// Grab the use operands first
SmallVector<MachineOperand *, 4> UsesToProcess(
llvm::make_pointer_range(MRI->use_nodbg_operands(RegSeqDstReg)));
for (auto *RSUse : UsesToProcess) {
for (unsigned I = 0; I != UsesToProcess.size(); ++I) {
MachineOperand *RSUse = UsesToProcess[I];
MachineInstr *RSUseMI = RSUse->getParent();
unsigned OpNo = RSUseMI->getOperandNo(RSUse);

if (SplatRC) {
if (RSUseMI->isCopy()) {
Copy link
Collaborator Author

@kosarev kosarev Jun 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this could be isFoldableCopy(), but I hesitate to use it without test coverage.

Register DstReg = RSUseMI->getOperand(0).getReg();
append_range(UsesToProcess,
make_pointer_range(MRI->use_nodbg_operands(DstReg)));
continue;
}
if (tryFoldRegSeqSplat(RSUseMI, OpNo, SplatVal, SplatRC)) {
FoldableDef SplatDef(SplatVal, SplatRC);
appendFoldCandidate(FoldList, RSUseMI, OpNo, SplatDef);
Expand Down
10 changes: 2 additions & 8 deletions llvm/test/CodeGen/AMDGPU/packed-fp32.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3238,11 +3238,8 @@ define amdgpu_kernel void @fadd_fadd_fsub_0(<2 x float> %arg) {
; GFX90A-GISEL-LABEL: fadd_fadd_fsub_0:
; GFX90A-GISEL: ; %bb.0: ; %bb
; GFX90A-GISEL-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
; GFX90A-GISEL-NEXT: s_mov_b32 s2, 0
; GFX90A-GISEL-NEXT: s_mov_b32 s3, s2
; GFX90A-GISEL-NEXT: v_pk_mov_b32 v[0:1], s[2:3], s[2:3] op_sel:[0,1]
; GFX90A-GISEL-NEXT: s_waitcnt lgkmcnt(0)
; GFX90A-GISEL-NEXT: v_pk_add_f32 v[0:1], s[0:1], v[0:1]
; GFX90A-GISEL-NEXT: v_pk_add_f32 v[0:1], s[0:1], 0
Copy link
Contributor

@arsenm arsenm Jun 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only test change is one globalisel case, so that's a hint this is just hiding a missed optimization that should have happened earlier in the pipeline. Do you have another example where this is useful?

Copy link
Collaborator Author

@kosarev kosarev Jun 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed elsewhere.

Copy link
Contributor

@arsenm arsenm Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs mir test if we're going through with this

Copy link
Collaborator Author

@kosarev kosarev Jul 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

; GFX90A-GISEL-NEXT: v_mov_b32_e32 v0, v1
; GFX90A-GISEL-NEXT: v_pk_add_f32 v[0:1], v[0:1], 0
; GFX90A-GISEL-NEXT: v_mov_b32_e32 v2, s0
Expand All @@ -3253,11 +3250,8 @@ define amdgpu_kernel void @fadd_fadd_fsub_0(<2 x float> %arg) {
; GFX942-GISEL-LABEL: fadd_fadd_fsub_0:
; GFX942-GISEL: ; %bb.0: ; %bb
; GFX942-GISEL-NEXT: s_load_dwordx2 s[0:1], s[4:5], 0x24
; GFX942-GISEL-NEXT: s_mov_b32 s2, 0
; GFX942-GISEL-NEXT: s_mov_b32 s3, s2
; GFX942-GISEL-NEXT: v_mov_b64_e32 v[0:1], s[2:3]
; GFX942-GISEL-NEXT: s_waitcnt lgkmcnt(0)
; GFX942-GISEL-NEXT: v_pk_add_f32 v[0:1], s[0:1], v[0:1]
; GFX942-GISEL-NEXT: v_pk_add_f32 v[0:1], s[0:1], 0
; GFX942-GISEL-NEXT: s_nop 0
; GFX942-GISEL-NEXT: v_mov_b32_e32 v0, v1
; GFX942-GISEL-NEXT: v_pk_add_f32 v[0:1], v[0:1], 0
Expand Down
30 changes: 28 additions & 2 deletions llvm/test/CodeGen/AMDGPU/si-fold-reg-sequence.mir
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,44 @@
# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -run-pass=si-fold-operands -verify-machineinstrs -o - %s
# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 5
# RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx900 -run-pass=si-fold-operands -verify-machineinstrs -o - %s | FileCheck %s

# Check that we don't hang on this.
---
name: fold_reg_sequence
body: |
bb.0:
liveins: $vgpr0_vgpr1, $vgpr2

; CHECK-LABEL: name: fold_reg_sequence
; CHECK: liveins: $vgpr0_vgpr1, $vgpr2
; CHECK-NEXT: {{ $}}
; CHECK-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 0
; CHECK-NEXT: [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 429
; CHECK-NEXT: [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE killed [[S_MOV_B32_1]], %subreg.sub0, [[S_MOV_B32_]], %subreg.sub1
; CHECK-NEXT: [[V_MUL_HI_U32_e64_:%[0-9]+]]:vgpr_32 = V_MUL_HI_U32_e64 $vgpr2, [[REG_SEQUENCE]].sub0, implicit $exec
; CHECK-NEXT: [[GLOBAL_LOAD_DWORD:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD $vgpr0_vgpr1, 0, 0, implicit $exec :: (load (s32), addrspace 1)
; CHECK-NEXT: [[V_MUL_HI_U32_e64_1:%[0-9]+]]:vgpr_32 = V_MUL_HI_U32_e64 [[GLOBAL_LOAD_DWORD]], [[REG_SEQUENCE]].sub0, implicit $exec
; CHECK-NEXT: S_ENDPGM 0
%0:sreg_32 = S_MOV_B32 0
%1:sreg_32 = S_MOV_B32 429
%2:sreg_64 = REG_SEQUENCE killed %1, %subreg.sub0, %0, %subreg.sub1
%3:vgpr_32 = V_MUL_HI_U32_e64 $vgpr2, %2.sub0, implicit $exec
%4:vgpr_32 = GLOBAL_LOAD_DWORD $vgpr0_vgpr1, 0, 0, implicit $exec :: (load (s32), addrspace 1)
%5:vgpr_32 = V_MUL_HI_U32_e64 %4, %2.sub0, implicit $exec
S_ENDPGM 0

...

# Fold through a COPY of REG_SEQUENCE.
---
name: fold_through_copy
body: |
bb.0:
; CHECK-LABEL: name: fold_through_copy
; CHECK: [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 0
; CHECK-NEXT: [[DEF:%[0-9]+]]:sreg_64_xexec = IMPLICIT_DEF
; CHECK-NEXT: [[V_PK_ADD_F32_:%[0-9]+]]:vreg_64_align2 = nofpexcept V_PK_ADD_F32 8, [[DEF]], 8, 0, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
%0:sreg_32 = S_MOV_B32 0
%1:sreg_64 = REG_SEQUENCE %0:sreg_32, %subreg.sub0, %0:sreg_32, %subreg.sub1
%2:sreg_64_xexec = IMPLICIT_DEF
%3:vreg_64_align2 = COPY %1:sreg_64
%4:vreg_64_align2 = nofpexcept V_PK_ADD_F32 8, %2:sreg_64_xexec, 8, %3:vreg_64_align2, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
...