Skip to content

[AMDGPU][SIInsertWaitcnts] Wait on all LDS DMA operations when no aliasing store is found #170660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pierre-vh merged 2 commits into from
Dec 8, 2025
Merged

[AMDGPU][SIInsertWaitcnts] Wait on all LDS DMA operations when no aliasing store is found #170660

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
427d6c4
[AMDGPU][SIInsertWaitcnts] Wait on all LDS DMA operations when no ali…
Pierre-vh Dec 4, 2025
0551da5
Update fix
Pierre-vh Dec 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2007,18 +2007,26 @@ bool SIInsertWaitcnts::generateWaitcntInstBefore(MachineInstr &MI,

// LOAD_CNT is only relevant to vgpr or LDS.
unsigned RegNo = FIRST_LDS_VGPR;
bool FoundAliasingStore = false;
if (Ptr && Memop->getAAInfo()) {
const auto &LDSDMAStores = ScoreBrackets.getLDSDMAStores();
for (unsigned I = 0, E = LDSDMAStores.size(); I != E; ++I) {
if (MI.mayAlias(AA, *LDSDMAStores[I], true))
if (MI.mayAlias(AA, *LDSDMAStores[I], true)) {
FoundAliasingStore = true;
ScoreBrackets.determineWait(LOAD_CNT, RegNo + I + 1, Wait);
}
}
} else {
ScoreBrackets.determineWait(LOAD_CNT, RegNo, Wait);
}
if (Memop->isStore()) {

// TODO?: Is it possible to have cases where we'd alias with both a
// store tracked in LDSDMAStores, and one that isn't ? If so, the
// current system would only wait on the tracked store, and not the
// "generic" entry.
if (!FoundAliasingStore)
ScoreBrackets.determineWait(LOAD_CNT, RegNo, Wait);

if (Memop->isStore())
ScoreBrackets.determineWait(EXP_CNT, RegNo, Wait);
}
}

// Loop over use and def operands.
Expand Down
5 changes: 4 additions & 1 deletion llvm/test/CodeGen/AMDGPU/lds-dma-waits.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ define amdgpu_kernel void @buffer_load_lds_dword_10_arrays(<4 x i32> %rsrc, i32
; GFX9-NEXT: s_waitcnt vmcnt(2)
; GFX9-NEXT: ds_read_b32 v7, v9 offset:1792
; GFX9-NEXT: ; wave barrier
; GFX9-NEXT: s_waitcnt vmcnt(0)
Copy link
Contributor

@jayfoad jayfoad Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this wait needed? Or is the test deliberately using more slots than SIInsertWaitcnts can track, so we have to be conservative?

Copy link
Contributor Author

@Pierre-vh Pierre-vh Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that's precisely what I was trying to fix. The wait is required here because we ran out of tracking slots.
#162077 increases the number of tracking slots so this will be improved

; GFX9-NEXT: ds_read_b32 v8, v9 offset:2048
; GFX9-NEXT: ; wave barrier
; GFX9-NEXT: ds_read_b32 v9, v9 offset:2304
Expand Down Expand Up @@ -288,6 +289,7 @@ define amdgpu_kernel void @buffer_load_lds_dword_10_arrays(<4 x i32> %rsrc, i32
; GFX10-NEXT: s_waitcnt vmcnt(2)
; GFX10-NEXT: ds_read_b32 v7, v9 offset:1792
; GFX10-NEXT: ; wave barrier
; GFX10-NEXT: s_waitcnt vmcnt(0)
; GFX10-NEXT: ds_read_b32 v8, v9 offset:2048
; GFX10-NEXT: ; wave barrier
; GFX10-NEXT: ds_read_b32 v9, v9 offset:2304
Expand Down Expand Up @@ -374,7 +376,7 @@ define amdgpu_kernel void @global_load_lds_no_alias_ds_read(ptr addrspace(1) noc
; GFX9-NEXT: v_mov_b32_e32 v0, s0
; GFX9-NEXT: s_lshl_b32 s0, s3, 2
; GFX9-NEXT: v_mov_b32_e32 v1, s0
; GFX9-NEXT: s_waitcnt vmcnt(1)
; GFX9-NEXT: s_waitcnt vmcnt(0)
; GFX9-NEXT: ds_read_b32 v0, v0 offset:512
; GFX9-NEXT: s_waitcnt vmcnt(0)
; GFX9-NEXT: ds_read_b32 v1, v1 offset:768
Expand All @@ -397,6 +399,7 @@ define amdgpu_kernel void @global_load_lds_no_alias_ds_read(ptr addrspace(1) noc
; GFX10-NEXT: v_mov_b32_e32 v0, s0
; GFX10-NEXT: s_lshl_b32 s0, s3, 2
; GFX10-NEXT: v_mov_b32_e32 v1, s0
; GFX10-NEXT: s_waitcnt vmcnt(0)
; GFX10-NEXT: ds_read_b32 v0, v0 offset:512
; GFX10-NEXT: s_waitcnt vmcnt(0) lgkmcnt(15)
; GFX10-NEXT: ds_read_b32 v1, v1 offset:768
Expand Down
1 change: 1 addition & 0 deletions llvm/test/CodeGen/AMDGPU/waitcnt-unscoped.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ define amdgpu_kernel void @test_waitcnt(ptr addrspace(1) %global_buffer, ptr add
; CHECK-NEXT: ; sched_barrier mask(0x00000000)
; CHECK-NEXT: v_mov_b32_e32 v1, s2
; CHECK-NEXT: v_mov_b32_e32 v2, s3
; CHECK-NEXT: s_waitcnt vmcnt(1)
Copy link
Contributor

@jayfoad jayfoad Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an example, could you explain why this wait is required? If I understand the metadata correctly then we can assume that the "Global to LDS load" does not alias with either of the DS_WRITEs.

Copy link
Contributor Author

@Pierre-vh Pierre-vh Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's too conservative, I'll look into fixing it
I think we should let LDSDMAStores grow infinitely so we can always check the aliasing stores, and then if the ID is out of bounds we just use the default one.
It'd fix the TODO I added as well I believe.

; CHECK-NEXT: ds_write_b32 v1, v3
; CHECK-NEXT: ds_write_b32 v2, v3
; CHECK-NEXT: ; sched_barrier mask(0x00000000)
Expand Down
Loading