[AMDGPU] Allocate scratch space for dVGPRs for CWSR #130055
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch only adds the instruction for disassembly support. We neither have an instrinsic nor codegen support, and it is unclear whether we actually want to ever have an intrinsic, given the fragile semantics. For now, it will be generated only by the backend in very specific circumstances.
This represents a hardware mode supported only for wave32 compute shaders. When enabled, we set the `.dynamic_vgpr_en` field of `.compute_registers` to true in the PAL metadata.
In dynamic VGPR mode, Waves must deallocate all VGPRs before exiting. If the shader program does not do this, hardware inserts `S_ALLOC_VGPR 0` before S_ENDPGM, but this may incur some performance cost. Therefore it's better if the compiler proactively generates that instruction. This patch extends `si-insert-waitcnts` to deallocate the VGPRs via a `S_ALLOC_VGPR 0` before any `S_ENDPGM` when in dynamic VGPR mode.
In dynamic VGPR mode, we can allocate up to 8 blocks of either 16 or 32 VGPRs (based on a chip-wide setting which we can model with a Subtarget feature). Update some of the subtarget helpers to reflect this. In particular: - getVGPRAllocGranule is set to the block size - getAddresableNumVGPR will limit itself to 8 * size of a block We also try to be more careful about how many VGPR blocks we allocate. Therefore, when deciding if we should revert scheduling after a given stage, we check that we haven't increased the number of VGPR blocks that need to be allocated.
The CWSR trap handler needs to save and restore the VGPRs. When dynamic VGPRs are in use, the fixed function hardware will only allocate enough space for one VGPR block. The rest will have to be stored in scratch, at offset 0. This patch allocates the necessary space by: - generating a prologue that checks at runtime if we're on a compute queue (since CWSR only works on compute queues); for this we will have to check the ME_ID bits of the ID_HW_ID2 register - if that is non-zero, we can assume we're on a compute queue and initialize the SP and FP with enough room for the dynamic VGPRs - forcing all compute entry functions to use a FP so they can access their locals/spills correctly (this isn't ideal but it's the quickest to implement) Note that at the moment we allocate enough space for the theoretical maximum number of VGPRs that can be allocated dynamically (for blocks of 16 registers, this will be 128, of which we subtract the first 16, which are already allocated by the fixed function hardware). Future patches may decide to allocate less if they can prove the shader never allocates that many blocks. Also note that this should not affect any reported stack sizes (e.g. PAL backend_stack_size etc).
rovka
requested review from
JanekvO,
arsenm,
jayfoad,
mbrkusanin,
perlfu,
rampitec and
shiltian
Member
|
@llvm/pr-subscribers-backend-amdgpu Author: Diana Picus (rovka) ChangesThe CWSR trap handler needs to save and restore the VGPRs. When dynamic VGPRs are in use, the fixed function hardware will only allocate enough space for one VGPR block. The rest will have to be stored in scratch, at offset 0. This patch allocates the necessary space by:
Note that at the moment we allocate enough space for the theoretical maximum number of VGPRs that can be allocated dynamically (for blocks of 16 registers, this will be 128, of which we subtract the first 16, which are already allocated by the fixed function hardware). Future patches may decide to allocate less if they can prove the shader never allocates that many blocks. Also note that this should not affect any reported stack sizes (e.g. PAL backend_stack_size etc). Patch is 27.97 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/130055.diff 9 Files Affected:
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 59cc08a59ed7c..b5196930a50f7 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -6020,8 +6020,13 @@ Frame Pointer
If the kernel needs a frame pointer for the reasons defined in
``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the
-kernel prolog. If a frame pointer is not required then all uses of the frame
-pointer are replaced with immediate ``0`` offsets.
+kernel prolog. On GFX12+, when dynamic VGPRs are enabled, the prologue will
+check if the kernel is running on a compute queue, and if so it will reserve
+some scratch space for any dynamic VGPRs that might need to be saved by the
+CWSR trap handler. In this case, the frame pointer will be initialized to
+a suitably aligned offset above this reserved area. If a frame pointer is not
+required then all uses of the frame pointer are replaced with immediate ``0``
+offsets.
.. _amdgpu-amdhsa-kernel-prolog-flat-scratch:
@@ -17133,33 +17138,35 @@ within a map that has been added by the same *vendor-name*.
.. table:: AMDPAL Code Object Hardware Stage Metadata Map
:name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table
- ========================== ============== ========= ===============================================================
- String Key Value Type Required? Description
- ========================== ============== ========= ===============================================================
- ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point.
- ".scratch_memory_size" integer Scratch memory size in bytes.
- ".lds_size" integer Local Data Share size in bytes.
- ".perf_data_buffer_size" integer Performance data buffer size in bytes.
- ".vgpr_count" integer Number of VGPRs used.
- ".agpr_count" integer Number of AGPRs used.
- ".sgpr_count" integer Number of SGPRs used.
- ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a
- directive to instruct the compiler to limit the VGPR usage to
- be less than or equal to the specified value (only set if
- different from HW default).
- ".sgpr_limit" integer SGPR count upper limit (only set if different from HW
- default).
- ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only).
- 3 integers
- ".wavefront_size" integer Wavefront size (only set if different from HW default).
- ".uses_uavs" boolean The shader reads or writes UAVs.
- ".uses_rovs" boolean The shader reads or writes ROVs.
- ".writes_uavs" boolean The shader writes to one or more UAVs.
- ".writes_depth" boolean The shader writes out a depth value.
- ".uses_append_consume" boolean The shader uses append and/or consume operations, either
- memory or GDS.
- ".uses_prim_id" boolean The shader uses PrimID.
- ========================== ============== ========= ===============================================================
+ =========================== ============== ========= ===============================================================
+ String Key Value Type Required? Description
+ =========================== ============== ========= ===============================================================
+ ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point.
+ ".scratch_memory_size" integer Scratch memory size in bytes.
+ ".lds_size" integer Local Data Share size in bytes.
+ ".perf_data_buffer_size" integer Performance data buffer size in bytes.
+ ".vgpr_count" integer Number of VGPRs used.
+ ".agpr_count" integer Number of AGPRs used.
+ ".sgpr_count" integer Number of SGPRs used.
+ ".dynamic_vgpr_saved_count" integer No Number of dynamic VGPRs that can be stored in scratch by the
+ CWSR trap handler. Only used on GFX12+.
+ ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a
+ directive to instruct the compiler to limit the VGPR usage to
+ be less than or equal to the specified value (only set if
+ different from HW default).
+ ".sgpr_limit" integer SGPR count upper limit (only set if different from HW
+ default).
+ ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only).
+ 3 integers
+ ".wavefront_size" integer Wavefront size (only set if different from HW default).
+ ".uses_uavs" boolean The shader reads or writes UAVs.
+ ".uses_rovs" boolean The shader reads or writes ROVs.
+ ".writes_uavs" boolean The shader writes to one or more UAVs.
+ ".writes_depth" boolean The shader writes out a depth value.
+ ".uses_append_consume" boolean The shader uses append and/or consume operations, either
+ memory or GDS.
+ ".uses_prim_id" boolean The shader uses PrimID.
+ =========================== ============== ========= ===============================================================
..
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
index 13e61756e3036..73c97a25f4d0a 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp
@@ -1439,8 +1439,15 @@ void AMDGPUAsmPrinter::EmitPALMetadata(const MachineFunction &MF,
MD->setEntryPoint(CC, MF.getFunction().getName());
MD->setNumUsedVgprs(CC, CurrentProgramInfo.NumVGPRsForWavesPerEU, Ctx);
- // Only set AGPRs for supported devices
+ // For targets that support dynamic VGPRs, set the number of saved dynamic
+ // VGPRs (if any) in the PAL metadata.
const GCNSubtarget &STM = MF.getSubtarget<GCNSubtarget>();
+ if (STM.isDynamicVGPREnabled() &&
+ MFI->getScratchReservedForDynamicVGPRs() > 0)
+ MD->setHwStage(CC, ".dynamic_vgpr_saved_count",
+ MFI->getScratchReservedForDynamicVGPRs() / 4);
+
+ // Only set AGPRs for supported devices
if (STM.hasMAIInsts()) {
MD->setNumUsedAgprs(CC, CurrentProgramInfo.NumAccVGPR);
}
diff --git a/llvm/lib/Target/AMDGPU/SIDefines.h b/llvm/lib/Target/AMDGPU/SIDefines.h
index 721601efcc804..8f9d099b25857 100644
--- a/llvm/lib/Target/AMDGPU/SIDefines.h
+++ b/llvm/lib/Target/AMDGPU/SIDefines.h
@@ -552,6 +552,7 @@ enum Id { // HwRegCode, (6) [5:0]
enum Offset : unsigned { // Offset, (5) [10:6]
OFFSET_MEM_VIOL = 8,
+ OFFSET_ME_ID = 8,
};
enum ModeRegisterMasks : uint32_t {
diff --git a/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp b/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp
index 97736e2410c18..430d1824ef464 100644
--- a/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp
@@ -691,17 +691,61 @@ void SIFrameLowering::emitEntryFunctionPrologue(MachineFunction &MF,
}
assert(ScratchWaveOffsetReg || !PreloadedScratchWaveOffsetReg);
- if (hasFP(MF)) {
+ unsigned Offset = FrameInfo.getStackSize() * getScratchScaleFactor(ST);
+ if (!mayReserveScratchForCWSR(MF)) {
+ if (hasFP(MF)) {
+ Register FPReg = MFI->getFrameOffsetReg();
+ assert(FPReg != AMDGPU::FP_REG);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_MOV_B32), FPReg).addImm(0);
+ }
+
+ if (requiresStackPointerReference(MF)) {
+ Register SPReg = MFI->getStackPtrOffsetReg();
+ assert(SPReg != AMDGPU::SP_REG);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_MOV_B32), SPReg).addImm(Offset);
+ }
+ } else {
+ // We need to check if we're on a compute queue - if we are, then the CWSR
+ // trap handler may need to store some VGPRs on the stack. The first VGPR
+ // block is saved separately, so we only need to allocate space for any
+ // additional VGPR blocks used. For now, we will make sure there's enough
+ // room for the theoretical maximum number of VGPRs that can be allocated.
+ // FIXME: Figure out if the shader uses fewer VGPRs in practice.
+ assert(hasFP(MF));
Register FPReg = MFI->getFrameOffsetReg();
assert(FPReg != AMDGPU::FP_REG);
- BuildMI(MBB, I, DL, TII->get(AMDGPU::S_MOV_B32), FPReg).addImm(0);
- }
-
- if (requiresStackPointerReference(MF)) {
Register SPReg = MFI->getStackPtrOffsetReg();
assert(SPReg != AMDGPU::SP_REG);
- BuildMI(MBB, I, DL, TII->get(AMDGPU::S_MOV_B32), SPReg)
- .addImm(FrameInfo.getStackSize() * getScratchScaleFactor(ST));
+ unsigned VGPRSize =
+ llvm::alignTo((ST.getAddressableNumVGPRs() -
+ AMDGPU::IsaInfo::getVGPRAllocGranule(&ST)) *
+ 4,
+ FrameInfo.getMaxAlign());
+ MFI->setScratchReservedForDynamicVGPRs(VGPRSize);
+
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_GETREG_B32), FPReg)
+ .addImm(AMDGPU::Hwreg::HwregEncoding::encode(
+ AMDGPU::Hwreg::ID_HW_ID2, AMDGPU::Hwreg::OFFSET_ME_ID, 1));
+ // The MicroEngine ID is 0 for the graphics queue, and 1 or 2 for compute
+ // (3 is unused, so we ignore it). Unfortunately, S_GETREG doesn't set
+ // SCC, so we need to check for 0 manually.
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_CMP_LG_U32)).addImm(0).addReg(FPReg);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_CMOVK_I32), FPReg).addImm(VGPRSize);
+ if (requiresStackPointerReference(MF)) {
+ // If at least one of the constants can be inlined, then we can use
+ // s_cselect. Otherwise, use a mov and cmovk.
+ if (AMDGPU::isInlinableLiteral32(Offset, ST.hasInv2PiInlineImm()) ||
+ AMDGPU::isInlinableLiteral32(Offset + VGPRSize,
+ ST.hasInv2PiInlineImm())) {
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_CSELECT_B32), SPReg)
+ .addImm(Offset + VGPRSize)
+ .addImm(Offset);
+ } else {
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_MOV_B32), SPReg).addImm(Offset);
+ BuildMI(MBB, I, DL, TII->get(AMDGPU::S_CMOVK_I32), SPReg)
+ .addImm(Offset + VGPRSize);
+ }
+ }
}
bool NeedsFlatScratchInit =
@@ -1831,9 +1875,17 @@ bool SIFrameLowering::hasFPImpl(const MachineFunction &MF) const {
return frameTriviallyRequiresSP(MFI) || MFI.isFrameAddressTaken() ||
MF.getSubtarget<GCNSubtarget>().getRegisterInfo()->hasStackRealignment(
MF) ||
+ mayReserveScratchForCWSR(MF) ||
MF.getTarget().Options.DisableFramePointerElim(MF);
}
+bool SIFrameLowering::mayReserveScratchForCWSR(
+ const MachineFunction &MF) const {
+ return MF.getSubtarget<GCNSubtarget>().isDynamicVGPREnabled() &&
+ AMDGPU::isEntryFunctionCC(MF.getFunction().getCallingConv()) &&
+ AMDGPU::isCompute(MF.getFunction().getCallingConv());
+}
+
// This is essentially a reduced version of hasFP for entry functions. Since the
// stack pointer is known 0 on entry to kernels, we never really need an FP
// register. We may need to initialize the stack pointer depending on the frame
diff --git a/llvm/lib/Target/AMDGPU/SIFrameLowering.h b/llvm/lib/Target/AMDGPU/SIFrameLowering.h
index 938c75099a3bc..9dac4bc8951e5 100644
--- a/llvm/lib/Target/AMDGPU/SIFrameLowering.h
+++ b/llvm/lib/Target/AMDGPU/SIFrameLowering.h
@@ -86,6 +86,10 @@ class SIFrameLowering final : public AMDGPUFrameLowering {
public:
bool requiresStackPointerReference(const MachineFunction &MF) const;
+
+ // Returns true if the function may need to reserve space on the stack for the
+ // CWSR trap handler.
+ bool mayReserveScratchForCWSR(const MachineFunction &MF) const;
};
} // end namespace llvm
diff --git a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
index 740f752bc93b7..6d75b83ea2223 100644
--- a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
+++ b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
@@ -455,6 +455,10 @@ class SIMachineFunctionInfo final : public AMDGPUMachineFunction,
unsigned NumSpilledSGPRs = 0;
unsigned NumSpilledVGPRs = 0;
+ // The size of the scratch space reserved for the CWSR trap handler to spill
+ // some of the dynamic VGPRs.
+ unsigned ScratchReservedForDynamicVGPRs = 0;
+
// Tracks information about user SGPRs that will be setup by hardware which
// will apply to all wavefronts of the grid.
GCNUserSGPRUsageInfo UserSGPRInfo;
@@ -780,6 +784,15 @@ class SIMachineFunctionInfo final : public AMDGPUMachineFunction,
BytesInStackArgArea = Bytes;
}
+ // This is only used if we need to save any dynamic VGPRs in scratch.
+ unsigned getScratchReservedForDynamicVGPRs() const {
+ return ScratchReservedForDynamicVGPRs;
+ }
+
+ void setScratchReservedForDynamicVGPRs(unsigned Size) {
+ ScratchReservedForDynamicVGPRs = Size;
+ }
+
// Add user SGPRs.
Register addPrivateSegmentBuffer(const SIRegisterInfo &TRI);
Register addDispatchPtr(const SIRegisterInfo &TRI);
diff --git a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
index adadf8e4e4e65..4c6d5f2d459f7 100644
--- a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
@@ -511,6 +511,14 @@ SIRegisterInfo::getLargestLegalSuperClass(const TargetRegisterClass *RC,
Register SIRegisterInfo::getFrameRegister(const MachineFunction &MF) const {
const SIFrameLowering *TFI = ST.getFrameLowering();
const SIMachineFunctionInfo *FuncInfo = MF.getInfo<SIMachineFunctionInfo>();
+
+ // If we need to reserve scratch space for saving the VGPRs, then we should
+ // use the frame register for accessing our own frame (which may start at a
+ // non-zero offset).
+ if (TFI->mayReserveScratchForCWSR(MF))
+ return TFI->hasFP(MF) ? FuncInfo->getFrameOffsetReg()
+ : FuncInfo->getStackPtrOffsetReg();
+
// During ISel lowering we always reserve the stack pointer in entry and chain
// functions, but never actually want to reference it when accessing our own
// frame. If we need a frame pointer we use it, but otherwise we can just use
diff --git a/llvm/test/CodeGen/AMDGPU/dynamic-vgpr-reserve-stack-for-cwsr.ll b/llvm/test/CodeGen/AMDGPU/dynamic-vgpr-reserve-stack-for-cwsr.ll
new file mode 100644
index 0000000000000..d420af4ca100c
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/dynamic-vgpr-reserve-stack-for-cwsr.ll
@@ -0,0 +1,263 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
+; RUN: llc -mtriple=amdgcn-amd-amdpal -mcpu=gfx1200 -mattr=+dynamic-vgpr < %s | FileCheck -check-prefix=CHECK %s
+
+; Make sure we use a stack pointer and allocate 112 * 4 bytes at the beginning of the stack.
+
+define amdgpu_cs void @amdgpu_cs() #0 {
+; CHECK-LABEL: amdgpu_cs:
+; CHECK: ; %bb.0:
+; CHECK-NEXT: s_getreg_b32 s33, hwreg(HW_REG_HW_ID2, 8, 1)
+; CHECK-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; CHECK-NEXT: s_cmp_lg_u32 0, s33
+; CHECK-NEXT: s_cmovk_i32 s33, 0x1c0
+; CHECK-NEXT: s_alloc_vgpr 0
+; CHECK-NEXT: s_endpgm
+ ret void
+}
+
+define amdgpu_kernel void @kernel() #0 {
+; CHECK-LABEL: kernel:
+; CHECK: ; %bb.0:
+; CHECK-NEXT: s_getreg_b32 s33, hwreg(HW_REG_HW_ID2, 8, 1)
+; CHECK-NEXT: s_delay_alu instid0(SALU_CYCLE_1)
+; CHECK-NEXT: s_cmp_lg_u32 0, s33
+; CHECK-NEXT: s_cmovk_i32 s33, 0x1c0
+; CHECK-NEXT: s_alloc_vgpr 0
+; CHECK-NEXT: s_endpgm
+ ret void
+}
+
+define amdgpu_cs void @with_local() #0 {
+; CHECK-LABEL: with_local:
+; CHECK: ; %bb.0:
+; CHECK-NEXT: s_getreg_b32 s33, hwreg(HW_REG_HW_ID2, 8, 1)
+; CHECK-NEXT: v_mov_b32_e32 v0, 13
+; CHECK-NEXT: s_cmp_lg_u32 0, s33
+; CHECK-NEXT: s_cmovk_i32 s33, 0x1c0
+; CHECK-NEXT: scratch_store_b8 off, v0, s33 scope:SCOPE_SYS
+; CHECK-NEXT: s_wait_storecnt 0x0
+; CHECK-NEXT: s_alloc_vgpr 0
+; CHECK-NEXT: s_endpgm
+ %local = alloca i32, addrspace(5)
+ store volatile i8 13, ptr addrspace(5) %local
+ ret void
+}
+
+; Check that we generate s_cselect for SP if we can fit
+; the offset in an inline constant.
+define amdgpu_cs void @with_calls_inline_const() #0 {
+; CHECK-LABEL: with_calls_inline_const:
+; CHECK: ; %bb.0:
+; CHECK-NEXT: s_getreg_b32 s33, hwreg(HW_REG_HW_ID2, 8, 1)
+; CHECK-NEXT: v_mov_b32_e32 v0, 15
+; CHECK-NEXT: s_cmp_lg_u32 0, s33
+; CHECK-NEXT: s_mov_b32 s1, callee@abs32@hi
+; CHECK-NEXT: s_cmovk_i32 s33, 0x1c0
+; CHECK-NEXT: s_mov_b32 s0, callee@abs32@lo
+; CHECK-NEXT: scratch_store_b8 off, v0, s33 scope:SCOPE_SYS
+; CHECK-NEXT: s_wait_storecnt 0x0
+; CHECK-NEXT: v_mov_b32_e32 v0, 0x47
+; CHECK-NEXT: s_cselect_b32 s32, 0x1d0, 16
+; CHECK-NEXT: s_swappc_b64 s[30:31], s[0:1]
+; CHECK-NEXT: s_alloc_vgpr 0
+; CHECK-NEXT: s_endpgm
+ %local = alloca i32, addrspace(5)
+ store volatile i8 15, ptr addrspace(5) %local
+ call amdgpu_gfx void @callee(i32 71)
+ ret void
+}
+
+; Check that we generate s_mov + s_cmovk if we can't
+; fit the offset for SP in an inline constant.
+define amdgpu_cs void @with_calls_no_inline_const() #0 {
+; CHECK-LABEL: with_calls_no_inline_const:
+; CHECK: ; %bb.0:
+; CHECK-NEXT: s_getreg_b32 s33, hwreg(HW_REG_HW_ID2, 8, 1)
+; CHECK-NEXT: v_mov_b32_e32 v0, 15
+; CHECK-NEXT: s_cmp_lg_u32 0, s33
+; CHECK-NEXT: s_mov_b32 s1, callee@abs32@hi
+; CHECK-NEXT: s_cmovk_i32 s33, 0x1c0
+; CHECK-NEXT: s_mov_b32 s0, callee@abs32@lo
+; CHECK-NEXT: scratch_store_b8 off, v0, s33 scope:SCOPE_SYS
+; CHECK-NEXT: s_wait_storecnt 0x0
+; CHECK-NEXT: v_mov_b32_e32 v0, 0x47
+; CHECK-NEXT: s_movk_i32 s32, 0x100
+; CHECK-NEXT: s_cmovk_i32 s32, 0x2c0
+; CHECK-NEXT: s_swappc_b64 s[30:31], s[0:1]
+; CHECK-NEXT: s_alloc_vgpr 0
+; CHECK-NEXT: s_endpgm
+ %local = alloca i32, i32 61, addrspace(5)
+ store volatile i8 15, ptr addrspace(5) %local
+ call amdgpu_gfx void @callee(i32 71)
+ ret void
+}
+
+; We're going to limit this to 16 VGPRs, so we need to spill the rest.
+define amdgpu_cs void @with_spills(ptr addrspace(1) %p1, ptr addrspace(1) %p2) #1 {
+; CHECK-LABEL: with_spills:
+; CHECK: ; %bb.0:
+; CHECK-NEXT: s_getreg_b32 s33, hwreg(HW_REG_HW_ID2, 8, 1)
+; CHECK-NEXT: global_load_b128 v[4:7], v[0:1], off offset:96
+; CHECK-NEXT: s_cmp_lg_u32 0, s33
+; CHECK-NEXT: s_cmovk_i32 s33, 0x1c0
+; CHECK-NEXT: s_wait_loadcnt 0x0
+; CHECK-NEXT: scratch_store_b128 off, v[4:7], s33 offset:80 ; 16-byte Folded Spill
+; CHECK-NEXT: s_clause 0x2
+; CHECK-NEXT: global_load_b128 v[8:11], v[0:1], off offset:112
+; CHECK-NEXT: global_load_b128 v[12:15], v[0:1], off offset:64
+; CHECK-NEXT: global_load_b128 v[4:7], v[0:1], off offset:80
+; CHECK-NEXT: s_wait_loadcnt 0x0
+; CHECK-NEXT: scratch_store_b128 off, v[4:7], s33 offset:64 ; 16-byte Folded Spill
+; CHECK-NEXT: global_load_b128 v[4:7], v[0:1], off offset:32
+; CHECK-NEXT: s_wait_loadcnt 0x0
+; CHECK-NEXT: scratch_store_b128 off, v[4:7], s33 offset:48 ; 16-byte Folded Spill
+; CHECK-NEXT: global_...
[truncated]
|
perlfu
reviewed
Mar 7, 2025
arsenm
reviewed
Mar 7, 2025
llvm/test/CodeGen/AMDGPU/dynamic-vgpr-reserve-stack-for-cwsr.ll
Outdated
Show resolved
Hide resolved
llvm/test/CodeGen/AMDGPU/dynamic-vgpr-reserve-stack-for-cwsr.ll
Outdated
Show resolved
Hide resolved
perlfu
approved these changes
Mar 11, 2025
Contributor
perlfu
left a comment
perlfu
left a comment
There was a problem hiding this comment.
LGTM
nit: can you note somewhere (in a comment) that ScratchReservedForDynamicVGPRs is in bytes -- the magic divide by 4 to set dynamic_vgpr_saved_count was not entirely obvious.
arsenm
approved these changes
Mar 12, 2025
qiaojbao
added a commit
to GPUOpen-Drivers/llvm-project
that referenced
this pull request
Mar 13, 2025
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/55/builds/8663 Here is the relevant piece of the build log for the reference |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The CWSR trap handler needs to save and restore the VGPRs. When dynamic VGPRs are in use, the fixed function hardware will only allocate enough space for one VGPR block. The rest will have to be stored in scratch, at offset 0.
This patch allocates the necessary space by:
Note that at the moment we allocate enough space for the theoretical maximum number of VGPRs that can be allocated dynamically (for blocks of 16 registers, this will be 128, of which we subtract the first 16, which are already allocated by the fixed function hardware). Future patches may decide to allocate less if they can prove the shader never allocates that many blocks.
Also note that this should not affect any reported stack sizes (e.g. PAL backend_stack_size etc).