Fix a hang when debugging HLSL compute shaders with enum types

[qwmnerbvqwmn]

This PR fixes a thread hang issue in the HLSL debugger that occurs when stepping through or inspecting a compute shader that contains enum type definitions.

When debugging an HLSL code containing an enum, renderdoc will crash, such as the following HLSL code.

// 1. Define the enumeration
enum ColorMode {
    RedState = 0,
    BlueState = 1
};

// 2. Function that returns an enum type
ColorMode DeterminePixelMode(uint2 pos) {
    // Simple pattern logic: toggle mode every 50 pixels
    return (pos.x % 100 < 50) ? ColorMode::RedState : ColorMode::BlueState;
}

RWTexture2D<float4> OutputTex : register(u0);

[numthreads(8, 8, 1)]
void CSMain(uint3 id : SV_DispatchThreadID) {
    // 3. Receive and evaluate the enum type
    ColorMode mode = DeterminePixelMode(id.xy);
    
    float4 color = (mode == ColorMode::RedState) 
                   ? float4(1.0, 0.0, 0.0, 1.0)  // Red
                   : float4(0.0, 0.0, 1.0, 1.0); // Blue

    OutputTex[id.xy] = color;
}

[Zorro666]

Thank you for this. It all looks ok. I would like to check the DXIL metadata you are seeing (I thought enum types were already supported by the DXIL debugger, but looking at your changes something is missing).

Would you be able to upload a RenderDoc capture file using your example HLSL so I can look at the generated DXIL and if possible please can you include the shader compilation flags and which version of DXC you are using, to help me recreate this in a local automated test.

[Zorro666]

I would be interested in your capture file which recreates the hang or error.

In my quick local test using an enum didn't cause a crash, it produced an error message about unhandled DICompositeType but the debug simulation worked correctly. The source variable was shown as a struct instead of as an enum, it was mapped to the correct SSA register and had the correct value e.g.

[qwmnerbvqwmn]

Thank you for this. It all looks ok. I would like to check the DXIL metadata you are seeing (I thought enum types were already supported by the DXIL debugger, but looking at your changes something is missing).

Would you be able to upload a RenderDoc capture file using your example HLSL so I can look at the generated DXIL and if possible please can you include the shader compilation flags and which version of DXC you are using, to help me recreate this in a local automated test.

This link(https://drive.google.com/file/d/12XYiIW-N2RjH6wmydqmQOzi5bpLIEkQc/view) is my capture file, captured using renderdoc v1.42. The dxc version is dxcompiler.dll: 1.8 - 1.8.2407.7 (416fab6b5); dxil.dll: 1.8(101.8.2407.12). Upon opening the capture file, searching for the ComputeScreenGlobalIllumineRadianceOut compute pass, clicking debug within this pass results in this error. At the same time, it will keep the renderdoc UI in a "Please Wait" window forever. Unless you kill the renderdoc process using the task manager, you cannot end renderdoc.

I apologize for giving an example earlier that was based on my own speculation but turned out to be useless. The actual trigger requires an enum to be used as a member of a struct, or as a local variable, and with the -Zi option in the dxc compilation options, using sm6.0+ or higher. This is a tested hlsl sample code that can trigger the bug.

// dxc compile option :  -E TestCSMain -T cs_6_6 -Zi -O3


// Define an enum
enum MyState
{
    STATE_IDLE = 0,
    STATE_BUSY = 1,
    STATE_DONE = 2
};

struct Data
{
    float4 value;
    // Condition A: Use enum as a type in struct
    MyState state; 
};

RWTexture2D<float3> Output : register(u0);

[numthreads(1, 1, 1)]
void TestCSMain(uint3 dtid : SV_DispatchThreadID)
{
    // Condition B: Use enum type as a local variable
    MyState currentState = (MyState)(dtid.x % 3);
    
    Data d;
    d.value = float4(dtid.x, 0, 0, 0);
    d.state = currentState;
    
    // Simple branch logic to ensure variables are not optimized away
    if (d.state == STATE_BUSY)
    {
        d.value.y = 1.0f;
    }
    
    Output[dtid.xy] = d.value.xyz;
}

In the capture file I provided above, the ComputeScreenGlobalIllumineRadianceOut contains an enum named SpecularCalculateScheme as a member of the struct LumenGlobalIllumineCalculateScheme, which triggered this bug.

Meanwhile, there are many operations in the capture file that can trigger other hangs, such as ScreenSpaceProbeMeshSDFTraceHitSampleRadiance and ScreenSpaceProbeMeshSDFTrace. Debugging these passes will trigger other errors. My HLSL code is too complex or contains errors. I will try to fix the errors when I have time later.

[Zorro666]

Thank you for explaining in detail the reproduction of the hang.
The hang (which is an infinite loop) is caused by having an enum inside a struct.
The infinite loop happens due to this code (without the code to handle DW_TAG_enumeration_type)

    case DXIL::DIBase::Type::CompositeType:
    {
      const DICompositeType *compositeType = base->As<DICompositeType>();
-->   typeData.baseType = typeMD; <-- Causes the infinite loop
      switch(compositeType->tag)

In addition to your change I will do a commit to fix the infinite loop for unhandled CompositeType's.