[Issue] CI Nightly: rocgdb tests failing due to missing core dumps + timeouts

[chiranjeevipattigidi]

Summary:
The CI job fails during the GDB testsuite run. Failures are concentrated in ROCm-specific GDB tests and appear driven by:

• inability to generate core files in the CI environment (Could not generate a core file), and
• test timeouts in interrupt / deep-stack tests.
  The test harness reports multiple LLVM failures (some may be in an “ignored failures” list), and a smaller set of unexpected ROCm failures that cause the overall job to fail.

GCC unexpected failures (1)

• gdb.rocm/corefile.exp
• gdb.rocm/device-interrupt.exp
• gdb.rocm/load-core-remote-system.exp

LLVM failures:

• gdb.rocm/runtime-core.exp: fault=pagefault: Could not generate a core file
• gdb.rocm/runtime-core.exp: fault=abort: Could not generate a core file
• gdb.rocm/runtime-core.exp: fault=assert: Could not generate a core file
• gdb.rocm/device-interrupt.exp: ... interrupt (timeout)
• gdb.rocm/deep-stack.exp: backtrace (timeout)

Full Log:

• https://github.com/ROCm/TheRock/actions/runs/23778448409/job/69342417935

Error Logs:

2026-03-31T12:30:50.0082742Z  ^
2026-03-31T12:30:50.0084105Z /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/fission-loclists.S:291:2: error: changed section flags for .debug_abbrev.dwo, expected: 0x80000000
2026-03-31T12:30:50.0084841Z  .section .debug_abbrev.dwo,"",@progbits
2026-03-31T12:30:50.0085104Z  ^
2026-03-31T12:30:50.0085693Z /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/fission-loclists.S:453:2: error: changed section flags for .debug_loc.dwo, expected: 0x80000000
2026-03-31T12:30:50.0086392Z  .section .debug_loc.dwo,"",@progbits
2026-03-31T12:30:50.0086627Z  ^
2026-03-31T12:30:50.0087216Z /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/fission-loclists.S:536:2: error: changed section flags for .debug_line.dwo, expected: 0x80000000
2026-03-31T12:30:50.0087918Z  .section .debug_line.dwo,"",@progbits
2026-03-31T12:30:50.0088161Z  ^
2026-03-31T12:30:50.0088790Z /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/fission-loclists.S:569:2: error: changed section flags for .debug_str_offsets.dwo, expected: 0x80000000
2026-03-31T12:30:50.0089519Z  .section .debug_str_offsets.dwo,"",@progbits
2026-03-31T12:30:50.0089782Z  ^
2026-03-31T12:30:50.0090362Z /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/fission-loclists.S:570:2: error: changed section flags for .debug_str.dwo, expected: 0x80000030
2026-03-31T12:30:50.0091043Z  .section .debug_str.dwo,"",@progbits
2026-03-31T12:30:50.0091276Z  ^
2026-03-31T12:30:50.0091835Z /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/fission-loclists.S:570:2: error: changed section entsize for .debug_str.dwo, expected: 1
2026-03-31T12:30:50.0092501Z  .section .debug_str.dwo,"",@progbits
2026-03-31T12:30:50.0092732Z  ^

2026-03-31T12:03:46.9136502Z Running /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.rocm/deep-stack.exp ...
2026-03-31T12:03:58.5510083Z [12:03:58Z] Mem: 1208.3/2267.5GB (53%) | CPU: 1% | Jobs: ~6/256 | Disk: 9782GB free
2026-03-31T12:04:00.4498959Z FAIL: gdb.rocm/deep-stack.exp: backtrace (timeout)
2026-03-31T12:04:00.5587094Z Running /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/intbits.exp ...
2026-03-31T12:04:01.4409513Z Running /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/arr-stride.exp ...
2026-03-31T12:04:01.7920919Z Running /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/dw2-epilogue-begin.exp ...
2026-03-31T12:04:02.7074897Z Running /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.dwarf2/fortran-var-string.exp ...
2026-03-31T12:04:04.2557943Z Running /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.rocm/step-schedlock-spurious-waves.exp ...
2026-03-31T12:04:25.2884389Z Running /__w/TheRock/TheRock/build/tests/rocgdb/gdb/testsuite/gdb.rocm/aspace-watchpoint.exp ..

Over Test Status

 [X] GCC: FAIL
 [X] Total Failed Tests: 1
 [X] Unexpected Failures (1):
        gdb.rocm/deep-stack.exp
 [!] Unused Ignored Failures (3):
        gdb.rocm/corefile.exp
        gdb.rocm/device-interrupt.exp
        gdb.rocm/load-core-remote-system.exp
 
 [X] LLVM: FAIL
 [X] Total Failed Tests: 12
 [~] Flaky Tests (passed after retry) (4):
        gdb.rocm/continue-over-kernel-exit.exp
        gdb.rocm/deref-scoped-pointer.exp
        gdb.rocm/generic-address.exp
        gdb.rocm/lane-execution.exp
 [!] Ignored Failures (LLVM) (11):
        gdb.dwarf2/ada-valprint-error.exp
        gdb.dwarf2/dw2-case-insensitive.exp
        gdb.dwarf2/dw2-cp-infcall-ref-static.exp
        gdb.dwarf2/dw2-entry-value.exp
        gdb.dwarf2/dw2-inline-param.exp
        gdb.dwarf2/dw2-param-error.exp
        gdb.dwarf2/dw2-skip-prologue.exp
        gdb.dwarf2/dw2-undefined-ret-addr.exp
        gdb.dwarf2/dw2-unresolved.exp
        gdb.dwarf2/fission-base.exp
        gdb.dwarf2/pr13961.exp
 [X] Unexpected Failures (1):
        gdb.rocm/deep-stack.exp
 [!] Unused Ignored Failures (7):
        gdb.dwarf2/dw2-empty-inline-ranges.exp
        gdb.dwarf2/dw2-entry-pc.exp
        gdb.dwarf2/dw2-unexpected-entry-pc.exp
        gdb.dwarf2/fission-dw-form-strx.exp
        gdb.rocm/corefile.exp
        gdb.rocm/device-interrupt.exp
        gdb.rocm/load-core-remote-system.exp

[lumachad]

Thanks. Yeah, I spotted these. We're still investigating what caused this, as it doesn't seem to be a change in rocgdb.

[ronlieb]

@pradeepk31 could you check if we ran this test as part of our ww11 promo ?

[pradeepk31]

Hi @ronlieb
We didn't see this failure in our staging cycle.
MI300 passing log - https://compute-artifactory.amd.com/artifactory/rocm-qa-test-logs/test-logs/TheRock/1142/Ubuntu-AMDInstinctMI300X_8-rocm_gdb/2026-03-29_17:36/rocm_gdb_1546315_2.log
MI350 passing log - https://compute-artifactory.amd.com/artifactory/rocm-qa-test-logs/test-logs/TheRock/1142/Ubuntu-AMDRadeonGraphics_8-rocm_gdb/2026-03-29_18:12/rocm_gdb_1489958_2.log
cc - @lajagapp

[lumachad]

I tracked it down to the compiler bump here: #4081

[ronlieb]

@pradeepk31 Luis asks in a side chat:
On your team's testing of rocgdb, could you please verify you're actually using your compiler to test things?

I'm afraid it's just using gcc, which would be unfortunate and wouldn't really test anything.

[chiranjeevipattigidi]

This is a blocker for rocm-systems bump Bump rocm-systems from 96e30b4 to d52f94d

[ronlieb]

would it be feasible to xfail the 1 specific test?
scott is making progress on the root cause and then fix @slinder1

[ronlieb]

@pradeepk31 or @chiranjeevipattigidi could you try adding the flag below and see if it resovles issue ? as a workaround

-flto-partions=1