T14 Gen 3 AMD (21CF): UEFI 1.61/R23ET85W appears to cause system to hang on resume under Linux

[aleatoricLilac]

Machine type: 21CFA000CD
OS: Fedora 43
Kernel: 6.17.12-300.fc43, and two or three previous versions in the 6.17 series

Issue:
After installing the 1.61 firmware update, on resume from standby under Linux the system would randomly hang at some point right after wake up was triggered; so far I've seen it manifesting as either a blank screen with the power indicator slow blinking as if still in standby, blank screen with a steady power indicator and no response to any input except FnLock toggling, or the screen actually turning on as well but freezing on the login screen.

This issue was not present in firmware 1.60/R23ET84W and so far appears to be gone after I rolled back the firmware to the previous version as well.

Reproduction:
There doesn't seem to be a way to trigger this reliably, but my experience has been that this usually occurs after about a day post first boot and after a couple sleep-wakeup cycles; it appears that waking up from a long (several hours in my case) sleep is likely to trigger this.

Notes:
The description of the update states that "Fixed issue that 10 minutes after system enters modern standby, the system will may hang up when resume." and it indeed fixed the hang on resume (blinking power indicator and no display) under Windows issue that has plagued the machine for years since purchase, so could it be that Windows and Linux somehow has contradictory requirements for resume?

Something else that might be relevant here is that my machine is the Chinese SKU with the corresponding system board that lacks the discrete TPM chip and also the DRTM ACPI table; no idea whether these are relevant, but these are the most apparent differences that can be observed without inspecting the physical board itself.

I believe that any difference between the system board versions might potentially be related to the issue here as this machine has an extensive repair history (fifth mobo and hopefully not counting :D), and the previous system board installed seemed to be some kind of non-Chinese SKU engineering sample (?) with both of the aforementioned missing items present and fwupdmgr security reporting SPI replay protection disabled somehow. Standby and resume worked perfectly fine with both Windows and Linux with said system board, and problems under Windows immediately came back after it was also replaced for a different failure.

[hughsie]

@mrhpearson this is beyond my pay grade :) Any ideas?

[mrhpearson]

Hi,

5 repairs sounds awful :(
I've created internal ticket LO-4168 so we can get details from the FW team on what that BIOS fix was and to see if there's a Linux impact that was missed.
Would you mind running the AMD sleep test tool (https://git.kernel.org/pub/scm/linux/kernel/git/superm1/amd-debug-tools.git) and see what that reports please? Just in case it gives any clues.

Mark

[aleatoricLilac]

Maybe later; it's getting a bit weird here as I just got the same (sporadic?) issue after moving to the 6.18.3 kernel while on the older 1.60 firmware. I'll see if this keeps happening and if the new kernel works with 1.61 firmware too first - not sure if there were suspend changes between the kernels - and try to nail down the exact working/not working combinations.

Also ultimately this might be (partly) a hardware problem, as I've seen quite a few anecdotes about low power instability with the R7 6800 series APU and personally had a previous motherboard fail in that way; but with how Windows suspend started to work normally and Linux started to fail post firmware update it feels unlikely to be purely coincidental or exclusively a hardware failure.

[aleatoricLilac]

amd-s2idle reports; there doesn't seem to be anything useful unfortunately.

kernel 6.17.12 - fw 1.60.html
kernel 6.17.12 - fw 1.61.html
kernel 6.18.4 - fw 1.60.html
kernel 6.18.4 - fw 1.61.html

Also some things I've found so far:

1. 6.18.x kernels sporadically fail to wake up with the older 1.60 firmware but can remain stable for as long as about 3-4 days with normal usage pattern; not sure if it's something with the 6.18 series or I'm looking at another mobo replacement in the near future.

2. Both 6.17 and 6.18 kernels with 1.61 firmware reliably fail to wake up within a day after about at most 4-5 cycles of varying time spent in standby.

3. In both situations the failure to wake up now consistently presents as steady power indicator LED and a blank screen. No useful kernel log whatsoever (all that gets recorded before a failed wakeup is just "PM: suspend entry (s2idle)"). Also not sure if I'm imagining things, but the coil whine from I assume the power supply bits kinda has a pattern of a louder buzz every 4 seconds or so; wonder if it's trying to bring up something and failing.

4. Waking up from standby under Windows seems to work fine with 1.61; probably didn't test it for long enough but in my experience it fails pretty quickly to a state of slow blink power indicator and no display with pre-1.61 firmware.

I guess for now I'm going to stay under Windows 11 for a bit just to be sure that standby works there, and it isn't actually the motherboard/CPU coincidentally deciding to quit once again around this time.

[aleatoricLilac]

So far suspend and wake up work fine under Windows so 1.61 indeed fixed the bug for Windows and then somehow made it worse for Linux.

Also there doesn't seem to be any reliable way to trigger it; probably the best bet you have here would be getting one with the same China-specific (assuming it is indeed related to a different configuration) system board (part lookup suggests 5B21J16031 for the model) and just simulate normal use of the system & suspend on it until it fails to wake up.

[aleatoricLilac]

So I just tried kernel 6.18.6 and firmware 1.62; sadly the problem persists and the laptop failed to wake up with the exact same symptoms after approximately one day of use and a few normal standby-wakeup cycles.

[mrhpearson]

Thanks for the details - I've passed them on. I don't have any feedback from the FW team yet I'm afraid,