Strange problems, unsure of cause

[doctea]

Hello! First off, thanks for all your hard work on this project!

I've been working on a project on Seeed XIAO RP2040 since February, and it was going pretty well - I was using a version of the framework and packages and libraries from around February 10th-ish, I think.

I started running into some weird crashes a couple of weeks ago, and got a raspberry debug probe to try and figure out what the problem was. I got that debugging working, but couldn't find the cause of the problem due to corrupt stack traces after an isr_hardfault.

Frustrated with this, I tried updating/reinstalling packages and libraries, hoping to maybe find a fix - but now I've got even stranger problems! Hope someone can maybe give me some ideas what's going on here.

In short, I seem to be getting crashes during task initialisation, but its a bit weird:-

• Uploading new firmware sometimes works and the device starts up perfectly and seems to run normally for hours.
• Pressing the 'reset' button on the Xiao resets and runs correctly.
• But it never starts up again after a power cycle - crashing during startup.
• Uploading the same firmware.uf2 file again still doesn't run - same crash during startup.
• Making a change, recompiling and uploading again sometimes seems to work, but again only until a power cycle.
• I've tried uploading via UF2/bootloader and via the cmsis-dap debug probe, with same results.
• I've tried a separate XIAO RP2040, with the same results.
• I think everything (except for the weird crashes that I was getting that lead me to update in the first place) seemed to be working OK before I upgraded/reinstalled packages, but I can't be sure exactly when these problem started.
• I've tried rolling the packages and my installed libraries back to as close to my original settings as I can manage, but it seems not to make any difference at all.
• Because of this seemingly random "same firmware file will run some times but not others", its very hard to properly determine if any changes I'm making are actually solving anything or making it worse.

I'm really stuck as to what could be causing this behaviour and how I could go about diagnosing and solving it. I'm not even sure if the framework is the right place to be looking for problems with this, or whether the platform is more appropriate? Any ideas or help much appreciated!

(In case looking at my project code and platformio config helps, its here: https://github.com/doctea/Microlidian)

[earlephilhower]

Without an MCVE there's not much more than generic things I could suggest.

Possibly there's some power/CPU limitation on the XAIO? We're running at 133MHz by default, which is w/in the Rp2040 spec but maybe isn't in the XAIO's limits due to power or something?

Could it be an uninitted global variable? On a reboot it would hold the same value as it held before reboot, which would probably be good. But on a power cycle it could contain garbage.

You should also be able to restart and debug using SWD. Just b main and run after a hardfault and then single step through the code to see if something pops up.

[doctea]

Thanks for your input - I'm working through even more of my code to attempt to pin it down.

The XIAO does handle 133Mhz fine, and actually I was overclocking this project to 200mhz for months previously without any apparent issues; dropping it back to the standard speed doesn't make any difference, anyway.

Uninitialised variables somewhere might make some sense, I've already fixed a couple of cases of those but without any improvement, I'll continue looking for more.

I've found that if I disable the screen functionality of my program then it will come back up after a cold boot OK, so I guess it seems likely that the problem is there somehow, whether in the code or something else.

Thanks for your help so far... back to debugging now!

[doctea]

Latest finding after laboriously stepping through the code is that its crashing when the TFT_eSPI library is attempting to pgm_read_word the font bitmap from an address of 0x69617661... I don't know how the flash RAM is memory mapped on this architecture but this strikes me as odd.

I still don't understand why this has only started being a problem recently, but I'll open an issue with on the TFT_eSPI repo to see if I can get any further insight there!

[earlephilhower]

0x69617661 looks like ASCII and is not a legal address! It expands to iava

[doctea]

0x69617661 looks like ASCII and is not a legal address! It expands to iava

good catch! i can't find that string in any of my program or library though, so not sure where that would come from..

it appears however that i have got my project working again - code now works between power on/offs and everything working as before. apparently the font wasn't getting set up in the graphics library, so that pointer was going to an uninitialised pointer location to try and read the font data. adding a call to my setup that initialised the font pointer made it all start working again.

i'm still baffled though, as i don't recall having to manually set the font previously to this, i can't find a mention of it when i tried a quick grep through my old commits, and i haven't seen any evidence yet that the call to set up the font once existed in my code, but was then accidentally deleted within the last few days..!

this also doesn't really explain why different builds would sometimes appear to work (fonts and all), even though this line was apparently previously missing?!

i also found along the way that one of my source files suddenly started complaining due to a typo in a define that should have prevented compilation for days.

the best guess i've got at the moment is that perhaps an old object file was hanging around somewhere somehow and causing builds to intermittently work/fail unexpectedly... or perhaps some file system problem had left a mangled package installation somehow (i did run out of disk space when building last week and had to clear space..) or... i don't know... multiple cosmic ray events? (and we also can't rule out the theory that i might just be stupid!)

thanks for your patience and input on these matters. and thanks again for all your work on this system - this frustrating weekend of debug notwithstanding, i've been having a lot of fun with this platform and its certainly been a learning experience :)