Core 1 hanging when Core 0 calls pow() -- how to debug?

[myklemykle]

Hello,

I am seeing a pretty strange issue on my current project: calling the pow() function on core 0 causes core 1 to hang.

By "hang" I mean that loop1 no longer produces any output, and the ISR running on core 1 locks up. Actually it seems that loop1() won't hang if I disable the ISR on core1. The core1 ISR copies a block of samples from a sine wave table into four higher-frequency buffers. It's called very often & is probably keeping core1 very busy, but the only math in it is basic algebra and float->int conversion. I'm trying to imagine what resource pow() uses that the two cores could be conflicting on.

I assume that with arduino-pico the call to pow() would use the RP2040 ROM math routines, but is there a way to confirm that?

Or, is there a way to tell the compiler to use the soft-floating point implementation for just this one call to pow()?

thanks for any advice!
-m-

[earlephilhower]

Can you give a small, 1-screen MCVE that reproduces the problem? pow is actually implemented in Newlib, which will eventually call the ROM routines to replace gcc intrinsics. If you could connect up a SWD debugger and get a stack trace of both cores that might also help debug things.

[myklemykle]

I'll work on that, it may take a little time.

[myklemykle]

FWIW, as I whittle this down, it's looking like the call to pow() on core 0 causes math on core 1 to slow down somehow. As I chop lines of code out of the inner loop of the core 1 ISR, loop1() goes from not responding at all, to intermittently responding, to responding properly. So it looks like the "hang" may be the core 1 ISR getting so slow it doesn't finish before the next interrupt. And yet, if I don't call pow() from core 0, the ISR on core 1 is plenty fast ...

[myklemykle]

Hmm, okay, I mischaracterized this. It now looks more like I'm hitting some threshold of math complexity, or maybe just code size, in the core 0 loop() function, that triggers this problem on core 1.

When I try to whittle down core0 loop anywhere at all, the problem goes away. And then I can add some more code elsewhere to bring it back.

So it's not about pow() specifically; I can cross the same boundary & cause the same problem by adding one line that casts an int to a float and divides by 3.

Anyway, because of that I'm not sure how small i can make the MCVE. The core0 loop() makes a lot of calls to many parts of the codebase.

[earlephilhower]

Are you doing FP or transcendental functions in the ISR? That's probably a no-no.

The ROM code could be using a shared HW resource (i.e. the HW divider) which you'd end up fighting over.

In any case, FP operations will be really slow on this core, anyway, even w/o fighting over locks since there is no FP hardware onboard. Fixed-point is your friend here, but it's obviously not trivial to move to from a pure FP setup.

[myklemykle]

Yeah, I am doing four FP adds and four FP->int conversions in the ISR. But it's been working great until now. I think you're right that this could totally be due to some shared HW resource, but I don't know what resources that would be. Each core has its own HW divider.

But maybe there's a conflict with both cores trying to execute the same FP library routine? That's freaky to consider, it wouldn't be about FP math at all, just about two cores trying to call any particular library or function at the same time. I don't 100% grok how the RP2040 bus fabric works, but if this sort of thing is a real problem then there's potentially a solution in keeping two different copies of the offending library in two different SRAM slices.

But my ISR should probably be fixed point, I have been thinking about converting it if/when necessary. Maybe that's now.