SerialPIO jitter during Flash write?

[myklemykle]

Hi,

I've got my RP2040 set up to send "analog MIDI" (aka rs-232 @ 32kbaud, 8N1) using SerialPIO. It's working pretty well, but I've been trying to track down why I see an occasional specific jitter glitch, where a MIDI clock packet is seen by the recipient as a MIDI stop packet.

With rs-232 on the wire, the zeroes are high and the bits are reversed, and there's a low start bit and a high stop bit. The MIDI stop packet seen by the recipient (11111100) is almost identical to a MIDI clock packet (11111000), except the signal is not held high (zero) quite long enough.

While trying to isolate the problem, I figured out that I can reliably trigger this glitch by writing a small bit of data to Flash, using LittleFS and file.write(). I'm trying to work out why that could be.

A power issue? Maybe writing Flash takes a lot of current, and that's making the GPIO output unstable? But actually this board has separate buffer chips driving the serial output, with their own 200 ma power supply. The flash chip datasheet says it never draws more than 15ma when writing.

On the other hand, I'm beginning to understand how the RP2040 bus system can have various kind of contention that cause the MCUs to stutter. At first I thought it made sense that a PIO could be hung waiting for a write to Flash to finish ... but actually the PIO would be reading from a FIFO, and this delay is only for a duration of 1 bit@32kbaud, whereas I imagined those FIFOs are written at least 8 bits at a time, probably 32, by a DMA controller. Do they ever contend for some other piece of the bus fabric in the midst of writing a byte? Or does the MCU get involved in the timing with SerialPIO ports, bit-banging style?

I'm a little stumped. I'm just wondering if there are any other known/obvious causes of jitter in the PIOs, or open bugs related to PIO Serial jitter. Of course PIO are designed to not do exactly that. I'm also wondering if I'm correct that data is only moved in/out of the Serial PIO fifos byte-by-byte by DMA?

And in general this might all have a much simpler explanation that I'm not thinking of, so any advice or ideas would be much appreciated. Thanks.

[earlephilhower]

The is no physical way that the clocking on the PIO can vary during transmission of a byte no matter what happens elsewhere in the chip. The PIO pulls an entire byte from the FIFO and then shifts bits out using its own PIO clock which is derived from the main system clock to work at the proper baud.

If the bus was under contention, it would just result in a larger delay between characters, not affect bits within i character.

IIRC you were doing some magic with positive/negative SerialPIO instances to get DC through some filter caps. Could you have an analog issue there, or could you have a timing skew between the 2 SerialPIO instances (i.e. you are not writing to the PIO at the same time and there is no guarantee they will align in time)?

[myklemykle]

OK, that's about what I thought re: clocking.

I'm sure you're right that my magic/satanic double-ended rs232 setup could be prone to a one-bit communication problem, but what puzzles me is that it's both rare (happens randomly once per 100-1000 msgs) and reproducible (happens about 1 in 3 times I write Flash).

The two PIO write() commands are not time-aligned, but they are side by side in the code & the binary, and have mostly seemed to be time-aligned-enough. But you're right, I see how a hanging Flash read could cause just this problem, if it happens while the MCU is somewhere between the start & end of the two SerialPIO.write() calls and trying to read instructions from Flash. But that is a very small section of the code, so I don't get why writing Flash is able to trigger the problem so reliably.

Maybe writing Flash causes the entire XIP cache to be flushed, so then the first call to SerialPO.write() runs slowly out of flash, but the one immediately thereafter runs more quickly from the cache? That's my best theory so far.

I don't actually need to write Flash while MIDI clock is running, that's just how I'm triggering the problem. But some other thing is causing the same contention, kind of randomly. Maybe it's some background USB activity, knocking the serial code out of the cache when it's called? But I thought that code all lived in ROM. I don't think I have any other non-deterministic stuff in my code.

Is there a macro/pragma to tell the compiler to run specific library methods or objects from SRAM? I know how to specify that for my own functions, but it'd be great to be able to tell it to copy SerialPIO.write() and all its subroutines to RAM at boot and run them from there. I have RAM to burn at the moment.

[earlephilhower]

Are you doing multicore code? When you do a flash write the opposite core needs to be frozen in a tight RAM loop (since it can't fetch insns from flash during the flash operation) so that might be what's going on. You can do noInterrupts(); SerialPIO1.write(a); SerialPIO2.write(a); interrupts(); to make those 2 writes as atomic as possible. And the flash erase-write cycle can take many milliseconds so it's a non-trivial delay (will vary w/flash chip, age, etc.).

Moving everything to RAM is not readily possible w/the core now. You can rewrite the lib/memmap_default.ld file to do it, but it's not been done before to my knowledge and it's not something I myself have looked into (and it's not going to be a simple menu option, of course).

You can check the datasheet, but I don't think the XIP flash cache is flushed on a flash operation since self-modifying code on the ARM is frowned upon. :)

As for USB interrupts, those only happen on core0 and the noInterrupts() stuff above will disallow them while you run those 2 steps.

[myklemykle]

You can do noInterrupts(); SerialPIO1.write(a); SerialPIO2.write(a); interrupts(); to make those 2 writes as atomic as possible.

Derp! That is the much simpler explanation I wasn't thinking of.

Problem solved! Thanks again!