Is an IRQ "queued" when interrupts are disabled?

[ma261065]

I have read the article https://docs.micropython.org/en/latest/reference/isr_rules.html but have one question that wasn't clear from there.

If I have a critical section in my main code that disables interrupts, updates a variable that is shared with the ISR, and then re-enables interrupts....

What happens if the interrupt fires while interrupts are disabled? Will it be missed? Or will the ISR be called after the interrupts are re-enabled?

[shariltumin]

I was playing with this code, pin interrupt without debouncing routine.

import machine, time
from machine import Pin

def pin_cb(p):
    state = machine.disable_irq()
    print("hello", p, time.ticks_us())
    # time.sleep(1) # crash! Interrupt wdt timeout on CPU0
    # time.sleep_us(200) # test with none, 100, 200
    machine.enable_irq(state)

touch = Pin(0, Pin.IN, Pin.PULL_UP)
touch.irq(trigger=Pin.IRQ_FALLING, handler=pin_cb)

Running on MicroPython v1.20.0-283 ESP32-S3 at 24MHz. The test is to touch GPIO-00 to GND and record the output.

1. No sleep

<IRQ>
>>> hello Pin(0) 995969670
hello Pin(0) 995970011
hello Pin(0) 995970259
hello Pin(0) 995970506
hello Pin(0) 995970753
hello Pin(0) 995971002
hello Pin(0) 995971245
...

2. time.sleep_us(100)

<IRQ>
>>> hello Pin(0) 895127834
hello Pin(0) 895128130
hello Pin(0) 895128309
hello Pin(0) 895128470
hello Pin(0) 895128650
hello Pin(0) 895128809
hello Pin(0) 895141088
...

3. time.sleep_us(200)

<IRQ>
>>> hello Pin(0) 995969670
hello Pin(0) 995970011
hello Pin(0) 995970259
hello Pin(0) 995970506
hello Pin(0) 995970753
hello Pin(0) 995971002
hello Pin(0) 995971245
...

From these results, we can conclude that the irq is not being captured and is not queued up during the period of machine.disable_irq().

So we want the priode between state = machine.disable_irq() and machine.enable_irq(state) to be as short as possible.

[robert-hh]

I cannot see how you came to your conclusion, and you did not tell how you created the IRQ events.
The Pin.irq handler of ESP32 runs scheduled Python code in a Python context. It is not technically a hard IRQ handler. In a hard IRQ handler interrupts at the same or lower level would anyhow be disabled. The scheduled handlers are queued. If another event happens before the first one is handled, it will be handled later. Whether IRQ handler or not, IRQs must be disabled as short as possible.

From previous test I know, that the IRQ latency for Pin.irq() at an otherwise idling ESP32 is 100µs - 400µs. Other reports much longer latencies, up to several 100 ms.

[shariltumin]

My setup is low-tech and simple. Using a female-male Dupont wire, I connect the female end to the GPIO-00 pin. As I mentioned earlier, the test was done by touching the male end to GND.

Above is a simple test script. Anyone who is interested can experiment and test the results for themselves.

I do not know how the interrupts were queued in between the irq disable and enable. I will take your word for it if you say they were queued.

For all practical purposes, these are what I know.

1. sleeping too long (to simulate heavy processing) will result in a system crash with "Interrupt wdt timeout on CPU0" even though MicroPython is running on CPU1!

2. If time.sleep_us(t), for example t==400, then tick_us() at interrupt n+1 minus tick_us() at interrupt n is greater than 400.

3. If there is no time.sleep_us(t) in the callback pin_cb(), the minimum tick_us() difference I got is 92us.

Touching GND will naturally cause the voltage level at GPIO-00 to fluctuate. The result shows this. This is how the interrupts were generated.

[robert-hh]

What happens is that touching the wires bounces and creates a burst of Pin.IRQ events. In the time between disable_irq() and enable_irq() these are not processed, but in the time between, while the code simply loops and schedules the irq handler the next time. The time differences you see in your list are mostly related to the IRQ latency.

[robert-hh]

I re-tested the latency. Since my last test a few years ago it has improved substantially. Using a simple test loop it is now ~36µs.

[shariltumin]

It is interesting to note that machine.disable_irq() and machine.enable_irq() actually implement a critical region.

mpconfigport.h:

...
#define MICROPY_BEGIN_ATOMIC_SECTION() portSET_INTERRUPT_MASK_FROM_ISR()
#define MICROPY_END_ATOMIC_SECTION(state) portCLEAR_INTERRUPT_MASK_FROM_ISR(state)
...

modmachine.c:

...
STATIC mp_obj_t machine_disable_irq(void) {
    uint32_t state = MICROPY_BEGIN_ATOMIC_SECTION();
    return mp_obj_new_int(state);
}
MP_DEFINE_CONST_FUN_OBJ_0(machine_disable_irq_obj, machine_disable_irq);

STATIC mp_obj_t machine_enable_irq(mp_obj_t state_in) {
    uint32_t state = mp_obj_get_int(state_in);
    MICROPY_END_ATOMIC_SECTION(state);
    return mp_const_none;
}
MP_DEFINE_CONST_FUN_OBJ_1(machine_enable_irq_obj, machine_enable_irq);
...

Now it all makes sense.

[rkompass]

@shariltumin @robert-hh

As a direct test the following setup came to mind:
Elicit another interrupt directly while being in the ISR of a first interrupt inside the code area where the irqs are disabled.

Like:

from machine import Pin

import micropython
micropython.alloc_emergency_exception_buf(160)

pin1 = Pin(14, Pin.OUT, value=0)
pin2 = Pin(14, Pin.OUT, value=0)

def isr1(pin):
    state = machine.disable_irq()
    pin2(1)
    pin2(0)
    machine.enable_irq(state)
    print('Pin1 risen')


def isr2(pin):
    print('Pin2 risen')

pin1.irq(isr1, trigger=Pin.IRQ_RISING, hard=True)
pin2.irq(isr2, trigger=Pin.IRQ_RISING, hard=True)

which works on the pyboard:
If I do a pin1(1) then it reports 'Pin1 risen' and directly afterwards 'Pin2 risen'.

I know that on the Pico there are pin has risen flags and pin has fallen flags for every pin that are like an event storage.
So the queuing of the interrupts may rather be due to the hardware. An interrupt routine certainly deletes those flags, which I assume happens in MP automatically before jumping into the python-ISR???

Now I repeated on the Pico W and observed, that only 'Pin2 risen' was reported.

Seems like the first interrupt was taken over? It seems to be a bug.

[robert-hh]

For comparison with the ESP32 results, you must not use hard=True.

[shariltumin]

pin1 = Pin(14, Pin.OUT, value=0)
pin2 = Pin(14, Pin.OUT, value=0)

Is this what you really want, i.e. pin1==pin2==Pin(14)

I did the same test on the ESP32-S2 but used pin1 = Pin(1) and pin2 = Pin(2), the result is

>>> pin1.irq(trigger=Pin.IRQ_FALLING, handler=isr1)
<IRQ>
>>> pin2.irq(trigger=Pin.IRQ_FALLING, handler=isr2)
<IRQ>
>>> pin1(1)
>>> pin1(0)
Pin1 risen
>>> Pin2 risen

as expected

[rkompass]

Thanks, you are right, that was the culprit. Now I get both reports 'Pin1 risen' and directly afterwards 'Pin2 risen' , as expected.
Same with hard=False.

[shariltumin]

Nice! Good example by the way.

[jimmo]

@ma261065

As discussed above, the ESP32 is a little bit complicated because it only uses "soft" interrupts.

So there are three parts here.

1. The actual interrupt source (e.g. a Pin edge). What happens varies between architectures, but typically a flag is set to indicate a pending IRQ.
2. The CPU interrupts the executing code to run the ISR (i.e. a C function). This is what's enabled/disabled by machine.disable_irq(). The C function schedules the Python handler.
3. The scheduler that calls the Python handler at some later point.

In some sense the exact same thing is implemented at two levels -- a low-level thing sets a flag (interrupt source, or scheduler), which the execution loop (the CPU, or the VM) will notice.

What happens if the interrupt fires while interrupts are disabled?

If interrupts are disabled, then the C handler will not be invoked, i.e. (2) above. The hardware may still set the pending bit for this IRQ, which means that the ISR can still be invoked when interrupts are enabled again.

Will it be missed? Or will the ISR be called after the interrupts are re-enabled?

It depends.

For example, multiple different interrupt sources might share the same pending bit. Or the interrupt source may fire multiple times.

So if you were relying on this to count the number of rising edges, then yes, you will miss interrupts. Or if you were waiting for events from multiple pins, and they happen to share the same IRQ line, then you might only see one.

But in the simplest possible case, yes, you will still see a single interrupt.

It's easy to verify this by triggering e.g. a pin edge while interrupts are disabled. You will see the handler invoked when they are re-enabled.

[shariltumin]

Hi Jim, thanks for this, well explained. Have you ever considered writing an e-book on MicroPython internals?

[andrewleech]

Hi Jim, thanks for this, well explained. Have you ever considered writing an e-book on MicroPython internals?

https://github.com/orgs/micropython/discussions/11330

[ma261065]

"...multiple different interrupt sources might share the same pending bit"

So how can we determine which interrupt sources share a bit?

For example, when a I2S completed interrupt is fired, how can we tell if its the same interrupt bit as, say, the pin-edge interrupt bit?

[glenn20]

In most cases, I believe that will be by reading the data sheets for the platform and its peripherals that you are using.

[jimmo]

Yep.

Practically this isn't very likely to be an issue. Even sources that share the same IRQ vector are going to have their own pending bit. For the most part the point I wanted to make is that you might miss multiple instances of an event, but you won't miss the fact that the event occurred at least once.

[ma261065]

Thanks. I will have to dig a bit deeper into the ESP32 data sheets to understand why I seem to be missing the I2S completed interrupt occasionally if I'm twiddling rotary knobs that generate a bunch of pin interrupts.

[ma261065]

I will have to dig a bit deeper into the ESP32 data sheets to understand why I seem to be missing the I2S completed interrupt occasionally if I'm twiddling rotary knobs that generate a bunch of pin interrupts.

OK, I figured it out (and learned a lot in the process)...

• The "I2S completed" callback is scheduled by a RTOS background task in the I2S code (in machine_i2s.c)
• The relevant line is mp_sched_schedule(descriptor.callback, MP_OBJ_FROM_PTR(self))
• However, there is no check to see if the call to mp_sched_schedule actually succeeds.
• If I spin the rotary encoder knobs on my device fast, they generate a LOT of interrupts.
• As per Jimmo's excellent description above, the interrupts on the hardware pins eventually call the micropython scheduler to run the callback.
• The problem is that all those pin handler callbacks are filling the scheduler queue
• Therefore the I2S Completed callback doesn't get called in this case, and my sound stops playing
• To prove that this is the reason, I added a check the return value from mp_sched_schedule in the I2S code, and it eventually returns false (queue full) if I spin the knobs fast

I don't know if there is any solution to this, as I guess everything is working "as designed", but it's an unfortunate interaction.

[jimmo]

Are you talking about mp_sched_schedule_node?

Yep, sorry should have mentioned that :)

"mp_sched_schedule_node is for running a C function..."

That's right, but your C function can just invoke the Python function using mp_call_function_*.

That shouldn't be an issue in this scenario, right?

Yep, and in this case that might actually be what you want... i.e. you want to de-dupe pending callbacks.

[ma261065]

Got it. I'll give it a go!! Thanks again...

[ma261065]

Oh well, I got stuck.

I couldn't figure out how to get a reference to the python I2S callback function from within the C function that is scheduled by mp_sched_schedule_node, in order to call the python callback with mp_call_function_1().

There doesn't seem to be any way for mp_sched_schedule_node to pass a parameter to the C function, nor does there seem to be a way for the C function to get a reference back to the I2S python object to get the callback function from it.

My code is below:

// FreeRTOS task used for non-blocking mode
STATIC void task_for_non_blocking_mode(void *self_in) {
    machine_i2s_obj_t *self = (machine_i2s_obj_t *)self_in;
    non_blocking_descriptor_t descriptor;

    for (;;) {
        if (xQueueReceive(self->non_blocking_mode_queue, &descriptor, portMAX_DELAY)) {
            if (descriptor.direction == I2S_TX_TRANSFER) {
                copy_appbuf_to_dma(self, &descriptor.appbuf);
            } else { // RX
                fill_appbuf_from_dma(self, &descriptor.appbuf);
            }
            // We have a reference to the callback here (descriptor.callback), but how to get it to the C function?
            mp_sched_schedule_node(&I2S_node, call_I2Scallback);
        }
    }
}

And the C function is:

STATIC void call_I2Scallback(mp_sched_node_t *node)
{
    mp_call_function_1(<HOW TO GET A REFERENCE TO THE PYTHON CALLBACK???>, MP_OBJ_NEW_SMALL_INT(42));
}

[jimmo]

The mp_sched_node_t should be a member of your machine_i2s_obj_t struct. So given the node pointer, you can work backwards to the machine_i2s_obj_t* via offsetof.

#include <stdio.h>
#include <stdint.h>
#include <stddef.h>

typedef struct {
    int y;
} mp_sched_node_t;

typedef struct _machine_i2s_obj_t {
    int x;
    mp_sched_node_t node;
} machine_i2s_obj_t;

int foo(mp_sched_node_t *node) {
    machine_i2s_obj_t *self = (machine_i2s_obj_t*)((uintptr_t)node - offsetof(machine_i2s_obj_t, node));
    return self->x;
}

int main(int argc, char** argv) {
    machine_i2s_obj_t t = { 3, { 7 } };
    printf("%d\n", foo(&t.node));
}

[ma261065]

Ahhh, thanks for teaching me about offsetof()

It all seems to work fine now, and no amount of rotary knob twiddling interrupts the music playback now.

Documenting the solution here for future reference:

1. Changed ~/micropython/ports/esp32/mpconfigport.h to enable the use of the node scheduler:
   #define MICROPY_SCHEDULER_STATIC_NODES (1)

2. In machine_i2s.c, added an mp_sched_node_t member to machine_i2s_obj_t, so it now looks like this:

typedef struct _machine_i2s_obj_t {
    mp_obj_base_t base;
    i2s_port_t port;
    mp_hal_pin_obj_t sck;
    mp_hal_pin_obj_t ws;
    mp_hal_pin_obj_t sd;
    int8_t mode;
    i2s_bits_per_sample_t bits;
    format_t format;
    int32_t rate;
    int32_t ibuf;
    mp_obj_t callback_for_non_blocking;
    io_mode_t io_mode;
    uint8_t transform_buffer[SIZEOF_TRANSFORM_BUFFER_IN_BYTES];
    QueueHandle_t i2s_event_queue;
    QueueHandle_t non_blocking_mode_queue;
    TaskHandle_t non_blocking_mode_task;
    mp_sched_node_t I2S_node;
} machine_i2s_obj_t;

3. Added a new function to machine_i2s.c:

STATIC void call_I2S_callback(mp_sched_node_t *node) {
    machine_i2s_obj_t *self = (machine_i2s_obj_t*)((uintptr_t)node - offsetof(machine_i2s_obj_t, I2S_node));
    mp_call_function_1(self->callback_for_non_blocking, MP_OBJ_FROM_PTR(self));
}

4. In machine_i2s.c, in the FreeRTOS background task, replaced the call to mp_sched_schedule with one to mp_sched_schedule_node:

// FreeRTOS task used for non-blocking mode
STATIC void task_for_non_blocking_mode(void *self_in) {
    machine_i2s_obj_t *self = (machine_i2s_obj_t *)self_in;

    non_blocking_descriptor_t descriptor;

    for (;;) {
        if (xQueueReceive(self->non_blocking_mode_queue, &descriptor, portMAX_DELAY)) {
            if (descriptor.direction == I2S_TX_TRANSFER) {
                copy_appbuf_to_dma(self, &descriptor.appbuf);
            } else { // RX
                fill_appbuf_from_dma(self, &descriptor.appbuf);
            }

            //mp_sched_schedule(descriptor.callback, MP_OBJ_FROM_PTR(self));
            mp_sched_schedule_node(&self->I2S_node, call_I2S_callback);
        }
    }
}