Using _thread library

[redhead-p]

I've been experimenting with using the _thread library to run a thread on the second core (core 1). It appears that ISR code associated with IRQ specifications for Pins and Timers appears to run on core 0 (the first core) irrespective of where the IRQ was specified. Similarly a function invoked using micropython.schedule() always appears to run on core 0. I assume that there is a single queue for functions scheduled using micropython.schedule() and that the head of queue function is always executed in core 0.

I know that in theory that micropython.schedule() is intended for use in 'hard' interrupt service routines to allow ISR code to continue in an interruptible context once the time critical elements of the ISR have been dealt with but I was wondering if it was thread safe and therefore using it from core 1 to execute a function in core 0 is permissible.

Thanks

Paul

[jimmo]

hi @redhead-p -- this is a known issue, see #9124

It's on the TODO list. Some thought required as to how this should work...

[jimmo]

Ah yes, sorry I should have given more detail, and yeah this ties into the "some thought requrired" :)

So right now the scheduler should actually run on both cores (i.e. the next available core takes the head of the queue), so I'm a bit perplexed why you're seeing it always run on core 0.

Here is an example that runs a thread on each core and continuously adds stuff to the scheduler:

import micropython
import _thread
import random
import time

def busy():
    while True:
        pass

def task(x):
    print(_thread.get_ident())
    time.sleep_ms(random.randint(500, 1000))
    micropython.schedule(task, None)

task(None)

_thread.start_new_thread(busy, ())
busy()

I see

537095120
537095120
536910352
537095120
536910352
537095120
536910352
537095120

So it seems that both cores are indeed available to run the scheduler.

I send a PR a while back to make some other scheduler changes, which included making it exclusively use core 0, but we ended up removing that before merging. I think this is still the right thing to do (and note that the equivalent feature in CPython only uses the main thread -- https://docs.python.org/3/c-api/init.html#c.Py_AddPendingCall and PR discussion here #8838 (comment)), but somehow we need to deal with the fact that for "soft" interrupts, it does make sense to be attached to a given core.

So I think realistically we're going to need to make the micropython.schedule() API more flexible (e.g. run-on-same, run-on-specific, run-on-any, etc).

[redhead-p]

so I'm a bit perplexed why you're seeing it always run on core 0.

My fault - poorly constructed test! I only put item in the queue. I get the same result as you with your code.

I added the following to your code:

from machine import Timer
.......
tim = Timer(period=500, mode=Timer.PERIODIC, callback=lambda t:print(f't {_thread.get_ident()}'))

It appears that the timer callback gets run on the first available core too.

In terms of my original query, for the behaviour of micropython.schedule() as described here I'd assume it has to be 'thread safe' as the action on one thread may affect code running on the other thread. However you cannot rely on which core the scheduled function will run.

So as far as I'm concerned I'm in a state of 'some thought required' too.

Thanks for your help in clarifying this. I look forward to the results of your further deliberations.

Paul

[peterhinch]

@jimmo If an ISR running on core 0 calls micropython.schedule() and the scheduler happens to select core 1, the code runs into the GIL issues we discussed here. To the normal user of a hard ISR, the purpose of micropython.schedule() is to run GIL-protected code, so the default should be to run on the same core as the main application.
[EDIT]
It seems that micropython.schedule(), when called from a hard ISR, does always run on core 0. I'm glad, but very confused about what the rules actually are...

This shows that it does work correctly when called from a hard ISR context:

import micropython
import _thread
import time
from machine import Pin

pout = Pin(16, Pin.OUT)  # Link these pins
ipin = Pin(17, Pin.IN)

ipin.irq(handler=lambda _: micropython.schedule(soft_isr, None), hard=True)

def soft_isr(_):  # Scheduled by hard ISR
    print(_thread.get_ident())

def busy():
    while True:
        time.sleep(1)
        print('Running')

_thread.start_new_thread(busy, ())
while True:
    pout(not pout())
    time.sleep_ms(200)

[rkompass]

@jimmo @peterhinch I wanted to ask you for a short explanation why you seem to be able to conclude from different thread id's (using get_ident()) to different CPU cores.
I tried the above example codes with print(f'CPU core: {mem32[0xd0000000]}') and observed that in Jimmo's example the CPU core id indeed changes between 0 an 1 (so the above conclusion seems justified although I don't understand why). With Peter's example the CPU core id is always 1.
After extending Peters example with PIO-generated interrupts I found, that when scheduled directly from PIO the ISR runs on CPU core 0, if not scheduled but generated from PIO it runs on core 1.
If the setup of the PIO happens on core 1 and the IRQ setup also is run there then only 2 IRQs are generated.
The example code, my observations are in the comments:

# Investigate on which CPU core (on RP2) a scheduled ISR runs

from micropython import schedule
from machine import Pin, mem32
from time import sleep_ms
import _thread

import rp2

# from official examples at https://github.com/raspberrypi/pico-micropython-examples/blob/master/pio/pio_1hz.py
@rp2.asm_pio(set_init=rp2.PIO.OUT_LOW)
def blink_1hz():
    irq(rel(0))
    set(pins, 1)  # Cycles: 1 + 1 + 6 + 32 * (30 + 1) = 1000
    set(x, 31)                  [5]
    label("delay_high")
    nop()                       [29]
    jmp(x_dec, "delay_high")    
    set(pins, 0)  # Cycles: 1 + 7 + 32 * (30 + 1) = 1000
    set(x, 31)                  [6]
    label("delay_low")
    nop()                       [29]
    jmp(x_dec, "delay_low")

def _isr(_):  # Scheduled by hard ISR
    print(f'CPU core: {mem32[0xd0000000]}')

def busy():
    while True:
        pass

def busy2():
    while True:
        sleep_ms(1000)
        print('Running')

ipin = Pin(8, Pin.IN)  

def _setup_sm_helper(irq=None): 
    sm = rp2.StateMachine(0, blink_1hz, freq=2000, set_base=ipin)   # Create the StateMachine with the blink_1hz program, outputting on ipin
    if irq == 'direct':
        sm.irq(lambda _: print(f'CPU core: {mem32[0xd0000000]}'))                  # --> CPU: 0       all the time
    elif irq == 'scheduled':
        sm.irq(lambda _: schedule(_isr, None))                                # --> CPU: 0       all the time, independent of sleep_ms within busy
    sm.active(1)                                                   # Start the StateMachine
        
def setup_sm(core=0, irq=None):
    if core:
        _thread.start_new_thread(_setup_sm_helper, (irq,))
        sleep_ms(100)
    else:
        _setup_sm_helper(irq)
    
# -------   Try different IRQ handlers to print the CPU id  ----------------

# setup_sm(core=0, irq=None); ipin.irq(trigger=Pin.IRQ_RISING, handler=lambda _: schedule(_isr, None), hard=True)
    # --> 'CPU core: 1'    , always

# setup_sm(core=0, irq='direct')
    # --> 'CPU core: 1'    , almost always, somtimes once 'CPU core: 0' first

# setup_sm(core=0, irq='scheduled')
    # --> 'CPU core: 0'    , always

# setup_sm(core=1, irq=None); ipin.irq(trigger=Pin.IRQ_RISING, handler=lambda _: schedule(_isr, None), hard=True)
    # --> 'CPU core: 1'    , always

# setup_sm(core=1, irq='direct')
    # --> 'CPU core: 1'    , then 'CPU core: 0'    , then nothing

setup_sm(core=1, irq='scheduled')
    # --> 'CPU core: 0'    , then 'CPU core: 0'    , then nothing


_thread.start_new_thread(busy, ())  # make CPU 1 busy
busy()                              # make CPU 0 busy

[redhead-p]

Sorry - this seems more problematic than originally intended.

@peterhinch

It seems that micropython.schedule(), when called from a hard ISR, does always run on core 0. I'm glad, but very confused about what the rules actually are...

I'm not sure about this. I modified your code to print both the core used for the hard element of the ISR and after the schedule.

import micropython
import _thread
import time
from machine import Pin, mem32

#pout = Pin(16, Pin.OUT)  # Link these pins
ipin = Pin(17, Pin.IN) # cheat version to avoid linking pins!

ipin.irq(handler=lambda _: micropython.schedule(soft_isr, mem32[0xd0000000]), hard=True)
ipin.init(mode = Pin.OUT) # now changed to out put - but interrupt still works!

def soft_isr(core):  # Scheduled by hard ISR
    print(core, mem32[0xd0000000],_thread.get_ident())

def busy():
    print('Running')
    for x in range(100):
        time.sleep_ms(200)


_thread.start_new_thread(busy, ())
for x in range(100):
    ipin(not ipin())
    time.sleep_ms(200)

I've also evened out the 'busy' routines. The results I get show that the 'hard' ISR runs on core 0 but post micropython.schedule() the core is indeterminate - sometimes running on core 1.

For information the memory at 0xd0000000 is the memory mapped register that contains the core id. 0 for core 0 and 1 for core 1.

@jimmo

So right now the scheduler should actually run on both cores (i.e. the next available core takes the head of the queue)

I haven't seen anything to contradict this. A lot of what follows is guess work etc so I stand to be corrected. Although you say 'the scheduler', I'd assume that in practice there are two instances, one per core, and it's the first one to check the queue that takes the head of queue.

Conversely the hard element of an ISR is run as set when the IRQ/ISR was setup. IRQs are routed via the NVIC and there is one NVIC per core so the IRQ is serviced directly by an ISR running on the core associated with the NVIC that was used for the set up. This is deterministic and I'd assume that unless specific measures have been taken otherwise this will always be core 0.

Where does this leave me in my attempts to make effective and safe use of core 1? I'm happy to work within the constraints of the _thread library and divide my application functionality between the two cores in a way that makes sense in my design. However I can't do all my processing in a thread safe manner, and only want to use the _thread locks to explicitly synchronise operations between the cores where this is part of the design. This means that I need to have certainty about what code is executing on which core and so I can't use micropython.schedule() with its current opportunistic approach to core allocation.

So I think realistically we're going to need to make the micropython.schedule() API more flexible (e.g. run-on-same, run-on-specific, run-on-any, etc).

From my perspective at the moment I'd be happy focus on the main use of micropython.schedule(). I.e. allowing transfer of control from a hard ISR to an interruptible context. If the scheduled code could always run on the same core as the 'hard' ISR then this meets my requirements and I guess would address Peter's concerns too. I'd assume that a simple way to achieve this would be to have two scheduler queues, one per core. An alternative, if all hard ISRs do run on core 0, might be to stop the core 1 instance from looking at the queue so scheduled code is always run on core 0.

[peterhinch]

@redhead-p @rkompass I can confirm that the scheduled code sometimes runs on core 1. I have raised #10690 - in that code sample the main code and the hard ISR run on core 0 but the scheduled code runs on core 1. In my view this is a bug, for the reason explained in the issue.

On the wider questions I'm looking forward to a response from @jimmo: he is the expert :)

[rkompass]

@redhead-p @jimmo Im my understanding the run-on-same, run-on-specific, run-on-any would require three scheduler queues where each core checks its own and the any-core-queue.?. That having said: Nice that Jimmo has the flexible API in mind, so my remark at the below issue 10690 was not necessary.

[jimmo]

@rkompass @peterhinch @redhead-p

I will try and respond in more detail ASAP, but one thing to note is that looking at the thread id can only prove the hypothesis that the scheduler can run on both cores, but it cannot disprove it. There are many reasons why it might appear that the scheduler always runs on the same core, despite being able to run on both.

In summary, the VM (which runs identically on both cores) will try and take items from the scheduler queue at certain times (i.e. at the pending exception check, executed just after a branch instruction). But there are lots of circumstances where one core essentially gets "starved" because the other thread will ~always get there first, especially if the two cores are not executing exactly the same code.

[redhead-p]

@jimmo @peterhinch @rkompass

I've just being re-running some of the tests described here on 1.20, noting that the release notes say 'There have also been important bug fixes to threading and concurrency when using both cores'. From the tests it appears that while there is a task running on core 1 (the second core), timer callbacks and soft ISRs run on the second core but if there is no task on the second core, then these are run on core 0 (the first core).

As @jimmo writes above, this test doesn't prove anything so it would be useful to know if this is now the expected behaviour following the bug fixes, but I'm not sure how to find the bugs to determine if this is the case. If this is the expected behaviour this is a positive step forward as the indeterminacy that made using _thread in conjunction with soft ISRs and timer callbacks has been removed so thanks in anticipation.

[Gadgetoid]

Just to throw my hat into the ring here, while looking into a (possibly related) hardlock when using Timers, I found that the timer callback would hop from core0 to core1 in a simplified repro when enabling the second core. Eg:

from machine import Timer
import _thread
import time

def Tick(timer):
  print(f"CPUID (Tick): {machine.mem32[0xd0000000]}")

def core1_task():
    while True:
        pass

t = Timer(period=1000,  callback=Tick)

print(f"CPUID (main): {machine.mem32[0xd0000000]}")

for x in range(2):
    time.sleep_ms(1000)

_thread.start_new_thread(core1_task, ())

while True:
  time.sleep_ms(1000)

This will output:

MPY: soft reboot
CPUID (main): 0
CPUID (Tick): 0
CPUID (Tick): 0
CPUID (Tick): 1
CPUID (Tick): 1
CPUID (Tick): 1
CPUID (Tick): 1

Reading above it seems there's no guarantee about which core a Timer callback will actually be serviced, by, it's first come first served, so it stands to reason that core1 - presumably unburdened by as much busywork - would take over servicing the timer. Nonetheless this is weird and unpredictable.

The original issue I was investigating was timers simply hardlocking the Pico when core1 is enabled. The following example will fire up both timers, which will be seemingly happy until core1 is enabled whereupon the system will invariably hardfault-

from machine import Timer
import _thread
import time


def TickA(timer):
  pass


def TickB(timer):
    pass


def core1_task():
    while True:
        pass


t1 = Timer(period=1000,  callback=TickA)
t2 = Timer(period=1000,  callback=TickB)


for w in ("Three", "Two", "One", "Hardlock..."):
    print(w)
    time.sleep_ms(1000)

_thread.start_new_thread(core1_task, ())


while True:
  print("Hello World")
  time.sleep_ms(1000)

I mention this on the off chance that these are somehow related issues, though I'm as of yet unsure why this hardlock occurs.