Is it possible to read a array value while it is being appended (race condition)

[GM-Script-Writer-62850]

If I have core 0 checking for a value in a array and core 1 appending entry value into the array is it possible to read partial data?

import _thread
from time import time

to_do=[]
def find_work():
   import random# If I import both time and random in the 1st three lines random is not defined???
   while True:# busy work for the sake of testing
      if random.randint(0, 100) == 69:
         to_do.append(["Found Something:",random.randint(0, 9),random.randint(0, 9),random.randint(0, 9)])

_thread.start_new_thread(find_work,())

while True:
   # in production there would be other stuff being done in this loop
   if to_do:
      x=to_do[0]
      to_do.pop(0)
      if x[1]==7 and x[2]==7 and x[3]==7:
         print("Winner @",time())

So far this test script has not crashed...

[2dof]

I think it is safer implement ring Buffer (or deque class) . Your code don't have any protection against overflow. also non-stop append/pop is not too efficient (memory allocation) for this type of task)

[robert-hh]

I faintly recall that task switches happen only between python statements. That would explain why the list in your code does not get corrupted.

[GM-Script-Writer-62850]

Even though each task is running on a different core?

[robert-hh]

Maybe not. It would break the benefit of two cores.

[2dof]

the problem is not that you run task on different cores but fact, that tasks do change on your data container "without knowing themselves" - later or sooner some concurrency / race condition memory problems etc will occur.

You should use some-safe data container implementation, at least implementation of queue by peterhinch using asyncio can be solution for You ( task scheduling, overflow checking ) : https://github.com/peterhinch/micropython-async/blob/master/v3/primitives/queue.py

If Your data format and size is always fixed ( numeric - 3 value vector ) instead of list (in peterhinch solution) a ring buffer based on ndarray can be used as main data container ( no dynamic memory allocation), to speed up.
any text messages can be added later .

[GM-Script-Writer-62850]

While test my real code i was unable observe a queue depth of over 0, it should be possible to make 8 in under 1 millisecond, but unless you are a TAS bot that can read the PICO's memory you are not gonna be able to accomplish that, even when trying to spam it i was able to get more than 1 item in the queue, as long as it is not possble to read to_do[0] and get incomplete data: {"name":"Human_readable_n it should not be a issue (i did have a bug like that once and it took a few years to crash; read small file written to ram)

my queue list items are objects {"name":"Human_readable_name", "type":1,"value":1,"time":time.ticks_ms()} each object is the entire job

core 0 is polling inputs and when they are high/low for a around 50 milliseconds it puts a entry into my array
core 1 (running uayncio) is checking for work in the queue for work and if no work is found to does it's own work (counting, reading sensors, run a timer (limit of 1 instance), make a url request)

[bixb922]

What version of Micropython are you using, and on what hardware?

[GM-Script-Writer-62850]

for the test script i was using my PICO W on.... MicroPython v1.19.1 on 2022-10-11;

• updates to MicroPython v1.19.1-796-gf4811b0b4 on 2023-01-13;

when i get my project live it will be on a w5500-evb-pico

[jimmo]

This code is definitely not safe. Concurrent access to mutable data structures (e.g. list) is not allowed across cores.

Note that the data structure in question is a list, not an array.

You need to use _thread.allocate_lock (https://docs.python.org/3.5/library/_thread.html#_thread.allocate_lock) and then protect the concurrent access.

I faintly recall that task switches happen only between python statements. That would explain why the list in your code does not get corrupted.

@robert-hh This is true for multithreading on a single core, protected via the GIL. Like you point out, the GIL would make multi-core multi-threading useless (and this applies to CPython too).

The rp2 port has no GIL so the threads run truly concurrently, but this requires very careful synchronisation.

You should use some-safe data container implementation, at least implementation of queue by peterhinch using asyncio can be solution for You ( task scheduling, overflow checking ) :

@2dof This solution is only suitable for asyncio and will provide no benefit here. Furthermore, asyncio can only use a single core.

the problem is not that you run task on different cores but fact, that tasks do change on your data container

@2dof In this case, it is entirely because they run on different cores. No built-in data structures are safe for concurrent access for operations like append/remove across cores.

[GM-Script-Writer-62850]

Should it be locked during the if to_do: line or is just the 2 following lines good? What would need to happen with the sample code for it to break

[jimmo]

@GM-Script-Writer-62850 The document that Peter linked to below (in conjunction with my feedback at peterhinch/micropython-async#94) should help.

[peterhinch]

I wrote this doc on this topic. It is primarily aimed at linking a uasyncio application with code running on another thread, or on another core. However the mechanisms of data pools and the ThreadsafeQueue class are applicable to any multi thread or multi core system.

@jimmo I would welcome your review of this doc.

[GM-Script-Writer-62850]

thanks for that, i did want to know how to use ThreadSafeQueue

every option just looks to add overhead that does not seem to be needed, i am far more concerned with processing speed than memory usage, and at this time using urequests is by far the easiest thing to trim down for as i should be able to make and keep a socket open and send all data over a single open socket connection and get rid of the overhead

i do not need nor want to batch processing, if there is something in the list it need to be processed otherwise it would feel sluggish
as long as it is not possible to read from the list and get a incomplete value as far as i am concerned that is perfect

if you want you can look at my real code: https://pastebin.com/eeRuFEUg

[jimmo]

every option just looks to add overhead that does not seem to be needed

@GM-Script-Writer-62850 can you clarify what you mean by this? If you want concurrent write access to a datastructure, you need some sort of synchronisation. (Yes, lock free data structures exist, but they are not used in MicroPython).

@jimmo I would welcome your review of this doc.

@peterhinch Thanks, this is very good. I will raise an issue on your repo with some feedback.

[rkompass]

After accepting that you need some overhead to protect your list from concurrent access (no GIL as discussed above) it might be helpful to see that making a thread-safe list is not too hard:

# A thread-safe list in micropython
import _thread
 
class ThreadSafeList():
    def __init__(self, l=None):
        self._l = l if l is not None else []
        self._lock = _thread.allocate_lock()
 
    def __getitem__(self, i):
        with self._lock:
            return self._l[i]

    def __setitem__(self, i, x):
        with self._lock:
            self._l[i] = x

    def __len__(self):
        with self._lock:
            return len(self._l)

    def __repr__(self):
        with self._lock:
            return repr(self._l)
 
    def append(self, value):
        with self._lock:
            self._l.append(value)
 
    def pop(self, i):
        with self._lock:
            return self._l.pop(i)

    # if necessary also define insert, remove, extend, count, .. the same way

# l = [1, 2, 3, 4, 5]
# tl = ThreadSafeList(l)
# print(tl[3])
# print(tl.pop(1))
# tl[1] = 9
# print('len(tl):', len(tl), 'tl:', tl)

This way you might keep your code. Also the overhead should not be too heavy, although I didn't benchmark it.

[GM-Script-Writer-62850]

urequest keep crashing my script as far as i can tell, sever logs say it send a 200 code, urequest said it got null and raises a error and it NEVER wants to happen while i have a USB cable connected for debugging, managed to get to see the error twice but it breaks every day, for it to have been the lack of a lock it would not have made it to the urequest code, at this point i just have a try: except: return# assume server got the message

This is what i have now, well effectively this is just test code

import _thread
from time import time

to_do=[]
def find_work():
   import random# If I import both time and random in the 1st three lines random is not defined???
   while True:# busy work for the sake of testing
      if random.randint(0, 100) == 69:
         lock.acquire()
         to_do.append(["Found Something:",random.randint(0, 9),random.randint(0, 9),random.randint(0, 9)])
         lock.release()
      time.sleep_ms(1)# plus ~100-500 us of work

_thread.start_new_thread(find_work,())
lock=_thread.allocate_lock()

while True:
   # in production there would be other stuff being done in this loop
   if to_do:
      lock.acquire()
      x=to_do[0]
      to_do.pop(0)
      lock.release()
      if x[1]==7 and x[2]==7 and x[3]==7:
         print("Winner @",time())
   time.sleep_ms(1)# plus ~100-500 us of work

[rkompass]

For your use case

    def __bool__(self):
        with self._lock:
            return bool(self._l)

would have to be added to the class, as you test in if to_do:
Now noting that it becomes clear that this testing access to the to_do list is not wrapped into the lock.acquire() .. lock.release() logic.
Except from this I do not notice any error.
You may do a x=to_do.pop(0) as the pop returns the popped value and save an instruction, btw..
As for urequest I don't know enough to help.