Cooperation on different levels?

[Fisherman48]

Two assyncronous tasks in one script.

1. setting up a webserver providing graphs of logged data from SD card - a few times a day, in a scheduled context

2. mqtt listen for data and log data on SD card - every 10 mins, using Timer interrupt for calling non blocking check

Working ok after one day test.
What can happend - with both tasks running without disabling/ protection ?
Are the better ways?
Br /Stefan

# setting up a web server using:
srv.setsockopt(socket.SOL_SOCKET, 20, cln_handler) # register

# and subscribe to mqtt for logging on SD card using
class cb: # take care of first argument ()
    def chk(self):
        print('check_msg')
        client.check_msg() # non blocking
cb1= cb()
from machine import Timer    
tim1 = Timer(1)
tim1.init(mode=Timer.PERIODIC, period=10000, callback=lambda t:cb1.chk())

[karfas]

Regarding the timer callback, see Writing Interrupt Handlers in the docs.
To do anything useful in the timer callback, you should use micropython.schedule().

For me, the use of the Timer for such a simple function looks overly complicated when a simple loop with a sleep(10) will do it.

Without proper locking, anything can happen to the resources shared by your "tasks" (in your case, most likely the files on the SD card).

Maybe you want to look into asyncio. There you have multiple tasks, queues, locking primitives, ....

[peterhinch]

The official MQTT client has a number of issues discussed in mqtt_as. I agree with @karfas : my recommendation is to use asyncio with mqtt_as.

[shariltumin]

Hey @Fisherman48, since you asked about cooperative tasks based on callback and interrupt, may I suggest the following script.

It may not be the right way to do it. It may not work for your project. But you can give it a try.

You need to add web-server and mqtt-client instead of pin_cb() and t1_cb() respectively. The script tries to simulate the tasks of your project as I understand it.

import os
from machine import SDCard, Timer, Pin

class SD():
   def __init__(my):
       sd = SDCard() # esp32 slot1 sdmmc - default
       os.mount(sd, '/sd')
       print('Info:', sd.info()) # error if not ready
       my.cr = '' # critical region
   def put(my, msg):
       w = 0
       if my.cr == '':
          my.cr = 'w'
          try:
             with open('/sd/data.log', 'a') as fw:
                fw.write(msg)
             w = len(msg)
             touch.irq(trigger=Pin.IRQ_FALLING, handler=pin_cb)
          except Exception as e:
             print('Err:', e)
          my.cr = ''
       else:
          print('@put CR:', my.cr, 'wait...')
       return w
   def get(my):
       r = ''
       if my.cr == '':
          my.cr = 'r'
          try:
             os.stat('/sd/data.log')
          except:
             pass
          else:
             try:
                with open('/sd/data.log', 'r') as fr:
                   r = fr.read()
                os.remove('/sd/data.log') # delete after read
             except Exception as e:
                print('Err:', e)
          my.cr = ''
       else:
          print('@get CR:', my.cr, 'wait...')
       return r

def t1_cb(t):
    put_ref = sd.put
    msg = f'{esp32.hall_sensor()} {esp32.raw_temperature()}\r\n'
    schedule(put_ref, (msg))

def pin_cb(p):
    touch.irq(handler=None)
    print('Data:', sd.get())

# simulate
from micropython import schedule
import esp32
sd = SD()
t1 = Timer(1)

touch = Pin(0, Pin.IN, Pin.PULL_UP) # Touch the pin to the ground to trigger

# IRQs
touch.irq(trigger=Pin.IRQ_FALLING, handler=pin_cb)
t1.init(mode=Timer.PERIODIC, period=5000, callback=lambda t:t1_cb(t))

I hope it is useful.

[karfas]

@shariltumin: This "locking" using a variable will not work, as it is definitely possible that the timer interrupt occurs between the if and the variable assignment, even when this might look working in the first place.

Please read the fine documentation Writing interrupt handlers, especially the chapter Critical sections.

[Fisherman48]

Hi,
Thank you for all comments, ideas and hints, @karfas @peterhinch @shariltumin
I can find solutions to my setup from your examples. There are a lot to take in and digest with new vocabulary and for me, the not intuitive thinking in the asyncio.

To se the big picture :
Are there a way to describe the different ways/tools to solve my problem in micropyhon, with concurrense, protection, locks, sempaphores, starving ... , in diffent levels / context (from machine level - user level like the OSI model) ?

Br /Stefan

[karfas]

Are there a way to describe the different ways/tools to solve my problem in micropyhon ?

There is no silver bullet. Every problem has it's own requirements (and therefore, it's own solution). It's an experience thing.
Sometimes it helps to draw NS-Diagrams, flowcharts and the like.

Just keep it simple and stupid at the beginning. Many applications like your TCP server + data logger don't need real concurrency (you can rewrite this later when you are more familiar with micropython/programming/asyncio/interrupts).

A simple outline for your problem:

# create socket, set to nonblocking
socket = ...
setsockopt(...)

loop_msec =100
cnt = 0
while True:
    sleep_ms(loop_ms)
    if poll(socket):
        # handle TCP connection and request
    if cnt >= 10000 // loop_msec:
        # do something else every 10 seconds
        cnt = 0
    cnt += 1

No Interrupts, no locking required, simple (but might also require around 100msec to serve the TCP request).

[shariltumin]

@karfas, why does critical region protection using a variable not work in my example? The sd.put(msg) was executed by the scheduler placed by the timer callback. This means that all possible sd.put(msg) are queqed in the scheduler and run sequentially.

The button IRQ callback is called directly, but not before the IRQ has been disabled with touch.irq(handler=None). I don't see a problem here.

I tested the script before posting it here. The touch.irq() happened whenever the trigger pin was connected to ground. The timer triggered at a steady rate. There must be contention for the critical region during the test, but I did not experincing problem.

Since we are on esp32, I even try lock = _thread.allocate_lock(). Using lock.locked(), lock.acquire() and lock.release() to protect the critical region.

I had problems with sd.write(). After some time the script hangs at fw.write(msg) without any error message, this is with the default slot1 sdmmc. With slot2, SPI sdcard interface I saw the following errors:

E (2381130) sdmmc_cmd: sdmmc_read_sectors_dma: sdmmc_send_cmd returned 0x107
E (2381140) sdmmc_cmd: sdmmc_write_sectors_dma: sdmmc_send_cmd returned 0x108

I don't know what's causing the problem. Bad sdcard, bad connections, bad programming?

---

@Fisherman48 said that his system worked fine after one day of testing. I am interested in a different way of doing things.

I admire people who try something out of the ordinary. Sure, you might make a big mistake and feel like a fool, but that is how you learn.

I am sure everyone wants a simple and clean solution. But sometimes it is good to be out of your comfort zone. We grow by doing something different. Just my 2 cents.

[karfas]

@shariltumin: It might work for your specific usecase, or it may fail once after hours/weeks/months running. It depends upon the exact timing of the interrupt and what the interrupted function is doing.

However, I can't recommend this kind of "locking" as a generic solution.

I've been there, done something similar (in C) and failed (~20 years ago) miserably. It required a few months debugging/logging on the machine of our customer, as our application used to run a few weeks without problems.

[peterhinch]

Using IRQ's with switches is a disaster because of contact bounce. This stuff is so much easier with asyncio. Th following is the gist of how this could be done:

from mqtt_as import MQTTClient
from mqtt_local import config  # WiFi credentials, broker IP etc
import asyncio
from primitives import ESwitch

async def messages(client):
    async for topic, msg, retained in client.queue:
        print(f'Topic: "{topic.decode()}" Message: "{msg.decode()}" Retained: {retained}')
        # Handle incoming message: your code here

async def switch_handler():
    es = ESwitch(Pin(0, Pin.IN, Pin.PULL_UP))
    while True:
        es.close.clear()
        await es.close.wait()
        # Handle switch closure: your code here

# Deal with WiFi outages
async def up(client):
    while True:
        await client.up.wait()
        client.up.clear()
        await client.subscribe('foo_topic', 1)  # This is where you subscribe to a topic

async def main(client):  # Main program
    try:
        await client.connect()
    except OSError:
        print('Connection failed.')
    else:
        asyncio.create_task(up(client))
        asyncio.create_task(messages(client))
        await switch_handler()

# Define configuration
config['keepalive'] = 120
config["queue_len"] = 1  # Use event interface with default queue
client = MQTTClient(config)  # Set up client.

try:
    asyncio.run(main(client))
finally:
    client.close()
    asyncio.new_event_loop()

The "handle incoming message" and "handle switch closure" sections would embody the application code. These are not truly concurrent so don't need any special precautions unless they compete for the same resource. They can access files or hardware devices.

[Fisherman48]

Thank you Peter.
Using the tutorial and your asyncio example with mqtt_as, results in >3weeks flawless mqttlogging on sd card. The old logger using robustmqtt had ~3 stops during that time!
However, the mqtt publishers of data in my application works much longer ( TBF=6 months or so) using rubust. Less handshaking ?

Still its annoying not using esp8266 with mqtt_as. Looks like a bigger task to learn Freezing the mqtt_as. Is it worth the effort and how much RAM is left on the 8266-12?

In the lua -world of ESP the is a https://nodemcu-build.com/ where you choose modules and get the right size new fw build in minutes on mail. Look more work here in MP.

[peterhinch]

Re reliability mqtt_as should be pretty good. We ran tests lasting months and nearly 1 million messages without issues. The quality of the power supply is crucial: ESPx chips ca take current spikes of nearly 0.5A.

As stated in the doc you can expect about 27K of free RAM on ESP8266.

Building with frozen code is something you only need to do once. Your app and config file need not be frozen, only mqtt_as.py.

[Fisherman48]

Building with frozen code is something you only need to do once. Your app and config file need not be frozen, only mqtt_as.

Yes.

To build a fw the esp8266 with the frozen mqtt_as - is there an easyway or ... ?
From my perspective and computer science education and programming up to mid 1980 's, I saw a process using a macro (a list of actions) to consolidate/link already compiled modules into a runable set of modules to install . But reading about all the steps in unfamilliar environments looks owerwhelming for me at the moment without investing a lot of time for doing it once. Of cource I learn a lot underways, but now I'm more focused in a flawless mqtt.
How to find the right way in for me the complex menu?

[peterhinch]

I can't really improve on the official build instructions.

[Fisherman48]

Ok, and mt pc/win is not up to the requirement for Docker so I leave this for the freezing long winter time here.

By the way, "the official mqtt has number of issues" why not change to the mqtt_as (or a slimmer if possible) as the official ?

Thanks for all help with examples, I'm now using async as a standard approach in my little home iot world.

/Stefan

[peterhinch]

why not change to the mqtt_as (or a slimmer if possible) as the official ?

See my thoughts on future development.

I know the maintainers have discussed this but it is quite a big job.