Can I run a web server and WEBREPL in parallel? #13161

lbendlin · 2023-12-09T03:29:57Z

lbendlin
Dec 9, 2023

I have tried many ways to run both a web server (on port 80) and webrepl (on port 8266). So far without success. The web server (doesn't matter if synchronous or asynchronous) always works, webrepl sometimes shows its page, sometimes lets me log in, but after that it either doesn't respond, or immediately disconnects. Tried to put webrepl.start before, after, in the middle of the web server code. All to no avail. If the web server is not started then webrepl runs just fine, including file transfers. Are the sockets conflicting? Is something blocking?

from machine import Pin, I2C
import socket
from time import sleep,localtime
import webrepl
import uasyncio as asyncio
#import bme280
i2c=I2C(0,sda=Pin(0), scl=Pin(1), freq=400000)    #initializing the I2C method 
#bme = bme280.BME280(i2c=i2c)          #BME280 object created
led = Pin("LED", Pin.OUT)
led.on()

## tried Microdot too
# from microdot import Microdot

# app = Microdot()

# @app.route('/')
# def index(request):
#     return 'Hello, world!'



# app.run(port=80)

## synchronous
# Open socket
#addr = socket.getaddrinfo('0.0.0.0', 80)[0][-1]

#s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
#s.bind(addr)
#s.listen(1)

#print('listening on', addr)

# Listen for connections
# while True:
#     sleep(1)
#     try:
#         cl, addr = s.accept()
#         print('client connected from', addr)
#         cl_file = cl.makefile('rwb', 0)
#         while True:
#             line = cl_file.readline()
#             if not line or line == b'\r\n':
#                 break
#         led.off()    
#         cl.send('HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n')
#         html = '<!DOCTYPE html><html><body>'+'UTC %d-%02d-%02dT%02d:%02d:%02dZ wd %d doy %d' % localtime() +'</body></html>'
#         cl.send(html)
#         cl.close()
#         sleep(0.5)
#         led.on()

#     except OSError as e:
#         cl.close()
#         print('connection closed')

## asynchronous
async def serve_client(reader, writer):
    led.off()
    print("Client connected")
    request_line = await reader.readline()
    print("Request:", request_line)
    # We are not interested in HTTP request headers, skip them
    while await reader.readline() != b"\r\n":
        pass
    # request = str(request_line)
    writer.write('HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n')
    html = '<!DOCTYPE html><html><head><link rel="icon" href="data:,"></head><body>'+'UTC %d-%02d-%02dT%02d:%02d:%02dZ wd %d doy %d' % localtime() +'</body></html>'
    writer.write(html)
    await writer.drain()
    await writer.wait_closed()
    print("Client disconnected")
    led.on()

async def main():
    print('Setting up webserver...')
    asyncio.create_task(asyncio.start_server(serve_client, "0.0.0.0", 80))
    while True:
        await asyncio.sleep(5)

try:
    webrepl.start( )
    asyncio.run(main())
finally:
    
    asyncio.new_event_loop()

sophiedegran · 2023-12-09T08:26:02Z

sophiedegran
Dec 9, 2023

I have microdot running in async mode and a socket client-server also in async mode with no problems.. I think what your describe is that one of your process is in sync mode .. (problably WEBREPL) . I saw somewhere that max port open by default is 5 but that's not your problem here...

0 replies

shariltumin · 2023-12-09T22:03:20Z

shariltumin
Dec 9, 2023

MicroPython's webrepl is implemented with a special socket option. For example,

 srv.setsockopt(soc.SOL_SOCKET, 20, cln_handler)

This socket option 20 is implemented in modsocket.c, line 503

            #if MICROPY_PY_SOCKET_EVENTS
        // level: SOL_SOCKET
        // special "register callback" option
        case 20: {
            if (args[3] == mp_const_none) {
                if (self->events_callback != MP_OBJ_NULL) {
                    socket_events_remove(self);
                    self->events_callback = MP_OBJ_NULL;
                }
            } else {
                if (self->events_callback == MP_OBJ_NULL) {
                    socket_events_add(self);
                }
                self->events_callback = args[3];
            }
            break;
        }
            #endif

this is to register a socket "readable" event callback on a socket.

In "modsocket.c" line 119, the socket_events_handler() is defined.

// Polls all registered sockets for readability and calls their callback if they are readable
void socket_events_handler(void) {
...
    // Call the callbacks
    for (socket_obj_t *s = socket_events_head; s != NULL; s = s->events_next) {
        if (FD_ISSET(s->fd, &rfds)) {
            mp_call_function_1_protected(s->events_callback, s);
        }
    }
...

In mpconfigport.h we have these related definitions:

#define MICROPY_PY_WEBREPL                  (1)
#define MICROPY_PY_SOCKET_EVENTS            (MICROPY_PY_WEBREPL)
...
#if MICROPY_PY_SOCKET_EVENTS
#define MICROPY_PY_SOCKET_EVENTS_HANDLER extern void socket_events_handler(void); socket_events_handler();
#else
#define MICROPY_PY_SOCKET_EVENTS_HANDLER
#endif
...

The socket_events_handler() is called by the mp_hal_delay_ms() function in mphalport.c line 162

...
void mp_hal_delay_ms(uint32_t ms) {
    uint64_t us = (uint64_t)ms * 1000ULL;
    uint64_t dt;
    uint64_t t0 = esp_timer_get_time();
    for (;;) {
        mp_handle_pending(true);
        MICROPY_PY_SOCKET_EVENTS_HANDLER
        MP_THREAD_GIL_EXIT();
        uint64_t t1 = esp_timer_get_time();
        dt = t1 - t0;
...

So in order for webrepl to work properly, the mp_hal_delay_ms() must be run on a regular basis.

Now, in which modules of the MicroPython system does the mp_hal_delay_ms() function run?

$ pwd
micropython/extmod

$ egrep mp_hal_delay_ms *
modlwip.c:    mp_hal_delay_ms(1);
modlwip.c:        mp_hal_delay_ms(50);
modlwip.c:                mp_hal_delay_ms(100);
modlwip.c:                    mp_hal_delay_ms(100);

modtime.c:    mp_hal_delay_ms((mp_uint_t)(1000 * mp_obj_get_float(seconds_o)));
modtime.c:    mp_hal_delay_ms(1000 * mp_obj_get_int(seconds_o));
modtime.c:        mp_hal_delay_ms(ms);

modwebrepl.c:        mp_hal_delay_ms(MICROPY_PY_WEBREPL_DELAY);

network_lwip.c:            mp_hal_delay_ms(100);

network_ninaw10.c:    for (; !(flags & rwf); mp_hal_delay_ms(5)) {

network_wiznet5k.c:        mp_hal_delay_ms(1);
network_wiznet5k.c:                mp_hal_delay_ms(1); // datasheet says 2us
network_wiznet5k.c:                mp_hal_delay_ms(160); // datasheet says 150ms
network_wiznet5k.c:                mp_hal_delay_ms(250);

There is at least one module that we can use in our script so that mp_hal_delay_ms() can be called programmatically, namely the time module.

"modtime.c" line 119

...
STATIC mp_obj_t time_sleep(mp_obj_t seconds_o) {
    #ifdef MICROPY_PY_TIME_CUSTOM_SLEEP
    mp_time_sleep(seconds_o);
    #else
    #if MICROPY_PY_BUILTINS_FLOAT
    mp_hal_delay_ms((mp_uint_t)(1000 * mp_obj_get_float(seconds_o)));
    #else
    mp_hal_delay_ms(1000 * mp_obj_get_int(seconds_o));
    #endif
    #endif
    return mp_const_none;
}
MP_DEFINE_CONST_FUN_OBJ_1(mp_time_sleep_obj, time_sleep);

STATIC mp_obj_t time_sleep_ms(mp_obj_t arg) {
    mp_int_t ms = mp_obj_get_int(arg);
    if (ms >= 0) {
        mp_hal_delay_ms(ms);
    }
    return mp_const_none;
}
MP_DEFINE_CONST_FUN_OBJ_1(mp_time_sleep_ms_obj, time_sleep_ms);
...

So calling time.sleep(p_sec) or time.sleep_ms(p_msec) will execute mp_hal_delay_ms(ms).

When mp_hal_delay_ms() is running, all socket events callbacks are executed by calling mp_call_function_1_protected(s->events_callback, s).

A few things to note:

socket functions are blocking by default, example:
1. cl, addr = s.accept() is blocking
2. cl_file.readline()
3. cl.send(html)
await asyncio.sleep(sec) uses ticks_add() to schedule the waiting task.

Therefore, mixing a socket event based task like webrepl with standard blocking socket functions or an async/await program will prevent socket event callbacks from running properly.

You can program useful network applications based on socket events, such as webrepl, telnet, ftpserver.

Below is a simple program that allows a server to send logs to multiple logging clients. The s.accept() is called in a callback so that multiple clients can connect to the server.

import network
import socket as soc
import time

network.WLAN(network.AP_IF).active(False)
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.config(pm=0) # disable power management
wlan.connect("YOUR_SSID", "YOUR_PASSWD")
time.sleep(5)
print(wlan.ifconfig())
ip = wlan.ifconfig()[0]

clients = []

def _con_handler(srv):
    global clients
    cs,ca = srv.accept()
    print('Serving:', ca)
    clients.append(cs)

def LOG():
    global ip
    if ip != '0.0.0.0':
       # start listening for telnet connections on port 2323
       print("Starting,", ip, "listening on port 2323")
       srv = soc.socket(soc.AF_INET, soc.SOCK_STREAM)
       srv.setsockopt(soc.SOL_SOCKET, soc.SO_REUSEADDR, 1)

       port = 2323 # default telnet port
       ai = soc.getaddrinfo(ip, port)
       addr = ai[0][4]

       srv.bind(addr)
       srv.listen(5) # cache 5 clients connection request
       srv.setblocking(False)
       # we are using socket event (read) callback!
       srv.setsockopt(soc.SOL_SOCKET, 20, _con_handler)
       return True
    else:
       print('No wifi, please try again')
       return False

if LOG():
   while True:
      if len(clients) > 0:
         print('#clients:', len(clients))
         w = f'The time now is {time.time()}'
         # w = input('==> ')
         for cs in clients:
             try:
                cs.write(w); cs.write('\r\n')
             except Exception as e:
                print('Socket write error:', e)
                print('Remove', cs, 'from list of clients')
                clients.remove(cs)
         print('Sleep 5 sec.')
         time.sleep(5)
      else:
         print('No clients yet. Sleep 5 sec.')
         time.sleep(5)

You can then use a telnet client, for example, to test it:

$ telnet 192.168.4.85 2323

Sorry for the lengthy explanation, but I hope this helps to clarify a bit why your setup is not working as expected, and at the same time we have learned a bit about this mysterious Option 20.

3 replies

sophiedegran Dec 9, 2023

Yes! Thank you sharil!.. I think the more simple to update a microdot web site is to simply use the upload example and adapt it ...

andrewleech Dec 9, 2023
Collaborator Sponsor

If you'd like to keep the async microdot usage, you could add a separate little async task to poll/handle the other socket tasks, eg

async def handle_events():
    while True:
        time.sleep(0)
        await asyncio sleep_ms(100)

asyncio.create_task(handle_events())

This will run the background events checked by time.sleep every 100ms

andrewleech Dec 9, 2023
Collaborator Sponsor

That being said, it would be nice to see an async version of webrepl, it would possibly be fairly easy to add async socket support to that too!

lbendlin · 2023-12-10T00:03:38Z

lbendlin
Dec 10, 2023
Author

If you'd like to keep the async microdot usage, you could add a separate little async task to poll/handle the other socket tasks, eg
async def handle_events():
    while True:
        time.sleep(0)
        await asyncio sleep_ms(100)

asyncio.create_task(handle_events())
This will run the background events checked by time.sleep every 100ms

Tried that too, but the behavior largely stays the same. webrepl shows me the password dialog and then hangs. Subsequent connection attempts are rejected,

`
MPY: soft reboot
mac d8:3a:dd:9a:42:32
ip 192.168.3.22
UTC 2023-12-09T23:56:01.000Z wd 5 doy 343
WebREPL server started on http://192.168.3.22:8266/
Started webrepl in normal mode
Setting up webserver...

WebREPL connection from: ('192.168.3.56', 60865)
Client connected
Client connected
Request: b'GET / HTTP/1.1\r\n'
Client disconnected

Concurrent WebREPL connection from ('192.168.3.56', 60870) rejected

`

Need to read up more on what @shariltumin posted, and see if I can modify the example to use web servers instead of telnet servers.
Would it make sense to put the web server onto the second core and let webrepl run in its natural habitat?

8 replies

lbendlin Dec 10, 2023
Author

No, they should work independently (unless I could press webrepl into also serving my pages?)

I changed according to your guidance but it's still hanging after the password.


async def main():
    print('Setting up webserver...')
    websrv = asyncio.create_task(asyncio.start_server(serve_client, "0.0.0.0", 80))
    repl = asyncio.create_task(aiorepl.task())
    
    await asyncio.gather(websrv,repl)

#webrepl_thread = _thread.start_new_thread(webrepl.start, ())
webrepl.start()
#try:
asyncio.run(main())

Output:
MPY: soft reboot
mac d8:3a:dd:9a:42:32
ip 192.168.3.22
UTC 2023-12-10T14:50:53.000Z wd 6 doy 344
WebREPL server started on http://192.168.3.22:8266/
Started webrepl in normal mode
Setting up webserver...
Starting asyncio REPL...
--> Client connected
Request: b'GET / HTTP/1.1\r\n'
Client disconnected
Client connected
Client connected

WebREPL connection from: ('192.168.3.56', 62996)

Carglglz Dec 10, 2023

This works with an ESP32, so I think it may be related to this #9297?

Not sure why you need WebREPL for but in the meantime an alternative could be using an async websocket server i.e. https://github.com/miguelgrinberg/microdot/blob/7a329d98a8836e302109d21900288815be9fd3cf/examples/websocket/echo_async.py and adapt aiorepl.execute to work with that, e.g.

...
@app.route('/repl')
@with_websocket
async def repl(request, ws):
    g = __import__("__main__").__dict__
    s = asyncio.StreamReader(sys.stdin)
    while True:
        cmd = await ws.receive()
        result = await aiorepl.execute(f"{cmd}\r", g, s)
        if result is not None:
            await ws.send(f"{repr(result)}\r\n")
        else:
            await ws.send("\r")
...

I've just tested this and it's limited but it kind of works...

lbendlin Dec 10, 2023
Author

My idea was to have a setup that I could modify remotely via WiFi). So my web server is happily running and serving pages, but maybe I want to implement a new "page" ,for example raw json like:

{"host":"d8:3a:dd:9a:xx:yy","temperature_c":12.9,"temperature":55.2,"pressure":1000.6,"humidity":84.4,"utc":"2023-12-10T20:48:57.000Z","wd":6,"doy":344}

or have the Pico push data instead of waiting for request.
Ideally I could modify the main.py over the air and then restart the Pico remotely too. Plus, all the other goodies like i2c.scan() etc.

I'll try your latest proposal - thank you!

Carglglz Dec 10, 2023

Ideally I could modify the main.py over the air

Yes, I think you can modify this example and make it work for that, just add the appropriate routes...
https://github.com/miguelgrinberg/microdot/blob/7a329d98a8836e302109d21900288815be9fd3cf/examples/uploads/uploads_async.py

then restart the Pico remotely too. Plus, all the other goodies like i2c.scan()

This should work with the /repl example above, the only thing missing is print, so just use str(), f-strings or repr() if you want to print something 👍🏼

sophiedegran Dec 10, 2023

And why not send dict or list.. via sockets or MQTT that can update your microdot web server page .. I thought your want to 'change' your web server files !

Can I run a web server and WEBREPL in parallel? #13161

Uh oh!

Replies: 3 comments · 11 replies

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewleech Dec 9, 2023 Collaborator Sponsor

Uh oh!

andrewleech Dec 9, 2023 Collaborator Sponsor

Uh oh!

lbendlin Dec 10, 2023 Author

Uh oh!

Uh oh!

lbendlin Dec 10, 2023 Author

Uh oh!

Uh oh!

Uh oh!

lbendlin Dec 10, 2023 Author

Uh oh!

Uh oh!

Replies: 3 comments 11 replies

andrewleech Dec 9, 2023
Collaborator Sponsor

andrewleech Dec 9, 2023
Collaborator Sponsor

lbendlin
Dec 10, 2023
Author

lbendlin Dec 10, 2023
Author

lbendlin Dec 10, 2023
Author