Asyncio Echo server, unwanted messages

[davefes]

Following an example TCP echo client and TCP echo server at:
https://docs.python.org/3/library/asyncio-stream.html
I wrote a Ping client and Echo server. The idea of the Ping is that I
can use watchdogs to reset either or both units if they fail.

This part I can get to work, but I think there might be something that
I am doing which may not be the correct asyncio-way.

Watching the server in rshell:

awaiting client connection
Received 'ping' from ('192.168.8.87', 63126)
Received 'ping'
Send: 'ping'
Received 'ping' from ('192.168.8.87', 63126)
Received 'ping'
Send: 'ping'

(CTRL-C, CTRL-D in rshell seems to provoke the issue)

Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)
Received 'ping' from ('192.168.8.87', 54124)
Received 'ping'
Send: 'ping'
Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)
Received '' from ('192.168.8.87', 63126)

I tried to find if there was a way to flush those ' ' messages from
reader.read(), but no luck there.

I can ignore these messages or do a machine.reset(),
but can't help feeling that my 2nd attempt at Asyncio is sub-standard.

The relevant part on the server:

async def handle_echo(reader, writer):
    while True:
        try:
            data = await reader.read(100)
            message = data.decode()
            addr = writer.get_extra_info('peername')
            print(f"Received {message!r} from {addr!r}")
        except:
            print("Close the connection")
            writer.close()
            await writer.wait_closed()

      # if message is '' do NOT process it, kludge!
        if message is not '':
            print(f"Received {message!r}")

            if message == 'ping':
                if config.watchdog is True:
                    wdt_feed()
                    wdt.feed()

                print(f"Send: {message!r}")

                writer.write(data)
                await writer.drain()

        await asyncio.sleep(1)

        
async def main():
    print('awaiting client connection')

    server = await asyncio.start_server(
        handle_echo, config.server, config.port)

#    async with server:
#        await server.serve_forever()

    async with server:
        while True:
            await asyncio.sleep(300)


# connect to hotspot
connect()

asyncio.run(main())

The "problem" also occurs when the system is working and the
watchdog kicks-in on the server, as the client is running before
server comes-up.

I would like to know why I get these unwanted messages and how to get
rid of them ... properly.

[davefes]

Thought this might be useful:
https://superfastpython.com/asyncio-server/#How_to_Close_an_Asyncio_Server
but I really do not want to shut the server down I would like it to drop the connection
to the alarm device and allow a new connection to be made on the next alarm.

Could be a more complex issue then I can handle:
python/cpython#114432

Edit:
In the above thread the server is made global in one of the examples, which I can do
for the alarm cases, before a machine.reset().

I am fairly happy for now.

[sophiedegran]

dave,
I experiment that the 'ping' to know if the machine is 'there' is not the good solution in micropython..
Why?
Too more energy for the stack!
Simple do an async connexion with StreamWriter and a clause try:
micropython is a little special in sockets!

[davefes]

@sophiedegran The ping client and server are on mains power. One of my many challenges is keeping ESP32s running 24/7, as well as keeping them connected in my local WiFi system.

micropython is a little special in sockets!

The only socket programming I have done is on the ESP32, so I don't mind if Micropython is "special" ... only that I can get things to work in a reasonable efficient manner :)

[glenn20]

StreamReader.read() will return b"" on EOF, ie. if the connection is closed from the remote or the connection is lost.

You could just cleanup and exit the handle_echo() function when you receive an empty read(), eg:

async def handle_echo(reader, writer):
    while True:
        try:
            data = await reader.read(100)
            if not data:
                writer.close()
                await writer.closed()
                return
            message = data.decode()
            ...

I believe that closing the writer should be the correct way to cleanup and ensure the connection is fully closed, but it may be unnecessary.

It looks from your second output printout above that you have two tasks running - one that keeps trying to read a closed connection and another one that is printing the 'ping's from a new connection.

[davefes]

It looks from your second output printout above that you have two tasks running - one that keeps trying to read a closed connection and another one that is printing the 'ping's from a new connection.

Correct. The pinging happens every 5 seconds or so and I am listening to two different alarm sources. The "magic" I haven't understood is that the program appears to be smart enough to listen to multiple connections and send messages back to the appropriate sender.

Also, I have a problem with message it either needs to be made a ' ' or do a full reset after an alarm message. However, it is working, thanks again for your suggestion.

[glenn20]

The "magic" is:

start_server() starts an asyncio task that waits for incoming connections on the port you specified.

When a new connection is made by a client:

• a new socket pair connection to the client is negotiated and
• another new task is created to run the callback (handle_echo() in your case) on the new socket pair
• the server task goes straight back to accepting new connections
  • and will create a new task for any more incoming connections

So, you get a new task for every new connection, which can run in parallel.

Also, I have a problem with message it either needs to be made a ' ' or do a full reset after an alarm message.

Hmm... I dont know what that means. You "shouldnt" need to perform a hard reset if there is a problem with the connection.

[davefes]

I am guessing the unique peername is what keeps things sorted.

My variable message doesn't get "flushed", say if it was you got an alarm. It needs to be cleared, or I keep getting the alarm. I am sure it is my coding error.

If I make it xyz or some non-relevant message then I don't need to reset the ESP32.

[glenn20]

Hey Dave,

I'm not sure exactly what will cause this behaviour, but it is almost what I would expect if the read() was raising an exception:

        try:
            data = await reader.read(100)
            message = data.decode()
            addr = writer.get_extra_info('peername')
            print(f"Received {message!r} from {addr!r}")
        except:
            print("Close the connection")
            writer.close()
            await writer.wait_closed()

If an exception of any kind occurs, you will close the connection, but you don't exit the task by returning from the function. So you will continue the execution with data left with the value it got from the last suuccessful iteration...and this will continue on each iteration. However, I would also expect it to also print out "Close the connection" again and again. For all you know, you may be getting exceptions raised by data.decode() due to some invalid unicode data in the incoming message (I dont believe that is the case, just illustrating a point).

Generally, it is considered bad practice to use catch-all exceptions, especially if you don't print out the error message. I would change the code above to:

async def handle_echo(reader, writer):
    addr = writer.get_extra_info('peername')
    while True:
        try:
            data = await reader.read(100)
            message = data.decode()
            print(f"Received {message!r} from {addr!r}")
        except Exception as err:
            print(err)
            print("Closing the connection...")
            writer.close()
            await writer.wait_closed()
            print("Closed.")
            break  # or return if you prefer

Then, if an exception does happen, you know what it is and have a chance to address it (or just accept it).

Also, you don't need to call writer.get_extra_info('peername') on each iteration and it may well raise an exception itself if called after closing the writer (but I have not checked).

[davefes]

I will get a chance to try your suggested improvements in a few days. That was a poor except example, I have been more explicit in previous attempts :)

Thank you for taking the time to make helpful suggestions ... I have lots to learn in the asyncio world.

[glenn20]

Years of unix network socket programming in C also helps with a lot of the concepts (though, many years ago now :).

[GM-Script-Writer-62850]

This is how i deal with those:

async def webUI(reader, writer):
	request_line = await reader.readline()
	print("\nClient connected:",request_line)
	if(request_line == b""):
		writer.write('HTTP/1.0 204 No Content')# is this right?
		await writer.drain()
		await writer.wait_closed()
		return print("Disconnect")

you can remove the print lines or put the connection message after that

i do check header data for the content length and use data=await reader.readexactly(length) if i need to read post data

[glenn20]

And a last bit of gratuitous, unsolicited advice (that I hope you won't be offended by Dave ;).

Here is a little cleanup of the code that, I think makes it clearer....

async def handle_echo(reader, writer):
    addr = writer.get_extra_info('peername')
    try:
        while (data := await reader.read(100)):
            message = data.decode()
            print(f'Received {message!r} from {addr!r}')
            if message == 'ping' and config.watchdog is True:
                wdt_feed()
                wdt.feed()
            print(f'Send: {message!r}')
            writer.write(data)
            await writer.drain()

    except Exception as err:
        print(err)
        try:
            with open('errors.txt', 'a') as outfile:
                outfile.write(err + '\n')
        except OSError:
            pass

    print('closing the connection...')
    writer.close()
    await writer.wait_closed()
    print('closed')

I believe it is easier to see at a glance more easily what this code is doing. Because the read() is in the whiile statement you can see at a glance it is reading till there is no more data (though I know some people dont like the walrus operator, but this is just the kind of usage it is recommended for).

The "happy path" is clear and easy to read inside the while loop, while all the error handling is outside, rather than interspersed among the normal execution path.

The handling of the closing and finalising is at the end where it will be executed if an exception occurs or not (and is not duplicated for the end of input or exception cases.

Note - untested code, but it "looks"right ;).

[davefes]

Not at all, having someone suggest better ways to do things reduces the anxiety that something could fall-over down the track.

If is wasn't for other people's good examples I would NOT have been able to do the projects that I have attempted over the last 20 years. The RF-stuff I learnt the hard way.

walrus operator ... now there's a new one.

Cheers

https://realpython.com/python-walrus-operator/

I will have to read this several times :)

[davefes]

Does the walrus example handle the not data and/or b'' cases automatically?

[glenn20]

Yes. These are assignment expressions (assignments which can be treated as expressions). In C, all assignments can also be used as expressions, eg: a = b = c = 0. That was not allowed in python till the special := (walrus) operator was introduced - and only in specific circumstances.

The return value of await reader.read(100) will be assigned to data. Then the value of data will be used as the value that is tested by the while statement.

When tested as a boolean, all non-empty str or bytes values will evaluate as True and empty str or bytes values evaluate as False. So, it is common to write if data: print(f"Data = {data}") rather than if data != b"": print(f"Data = {data}") but they are (almost) functionally equivalent and both are quite acceptable.

Without assignment expressions, the code would have to be:

        while True:
            data = await reader.read(100)
            if not data:
                break
            message = data.decode()

instead of:

        while (data := await reader.read(100)):
              message = data.decode()

If you prefer, you could also write:

        while (data := await reader.read(100)) != b"":
              message = data.decode()

[davefes]

After my first read of the link, I could tell there was more going on then I could comprehend. Thanks for unpacking it, I am starting to see that it reads in a more concise way.