ESP32 wroom 32E - Size limit sending data using socket ?

[Fisherman48]

Using a nonBlocking server from this forum - for sending data to Google Chart to get ScatterCharts.
Up to approx 5700 bytes is ok - but more data is not working/ no Charts. From old info people use 50k rows of 10 cols of data and I'm only using 100 cols and 10 cols.
I dont know what is going on under the hood here but are there any limits from ESP/ python/ socket / my code ?

 import socket, select, time, api

def doReq(request): # handles all requests 
    import B_userMenu, C_graphPrep
    if request.find('GET / HTTP/1.1')>0:
        t=B_userMenu.getMenu()
        return t
    
    if request.find('GET /favicon.ico')>0: # ett juste svar
        return ''
    
    if request.find('GET /action_page.php')>0:
        h = C_graphPrep.getGraph(request)
        return h

# set up nonBlockingServer in background, 
bresp=bytes('HTTP/1.1 200 OK\n'+'Content-Type: text/html\n'+'Connection: close\n\n', 'utf-8')

def req_handler(cs): # main loop
    try:
       req = cs.read()
       if req:
          print('req:', req)
          rep=doReq(str(req)) # act on request
          rep=rep + "\n\n"
          brep=bytes(rep, 'utf-8')
          cs.write(bresp)
          #while len(brep)>2000: # > ~5700 bytes GoggleChart dont answer!
          #    part=brep[:2000]  # no diff sending smaller group size
          #    cs.write(part)
          #    print(len(part))
          #    brep=brep[2000:]
          cs.write(brep)
          print(len(brep))
       else:
          print('Client close connection')
    except Exception as e:
        print('Err:', e)
    cs.close()

def cln_handler(srv):
    cs,ca = srv.accept()
    print('Serving:', ca)
    cs.setblocking(False)
    cs.setsockopt(socket.SOL_SOCKET, 20, req_handler)

api.doConnectWIFI()

port = 80
addr = socket.getaddrinfo('0.0.0.0', port)[0][-1]
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(addr)
srv.listen(5) # at most 5 clients
srv.setblocking(False)
srv.setsockopt(socket.SOL_SOCKET, 20, cln_handler)
`
BR /Stefan   

[davefes]

On a ESP32 WROOM 32U when sending email to Gmail's SMPT server I had to use file chunking to have success. On the ESP32 SPIRAM variant it didn't seem to be necessary.

 #  send data in 1024 chunks
    try:
        with open('datalog.csv', 'rb') as file:
            for chunk in range(0, file_size, chunk_size):
                this_chunk_size = min(chunk_size, file_size-chunk)
             #  handles the last dangling chunk that would be smaller than the full 1024 bytes
                smtp.write(file.read(this_chunk_size))
                time.sleep_ms(200) #  guarantee consecutive writes
    except OSError as error:
        try:
            with open('errors.txt', 'a') as outfile:
                outfile.write('error = ' + str(error) + '\n')
                outfile.write('did not handle datalog.csv properly' + '\n')
        except OSError:
            pass

Note the guarantee consecutive writes, thanks to glenn20
Maybe, that helps.

[Fisherman48]

After I learned more about request/ response between Chrome and ESP32 and using your time.sleep_ms(200) between chunks - I can't see any size limits for graph data on GoogelChart. And it works with chunkSize 1024 and 4096 byte.

Thank you davefes

BR /Stefan

[davefes]

Ah, file size I see I left that parameter out of my response. I use:
file_size = os.stat('datalog.csv')[6]

[davefes]

O dear ... after working properly for a year or more today I got a :
'did not handle datalog.csv properly'
... changed the sleep period to 500ms. I did find that if I used less than 50-100ms I would still get errors.

Good luck!

[sophiedegran]

Thanks dave I will put that in my memory (somewhere), sure we'll have others questions about that...

[Fisherman48]

Next step: ESP32 WROOM 32U, python v1.22,

Using my socket (see start question) to feed Goole Chart html body - is ok with body size up to ~5k bytes. Clients Windows 10 PC, Android mobile and PADs.
<5k works using 1k, 4k and 8k chunks, > 5k not ok regardles chunk size and delays

CHAT GPT about this:
"... the buffer size on a socket can depend on several factors including the specific TCP/IP stack being used, the available memory on the device, and any custom configurations applied by the developer.

Socket Options: The ESP-IDF provides socket options that allow developers to control aspects of socket behavior, including buffer sizes. For example, the setsockopt() function can be used to set the receive buffer size (SO_RCVBUF) and the send buffer size (SO_SNDBUF) for a socket.
Application Requirements: Developers should consider the specific requirements of their application when choosing buffer sizes for sockets. Factors such as the expected data rate, message size, and latency tolerance can all impact the optimal buffer size. "

Why a limit at 5k - I need >15 k. Any ideas ?

BR /Stefan