Strategy for transferring data over UART

[ThinkTransit]

I have an ESP32 connected to a GSM modem. I want to download files from the GSM modem file system which involves the modem pushing a stream of data through the UART. The total amount of data could be as large as 500kB.

The problem is that I think the ESP32 has a hardcoded buffer size of 128 characters espressif/esp-idf#9682

What would be a good way to ensure I can capture the full 500kB without allowing the buffer to overflow?

The best I have come up with so far is just a loop that reads batches of characters however it feels very hacky, is there a better way of doing this?

  big_buffer = bytearray()
  last_data_time = time.ticks_ms()

  while True:
      # Sleep to allow buffer to capture some data
      # This would be an asyncio.sleep() in production
      time.sleep_ms(5)

      avail = self.uart.any()
      print(f"Available characters: { avail }")
      
      if avail > 0:
          last_data_time = time.ticks_ms()
      
      if avail > 100:
          data = self.uart.read(100)
          big_buffer += data
      
      elif avail > 0 and avail < 100:
          data = self.uart.read(avail)
          big_buffer += data
          
      # Return when buffer is empty for more than 500ms
      elif avail == 0 and time.ticks_ms() > (last_data_time + 500):
          return big_buffer

[robert-hh]

The FIFO of the UART hardware is 128 byte. Independent from that you can specify a driver buffer size with the rxbuf parameter of the UART constructor. The default value for rxbuf is 256. It must not be less than 128. Due to the data types in the code, the size is limited to 65535. Which is quite a bit, since this buffer is used to buffer data while you code is busy otherwise.

[ThinkTransit]

Thanks Rob that's interesting, I didn't realise the values were hardware/software and independent of each other.

With a larger buffer I can make my code much less agressive as using115200 baud it only needs 15kb per second of data roughly.

[robert-hh]

Looking at your code, adding data to a buffer is not effective. It is better to allocate the largest expected size once and use memoryview and uart.read_into() to insert data at a gives place in the buffer.

[ThinkTransit]

Thanks @robert-hh, great advice, implemented now with memoryview and readinto.

  big_buffer = bytearray(2048*1024) # 2MB buffer, max file we can receive
  mv = memoryview(big_buffer)

  last_data_time = time.ticks_ms()

  # track the position in buffer
  position = 0
  while True:
      # Sleep to allow buffer to capture some data, 50ms allows about 600 characters
      # This would be an asyncio.sleep() in production
      time.sleep_ms(50)

      # Any data available?
      avail = self.uart.any()
      
      if avail > 0:
          # record last time data received for timeout
          last_data_time = time.ticks_ms()
          
          self.uart.readinto(mv[position:position+avail], avail)
          position += avail

      # Return when buffer is empty for more than 500ms
      elif avail == 0 and time.ticks_ms() > (last_data_time + 500):
          return big_buffer[0:position]

  # Clean up
  gc.collect()