asyncio await writer.drain() hangs forever

[GarikFirst]

I got simple web server that returns on GET request one page (about 100 lines). I’m using ESP32 as access point in this case.

Here is the code snippet of it:

async def web_server() -> None:
    async def client_handler(
        reader: asyncio.StreamReader, writer: asyncio.StreamWriter
    ):

        request = str(await reader.read(2048))
        print(request)

        print("Headers")
        writer.write(
            bytes("HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n", "utf-8")
        )

        if "GET" in request:
            print("Response to buffer")
            writer.write(bytes(get_index_page(), "utf-8"))
            print("Response to buffer - finished")

        if "POST" in request:
            # pylint: disable=use-maxsplit-arg
            print((request).split("\\r\\n\\r\\n")[-1])

        print("Start of the drain")
        await writer.drain()
        print("Finish of the drain")

        await writer.wait_closed()
        print("Socket closed")

    await asyncio.start_server(client_handler, "0.0.0.0", 80)
    print("Web server started")

    while True:
        await asyncio.sleep(1)

Problem is it’s often just hang on writer.drain(), like this:

Web server started
Successful:
b'GET / HTTP/1.1\r\nHost: 192.168.0.1\r\nUser-Agent: curl/8.6.0\r\nAccept: */*\r\n\r\n'
Headers
Response to buffer
Response to buffer - finished
Start of the drain
Finish of the drain
Socket closed

HANG:
b'GET / HTTP/1.1\r\nHost: 192.168.0.1\r\nUser-Agent: curl/8.6.0\r\nAccept: */*\r\n\r\n'
Headers
Response to buffer
Response to buffer - finished
Start of the drain

What am I doing wrong?

[GarikFirst]

So I figured out it’s happens only on large amount of data (around 15 Kb), small string passes smoothly. Any suggestions?

[GarikFirst]

Actually, it’s not the size - it’s content of the string. If I pass 15Kb of in memory generated random symbols - they pass smoothly.

What actually helped is to split the string by newline and then pass to socket string by string. But I have no idea why it working like that.

[andrewleech]

Is it possibly related to utf-8 / international characters in the data?

What version micropython are you using?

I believe there were a few issues worked on recently that related to string length calculations being thrown out by utf-8 multi-byte characters.

[GarikFirst]

I have tried all version from v1.20.0 to v1.23.0 - same result on all of them

[andrewleech]

The changes I'm thinking of were posibly more recent that that, so it might behave differently on a nightly (master) build.
Otherwise narrowing down the data that's causing it will likely be needed to see where the lock up is occuring.

[davefes]

I wonder if I am seeing something similar:
https://github.com/orgs/micropython/discussions/15612

Maybe a try/except block might show the error.

[GarikFirst]

Full web server code with comments:

async def web_server() -> None:
    async def client_handler(
        reader: asyncio.StreamReader, writer: asyncio.StreamWriter
    ):
        print("Web server. Socket opened")
        try:
            request = str(await asyncio.wait_for(reader.read(2048), 3))
            print("Web server. New request received")

            writer.write("HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n")
            await asyncio.wait_for(writer.drain(), 10)
            print("Web server. Headers sent")

            if "GET" in request:
                # Not working like that
                writer.write(get_index_page().encode("utf-8"))
                await asyncio.sleep(1)
                print("Web server. Sending")
                await asyncio.wait_for(writer.drain(), 10)
                print("Web server. Send")
                await asyncio.sleep(5)

                # Working like this
                # for line in get_index_page().split("\n"):
                #     writer.write(line + "\n")
                #     print("Web server. Sending chunk")
                #     await asyncio.wait_for(writer.drain(), 10)
                #     print("Web server. Chunk sent")

            if "POST" in request:
                # pylint: disable=use-maxsplit-arg
                await update_config(
                    json.loads((request.split("\\r\\n\\r\\n")[-1])[:-1])
                )
        except asyncio.TimeoutError:
            print("Web server. ERROR. Timeout")
        finally:
            print("Web server. Closing socket")
            await writer.wait_closed()
            print("Web server. Socket closed")

And output that I get (I have tried timeout up to 30 minutes - no results):

Web server. ERROR. Timeout
Web server. Closing socket
Web server. Socket closed
Web server. Socket opened
Web server. New request received
Web server. Headers sent
Web server. Sending
Web server. ERROR. Timeout

Data Im actually sending below:

<!DOCTYPE html>
<html lang="ru">

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>MQTT Poller Setup Page</title>
</head>

<body>
  <h1>MQTT Poller настройка устройства id 08d1f9e6aa68</h1>
  <form>
    <fieldset id="sensorsSet">
      <legend><b>sensors</b></legend>
      <div id="sensors">
        <div id="sensor_hqditl2c3znp" class="sensor">
          <input type="button" class="btn" value="-" style="width: 25px;" onclick="sensor_hqditl2c3znp.remove()">
          <label class="flbltxt sensorlbl">pin
            <input name="pin" type="number" min=1 value="18" required style="width: 50px;">
          </label>
          <label class="flbltxt sensorlbl">poll_period
            <input name="poll_period" type="number" min=1 value="1" required style="width: 80px;">
          </label>
          <label class="flbltxt sensorlbl">type
            <select name="sensor_type" onchange="changeFieldsVisibility(sensor_hqditl2c3znp)" required>
              <option disabled value> -- Выберите тип датчика -- </option>
              <option selected value="onoff">onoff</option>
              <option value="analog">analog</option>
            </select>
            </label>
            <label class="flbltxt sensorlbl">inverted
              <input name="inverted" class="typeSpecific onoff" type="checkbox" checked>
            </label>
            <label class="flbltxt sensorlbl" hidden>low_threshold
              <input name="low_threshold" class="typeSpecific analog" type="number" value="0" min=0 max=65535 hidden style="width: 80px;">
            </label>
        </div>
        <div id="sensor_42q645638smv" class="sensor">
          <input type="button" class="btn" value="-" style="width: 25px;" onclick="sensor_42q645638smv.remove()">
          <label class="flbltxt sensorlbl">pin
            <input name="pin" type="number" min=1 value="33" required style="width: 50px;">
          </label>
          <label class="flbltxt sensorlbl">poll_period
            <input name="poll_period" type="number" min=1 value="1" required style="width: 80px;">
          </label>
          <label class="flbltxt sensorlbl">type
            <select name="sensor_type" onchange="changeFieldsVisibility(sensor_42q645638smv)" required>
              <option disabled value> -- Выберите тип датчика -- </option>
              <option value="onoff">onoff</option>
              <option selected value="analog">analog</option>
            </select>
            </label>
            <label class="flbltxt sensorlbl" hidden>inverted
              <input name="inverted" class="typeSpecific onoff" type="checkbox" hidden>
            </label>
            <label class="flbltxt sensorlbl">low_threshold
              <input name="low_threshold" class="typeSpecific analog" type="number" value="700" min=0 max=65535 style="width: 80px;">
            </label>
        </div>
        <div id="sensor_q9c4dhmzcfim" class="sensor">
          <input type="button" class="btn" value="-" style="width: 25px;" onclick="sensor_q9c4dhmzcfim.remove()">
          <label class="flbltxt sensorlbl">pin
            <input name="pin" type="number" min=1 value="34" required style="width: 50px;">
          </label>
          <label class="flbltxt sensorlbl">poll_period
            <input name="poll_period" type="number" min=1 value="1" required style="width: 80px;">
          </label>
          <label class="flbltxt sensorlbl">type
            <select name="sensor_type" onchange="changeFieldsVisibility(sensor_q9c4dhmzcfim)" required>
              <option disabled value> -- Выберите тип датчика -- </option>
              <option selected value="onoff">onoff</option>
              <option value="analog">analog</option>
            </select>
            </label>
            <label class="flbltxt sensorlbl">inverted
              <input name="inverted" class="typeSpecific onoff" type="checkbox">
            </label>
            <label class="flbltxt sensorlbl" hidden>low_threshold
              <input name="low_threshold" class="typeSpecific analog" type="number" value="0" min=0 max=65535 hidden style="width: 80px;">
            </label>
        </div>
      </div>
      <input type="button" class="btn" value="Добавить датчик" onclick="addSensor()">
    </fieldset>
    <fieldset>
      <legend><b>wifi</b></legend>
      <label class="flbl flbltxt" for="ssid">ssid</label>
      <input id="ssid" type="text" name="ssid" value="Sladkij Tigr Top Tier"><br>
      <label class="flbl flbltxt" for="password">password</label>
      <input id="password" type="text" name="password" value="SoSnooley"><br>
      <label class="flbl flbltxt" for="timeout">timeout</label>
      <input id="timeout" type="number" name="timeout" value="120"><br>
    </fieldset>
    <fieldset>
      <legend><b>mqtt</b></legend>
      <label class="flbl flbltxt" for="server">server</label>
      <input id="server" type="text" name="server" value="87.249.16.106"><br>
      <label class="flbl flbltxt" for="ssl">ssl</label>
      <input id="ssl" type="checkbox" name="ssl" value="False"><br>
      <label class="flbl flbltxt" for="ssl_params">ssl_params</label>
      <input id="ssl_params" type="text" name="ssl_params" value="{}"><br>
      <label class="flbl flbltxt" for="username">username</label>
      <input id="username" type="text" name="username" value="UMP_PRIME_PROGRESS_TEST"><br>
      <label class="flbl flbltxt" for="port">port</label>
      <input id="port" type="number" name="port" value="1884"><br>
      <label class="flbl flbltxt" for="password">password</label>
      <input id="password" type="text" name="password" value="SXFVC45ydfgsdfg3"><br>
      <label class="flbl flbltxt" for="command_topic">command_topic</label>
      <input id="command_topic" type="text" name="command_topic" value="command"><br>
      <label class="flbl flbltxt" for="incoming_queue_size">incoming_queue_size</label>
      <input id="incoming_queue_size" type="number" name="incoming_queue_size" value="10"><br>
    </fieldset>
    <fieldset>
      <legend><b>system</b></legend>
      <label class="flbl flbltxt" for="watchdog_timer">watchdog_timer</label>
      <input id="watchdog_timer" type="number" name="watchdog_timer" value="0"><br>
      <label class="flbl flbltxt" for="ping_send_period">ping_send_period</label>
      <input id="ping_send_period" type="number" name="ping_send_period" value="180"><br>
      <label class="flbl flbltxt" for="state_led_pin">state_led_pin</label>
      <input id="state_led_pin" type="number" name="state_led_pin" value="25"><br>
      <label class="flbl flbltxt" for="web_config_mode_pin">web_config_mode_pin</label>
      <input id="web_config_mode_pin" type="number" name="web_config_mode_pin" value="22"><br>
    </fieldset>
    <fieldset>
      <legend><b>ntp</b></legend>
      <label class="flbl flbltxt" for="ntp_server">ntp_server</label>
      <input id="ntp_server" type="text" name="ntp_server" value="pool.ntp.org"><br>
      <label class="flbl flbltxt" for="ntp_sync_period_hours">ntp_sync_period_hours</label>
      <input id="ntp_sync_period_hours" type="number" name="ntp_sync_period_hours" value="1"><br>
    </fieldset>
    <input type="submit" class="btn" value="Применить изменения" style="margin-top: 5px;">
    <input type="reset" class="btn" value="Сбросить форму">
  </form>
</body>

<style>
  body {
    font-family: system-ui;
  }

  .flbl {
    display: inline-block;
    width: 200px;
  }

  .flbltxt {
    color: darkslategray;
  }

  .sensorlbl {
    margin-left: 8px;
  }


  fieldset {
    border-radius: 5px;
    border-color: lightgray;
    margin-bottom: 10px;
  }


  input {
    height: 28px;
    margin-bottom: 2px;
  }

  input:not([type="button"]) {
    width: 230px;
  }

  .btn {
    border-radius: 5px;
    border-width: 1px;
    background-color: ghostwhite;
  }

  select {
    height: 36px;
  }

  input[type=checkbox] {
    width: 32px;
    height: 32px;
    vertical-align: middle;
  }

  .sensor {
    margin-bottom: 10px;
  }
</style>

<script>
  const form = document.querySelector("form")

  form.addEventListener("submit", event => {
    event.preventDefault();

    function convertCheckboxCheckedToValue(field) {
      switch (field.matches("[type='checkbox']")) {
        case true:
          return field.checked;
          break;
        default:
          return field.value;
          break;
      }
    }

    const configData = {}
    configData.sensors = []

    const fieldGroups = document.querySelectorAll("fieldset:not([id='sensorsSet'])")
    for (const fieldGroup of fieldGroups) {
      const fields = fieldGroup.querySelectorAll("input:not([type=button])")
      const fieldGroupName = fieldGroup.querySelector("legend").innerText;
      configData[fieldGroupName] = {}
      for (const field of fields) {
        configData[fieldGroupName][field.name] = convertCheckboxCheckedToValue(field)
      }
    }

    const sensorsData = document.getElementById("sensors").children
    for (const sensorData of sensorsData) {
      const sensorFields = sensorData.querySelectorAll("input:not([type=button]):not([hidden]), select")
      const sensor = {}
      for (const field of sensorFields) {
        switch (field.name) {
          case "sensor_type":
            field.name = "type"
            break;
          case "pin":
            field.name = "number"
            break;
          default:
            break;
        }
        sensor[field.name] = convertCheckboxCheckedToValue(field);
      }
      configData.sensors.push(sensor)
    }

    const sensorsPins = []
    for (const sensor of configData.sensors) {
      sensorsPins.push(sensor.number)
    }
    if (new Set(sensorsPins).size !== sensorsPins.length) {
      window.alert("Номера pin должны быть уникальны!");
    } else {
      if (confirm("Изменения будут применены и устройство будет перезагружено")) {
        const payload = JSON.stringify(configData, (key, value) => {
          if (key == "ssl_params") {
            value = JSON.parse(value)
          }
          if (!isNaN(value) && key != "inverted" && key != "ssl") {
            value = Number(value)
          }
          return value
        })

        fetch("http://192.168.0.1", {
          method: "POST",
          mode: "no-cors",
          body: payload,
          headers: {
            "Content-Type": "application/json; charset=UTF-8"
          }
        });
      }

      document.querySelector("body").innerHTML = '<p style="font - family: system - ui; ">Дождитесь окончания перезагрузки устройства</p>'
    }
  });


  function addSensor() {
    const elementId = "sensor_" + Math.random().toString(36).slice(2)
    const sensor = document.createElement("div");
    sensor.setAttribute("id", elementId);
    sensor.setAttribute("class", "sensor")

    sensor.innerHTML =
      `<input type="button" class="btn" style="width: 25px;" value="-" onclick="${elementId}.remove()">
        <label class="flbltxt sensorlbl">pin
          <input name="pin" type="number" min=1 required style="width: 50px;">
        </label>
        <label class="flbltxt sensorlbl">poll_period
          <input name="poll_period" type="number" min=1 required style="width: 80px;">
        </label>
        <label class="flbltxt sensorlbl">type
          <select name="sensor_type" onchange="changeSensorFieldsVisibility(${elementId})" required>
            <option disabled selected value> -- Выберите тип датчика -- </option>
            <option value="onoff">onoff</option>
            <option value="analog">analog</option>
          </select>
        </label>
        <label class="flbltxt sensorlbl" hidden>inverted
          <input name="inverted" class="typeSpecific onoff" type="checkbox" hidden>
        </label>
        <label class="flbltxt sensorlbl" hidden>low_threshold
          <input name="low_threshold" class="typeSpecific analog" type="number" value=0 min=0 max=65535 hidden style="width: 80px;">
        </label>`

    document.getElementById("sensors").appendChild(sensor);
  }


  function changeSensorFieldsVisibility(sensor) {
    const selectedType = sensor.querySelector("select").value
    const elements = sensor.querySelectorAll(".typeSpecific")

    for (const element of elements) {
      switch (element.classList.contains(selectedType)) {
        case true:
          element.removeAttribute("hidden")
          element.parentElement.removeAttribute("hidden")
          if (!element.matches("[type='checkbox']")) {
            element.setAttribute("required", "")
          }
          break;
        default:
          element.setAttribute("hidden", "")
          element.parentElement.setAttribute("hidden", "")
          if (!element.matches("[type='checkbox']")) {
            element.removeAttribute("required")
          }
          break;
      }
    }
  }
</script>

</html>