Skip to content

SDK send queue retries a dead serial port forever, and fragments its own error grouping #1427

Description

@garthvh

Two problems in packages/sdk/src/core/queue/Queue.ts, both visible in the
web flasher's telemetry, where this is the single largest source of error
volume by a wide margin — more than every other error in the project combined.

Raw counts are in Datadog (RUM, us5). The query is:

@type:error @service:meshtastic-web-flasher @error.message:*"Error sending packet"*

The loop never ends

In sendPacket:

while (this.queue.filter((p) => !p.sent).length > 0) {
  const item = this.queue.filter((p) => !p.sent)[0];
  if (item) {
    await new Promise((resolve) => setTimeout(resolve, 200));
    try {
      await writer.write(item.data);
      item.sent = true;
    } catch (error) {
      const err = error as { code?: string };
      if (err?.code === "ECONNRESET" || err?.code === "ERR_INVALID_STATE") {
        writer.releaseLock();
        this.lock = false;
        throw error;
      }
      console.error(`Error sending packet ${item.id}`, error);
    }
  }
}

item.sent is only set on success, so a failed write leaves the same item at
the head of the queue and the while condition still true. The loop retakes
the same item, waits 200 ms, fails again, and reports again — indefinitely.

The escape hatch does not fire for the failures that actually happen. It tests
err.code, but the observed errors are a TypeError and DOMExceptions:

  • TypeError: Cannot write to a CLOSED writable stream
  • NetworkError: The device has been lost.
  • NetworkError: Port has been closed
  • UnknownError: An unknown system error has occurred.

None of those carry .codeDOMException exposes .name. So every
real-world "the device is gone" case falls through to the console.error and
loops. At 200 ms per attempt a tab left open reports the same packet tens of
thousands of times, and individual sessions in Datadog do exactly that.

All four of these are terminal: the port is closed or the device is unplugged.
No number of retries fixes them.

The packet ID is in the message string

console.error(\Error sending packet ${item.id}`, error)` interpolates the ID,
so one fault becomes thousands of distinct message strings. Datadog groups by
message, so a single stuck session fragments into thousands of separate
issues rather than one. That is most of why this went unnoticed: nothing
aggregates, and you cannot find it without a wildcard search.

Suggested fix

  • Treat a write failure on a closed or lost stream as terminal. Match on
    error.name for the DOMException cases, or simply give up on any write
    failure after marking the item failed — the current allowlist will keep
    missing cases as browsers change their error types.
  • Cap attempts per item regardless, so no failure mode can loop forever.
  • Move the packet ID out of the message and into a structured field, so the
    fault groups as one issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions