Skip to content

fix(client): normalize headers and response handling for local /request#106

Open
Miskler wants to merge 6 commits intodaijro:mainfrom
Miskler:pr1
Open

fix(client): normalize headers and response handling for local /request#106
Miskler wants to merge 6 commits intodaijro:mainfrom
Miskler:pr1

Conversation

@Miskler
Copy link

@Miskler Miskler commented Aug 31, 2025

Problem

The TLS client implementation was brittle in multiple places:

  • Response body: server sometimes returned bytearray(...), HTML error pages, or gzip-encoded data → JSON decoding failed with orjson.JSONDecodeError.

  • Response headers: geventhttpclient exposed headers as generators or bytes, not as a clean dict, causing .items() errors and type mismatches.

  • Request headers: outgoing headers mixed str and bytes or contained non-string values (int), leading to

    • TypeError: can't concat str to bytes in geventhttpclient;
    • BrowserContext.set_extra_http_headers: expected string, got number when rendering with Playwright.

When it happens

  • Whenever the local Go server returns non-JSON payloads (HTML errors, bytearray(...) wrappers, gzip).
  • When headers are not normalized (generator, bytes, or numeric values).
  • When sending requests to /request without a proper Content-Type: application/json.
  • During .render(), if any header value is not a string (e.g. X-City: 229 as an integer).

Solution

  • Added _normalize_headers to coerce any header format (generator, bytes, tuples) into dict[str,str|list[str]].

  • Introduced _SanitizedCID, a CaseInsensitiveDict subclass, that forces all header values to str on insert/update.

  • Outgoing requests to /request now always use geventhttpclient.header.Headers with Content-Type: application/json and Accept: application/json.

  • Response body is read once and normalized:

    • decompress via Content-Encoding: gzip;
    • unwrap bytearray(...) repr if present;
    • explicit error if server returns HTML/non-JSON.
  • Minor fixes: safe handling of timeoutMilliseconds, insecureSkipVerify, and base64 only for true byte bodies.

  • Improved error messages with safe body prefixes for diagnostics.

Result

  • Stable JSON parsing regardless of how the local server formats its response.
  • No more type errors from header handling in geventhttpclient or Playwright.
  • Clearer, actionable error messages when the server responds with invalid payloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant