Skip to content

Commit 6ae1c43

Browse files
sntransntran
committed
feat: improve URL parsing and MIME sniffing
Add MIME parsing and sniffing support for response bodies, including automatic blob type detection when headers are absent or fall back to application/octet-stream. Refactor the URL stack into dedicated encoding, IP, parser, punycode, and consolidated IDNA helpers. This aligns Web.URL with WHATWG search/hash semantics, improves special and non-special setters, fixes file URL normalization, and preserves remote-like and rclone targets across Request, Resolver, TCP, and URLPattern flows. Split URL and MIME compliance coverage into focused WPT-backed suites and make mix precommit export coverage data so the full gate reports a stable 100.0% summary. Update the README and changelog to document the new MIME behavior, broader URL support, and the full release gate.
1 parent 56e24ad commit 6ae1c43

30 files changed

Lines changed: 4607 additions & 442 deletions

‎CHANGELOG.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ All notable changes to this project are documented in this file.
88

99
- `Web.File`.
1010
- `Web.FormData` struct and streaming multipart parser.
11+
- `Web.MIME` for MIME parsing, normalization, and signature-based sniffing
12+
used by response body helpers.
1113
- `Web.URLPattern` with WHATWG-style pattern parsing, compilation, matching,
1214
generation, component comparison, and ambient route param injection via
1315
`match_context/3`.
@@ -36,6 +38,19 @@ All notable changes to this project are documented in this file.
3638
### Changed
3739

3840
- `use Web` now aliases `Web.AsyncContext`.
41+
- `Web.URL` now aligns its internal state and serialization around WHATWG
42+
`search` and `hash` semantics, with improved special/non-special setter
43+
behavior, `file:` path normalization, IDNA handling, and rclone URL
44+
serialization.
45+
- `Web.URLPattern` now reuses `Web.URL.Punycode` for host ASCII
46+
normalization and preserves the leading slash for malformed
47+
scheme-looking pathname inputs.
48+
- `Web.Request.new/2`, `Web.Resolver.resolve/1`, and
49+
`Web.Dispatcher.TCP` now handle malformed, path-only, and remote-like
50+
targets more defensively.
51+
- `Web.Response.blob/1` and response content-type resolution now parse
52+
explicit MIME types and sniff bodies when headers are absent or fall back
53+
to generic binary types.
3954
- `Web.Promise.new/1` and `Web.Stream` task callbacks now capture and restore
4055
`Web.AsyncContext` snapshots, preserving logger metadata, scoped variables,
4156
`$callers`, and ambient abort signals across spawned tasks.
@@ -61,6 +76,9 @@ All notable changes to this project are documented in this file.
6176
- `Web.CompressionStream` and `Web.DecompressionStream` now emit context-aware
6277
zlib observability logs from restored task contexts, including ambient
6378
metadata such as `request_id`, `user`, and console grouping depth.
79+
- URL and MIME compliance coverage is now split into focused WPT-backed test
80+
suites, and `mix precommit` now uses exported coverage data so the full
81+
gate reports stable `100.0%` coverage summaries.
6482

6583
## [0.3.0] - 2026-04-10
6684

‎README.md‎

Lines changed: 33 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -433,6 +433,16 @@ if res.ok do
433433
data = await(Response.json(res))
434434
end
435435

436+
# MIME-aware blobs parse explicit types and sniff generic binaries
437+
html =
438+
Response.new(
439+
body: "<!doctype html><html><body>ok</body></html>",
440+
headers: [{"content-type", "application/octet-stream"}]
441+
)
442+
443+
await(Response.blob(html)).type
444+
# => "text/html"
445+
436446
# Multi-protocol support (HTTP/TCP)
437447
req = Request.new("tcp://localhost:8080", method: "SEND", body: "ping")
438448
```
@@ -466,16 +476,29 @@ full-stack example that combines explicit abort signals, logger metadata,
466476
console grouping, multipart file parsing, and `CompressionStream`.
467477

468478
### `Web.URL` & `Web.URLSearchParams`
469-
Pure, immutable URL parsing and ordered query parameter management.
479+
Pure, immutable WHATWG URL parsing with ordered search parameter
480+
management, IDNA host handling, and support for special,
481+
non-special, `file:`, and rclone-style URLs.
470482

471483
```elixir
472-
# URL parsing
484+
# WHATWG-style URL parsing
473485
url = URL.new("https://user:pass@example.com:8080/p/a/t/h?query=string#hash")
474-
url.port # => 8080
475-
476-
# Params management
477-
params = URLSearchParams.new("foo=bar&foo=baz")
478-
URLSearchParams.get_all(params, "foo") # => ["bar", "baz"]
486+
URL.port(url) # => "8080"
487+
URL.search(url) # => "?query=string"
488+
489+
# Ordered search params management
490+
params =
491+
URL.search_params(url)
492+
|> URLSearchParams.set("query", "updated")
493+
|> URLSearchParams.append("page", "1")
494+
495+
url = %{url | search_params: params}
496+
URL.href(url)
497+
# => "https://user:pass@example.com:8080/p/a/t/h?query=updated&page=1#hash"
498+
499+
# Non-standard but common remote targets are preserved
500+
URL.href(URL.new("my_s3:bucket/file.txt"))
501+
# => "my_s3:bucket/file.txt"
479502
```
480503

481504
### `Web.Headers` — Security-First
@@ -547,10 +570,12 @@ view = Uint8Array.new(buffer, 10, 100) # Offset 10, Length 100
547570
---
548571

549572
## 🧪 Industrial-Grade Testing
550-
Reliability is a core requirement. `Web` features exhaustive coverage for stream transitions, body consumption, and redirect handling.
573+
Reliability is a core requirement. `Web` combines focused JSON Test data from
574+
Web Platform Tests with property tests and strict lint and coverage gates.
551575

552576
```bash
553577
mix test --cover
578+
mix precommit
554579
```
555580

556581
---

‎lib/web/body.ex‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,14 @@ defmodule Web.Body do
181181
@doc false
182182
def blob(%{body: body} = struct) do
183183
consume_body(body, fn binary ->
184-
Web.Blob.new([binary], type: content_type(struct))
184+
type =
185+
if Map.get(struct, :__struct__) == Web.Response do
186+
Web.Response.resolved_content_type(struct, binary)
187+
else
188+
content_type(struct)
189+
end
190+
191+
Web.Blob.new([binary], type: type)
185192
end)
186193
end
187194

@@ -382,8 +389,15 @@ defmodule Web.Body do
382389
end
383390
end
384391

385-
defp content_type(%{headers: %Web.Headers{} = headers}) do
386-
Web.Headers.get(headers, "content-type", "")
392+
defp content_type(%{headers: %Web.Headers{} = headers} = struct) do
393+
if Map.get(struct, :__struct__) == Web.Response do
394+
Web.Response.resolved_content_type(struct)
395+
else
396+
headers
397+
|> Web.Headers.get("content-type", "")
398+
|> Web.MIME.parse()
399+
|> Kernel.||("")
400+
end
387401
end
388402

389403
defp content_type(_), do: ""

‎lib/web/dispatcher/tcp.ex‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -138,7 +138,7 @@ defmodule Web.Dispatcher.TCP do
138138
end
139139

140140
defp connection_target(url) do
141-
if Web.URL.rclone?(url) do
141+
if Web.URL.rclone?(url) or remote_like_url?(url) do
142142
authority =
143143
url
144144
|> Web.URL.pathname()
@@ -168,4 +168,9 @@ defmodule Web.Dispatcher.TCP do
168168
{host, port}
169169
end
170170
end
171+
172+
defp remote_like_url?(url) do
173+
Web.URL.hostname(url) == "" and Web.URL.protocol(url) != "" and
174+
not String.starts_with?(Web.URL.pathname(url), "/")
175+
end
171176
end

0 commit comments

Comments
 (0)