Conversation
There was no way to tell whether fetch-path time was network or CPU, so there was no basis for deciding what to optimise. This measures both halves separately over a caller-supplied URL list. Reports p50/p90/p99 for fetch and extract, the extract share of measured time, and how many HTTP 200s carry under 10 KB of HTML or under 50 extracted words — the shape downstream consumers re-fetch through a browser, which costs an order of magnitude more than a direct fetch. The URL list is passed at runtime and never committed: a realistic corpus comes from real traffic, which is not public data. benchmarks/latency.md documents how to run it and, more importantly, how to read it — including that p99 sits at the configured timeout rather than measuring anything, and that a thin-body trip means opposite things depending on whether it fired on word count or byte size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The client-level `.timeout()` governs getting a response head; it does not cover streaming the body. The read loop had a size ceiling but no deadline, so a peer that stopped sending mid-body held the connection open indefinitely and a fetch could run well past the configured timeout. Measured on a real corpus: a 12s timeout produced fetches of 22.6s. The size ceiling cannot help — a stalled stream never reaches it. Bounded as an IDLE window rather than a total-body deadline, and that distinction is the whole point. A total deadline punishes size instead of stalling: the first attempt at this killed a healthy 16.5 MB CSV export outright, and where it survived it did so only by failing and retrying, which cost more than the original problem. A stalled peer is distinguishable because it stops delivering chunks, so the clock resets on every chunk and only silence trips it. That download now completes normally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Latency depends heavily on where the request leaves from, and the
harness could only measure this host's own connection. Adds:
- BENCH_PROXY_FILE — builds the rotating pool, so pool behaviour
(per-host client affinity, reuse across the pool) is measured rather
than assumed. WEBCLAW_PROXY still routes everything through one.
- BENCH_LABEL — tags every row, so runs across egress paths can be
concatenated and compared per-URL rather than only in aggregate.
- success rate and total bytes fetched — metered egress is billed per
GB, so a run's cost is now visible alongside its latency.
Proxy lists hold credentials and stay outside the repo, same as the URL
corpus.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`error` was set only on transport failure, so a 403 challenge page — which arrives quickly, carries a body, and raises no error — scored as a successful fetch. That lets an egress path that gets BLOCKED look faster than one served the real page. This was not hypothetical. An A/B of two proxy exits produced an apparent 28x speedup that was a 403 stub (6.4 KB, 0 words) timed against a real 200 (445 KB). Under a status + body-size gate the same comparison is ~2x, and the pooled result across the corpus drops from "halves latency" to a paired median around 1.5-2.1x. Latency percentiles now cover 2xx only; non-2xx and transport errors are counted and reported separately, with the distinct status codes listed so a suspiciously fast arm is visible rather than flattering. benchmarks/latency.md gains the two traps this exposed: fast-failure scoring, and a noise floor measured at 1.8x mean across four runs of an unchanged config — which is larger than most effects anyone will want to claim. Includes the design that actually settles such a comparison: interleaved arms, a null A-vs-A arm, dedup to backends rather than hostnames, and paired medians instead of aggregate p50 deltas. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MCP registry has advertised 0.6.17 since 2026-07-22, but no matching tag was ever cut, so 0.6.16 stayed the newest release and the registry pointed at a version that did not exist. Bumps the workspace to 0.6.17 so a tag can be pushed and the two agree. Also drops a competitor's name from the Chinese README. The registry manifest was reworded at the same time as the 0.6.17 entry and that text is live and clean, but README_zh-CN kept the old "self-hostable <competitor> alternative" phrasing while the English README had already been changed — so the claim stayed visible on the repo page. Both now describe webclaw on its own terms. Note the registry keeps immutable per-version history: the 0.1.6 and 0.6.16 entries still carry the old wording and cannot be edited. Only the latest entry is served, and that one is already correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 023c836.
Three changes to the core HTTP path, all deterministic and unit-verified. No behaviour change. 1. `into_text` no longer copies the body. `Response.body` becomes `Vec<u8>`, so the valid-UTF-8 case — effectively every page — hands its allocation straight to `String` instead of allocating a second full-size buffer and memcpy'ing into it. Measured: 10 KB 7.0 -> 1.8 us, 500 KB 337 -> 55 us, 16 MB 5269 -> 1508 us. Peak resident per in-flight fetch drops from 2x to 1x the body size (100 MB -> 50 MB at MAX_BODY_BYTES). A second effect compounds it: `String::from_utf8`'s validation beats `from_utf8_lossy`'s chunked scan on ASCII-dominant input, 512 KB pure ASCII 122 -> 9.4 us. Invalid UTF-8 must stay byte-identical, so the fallback decodes `e.as_bytes()` rather than re-deriving from the moved buffer. Covered by a 19-case equivalence table (BOM, truncated 2/3/4-byte sequences, lone surrogate, overlong NUL, >U+10FFFF, windows-1252, bad bytes at buffer edges) asserting the output matches the old implementation exactly, plus a pointer-identity test proving the valid path does not re-allocate. Charset handling is deliberately untouched. 2. `pool_max_idle_per_host(8)` -> 32. wreq's default is `usize::MAX`, so this call was REMOVING pooling capacity, not adding it. At 20 concurrent requests to an HTTP/1.1 origin, 8 connections were pooled and 12 closed on release — a 60% reconnect rate in steady state, each costing a fresh TCP + TLS handshake. Kept bounded rather than removed because `BrowserProfile::Random` builds six clients, each with its own pool. 3. Dropped a redundant `HeaderMap` clone in the extract path; both consumers take `&HeaderMap`. 26 allocations / ~2.8 KB per fetch. Deliberately NOT done: seeding the buffer with `content_length()` as a capacity hint. It is `None` for essentially every HTML page (the decompression layer reports no exact size hint), and where it is `Some` it is an unverified attacker-chosen wire value — `Content-Length: 52428799` followed by FIN would commit 50 MB before the first byte arrives, inverting the property `check_body_ceiling` exists to preserve. A fixed hint measured slower at 16 MB and over-allocates 6.8x for a small page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`CrawlResult.pages` collects every page's full extraction and the crawl only returns once it has finished, so peak memory is O(pages x page size) with all of it live at once. Measured at ~90 KB per extracted page: ~440 MB for 5 000 pages, ~4.4 GB for 50 000. That, not the crawl itself, is what the page ceilings are protecting. `progress_tx` already streamed each page as it completed, but the page was pushed onto `pages` regardless, so a streaming consumer paid the full retention cost anyway. `stream_only` drops the page once reported. Peak memory then stays flat regardless of page count, which is what makes raising or removing a ceiling viable. Page counting moves off `pages.len()` to explicit counters, so total/ok/errors stay correct when nothing is retained — those are all a streaming consumer has left. In stream-only mode the page is moved into the channel rather than cloned, since nothing downstream retains it. Defaults to false: every existing caller keeps collecting, and a test pins that default because flipping it would silently empty `CrawlResult::pages` for all of them. Not addressed here, and still bounding a very large crawl: the visited set and frontier are held as owned Strings (frontier capped at 10x max_pages), so they grow with the URL space rather than the page count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`CrawlConfig.max_pages` was a required `usize`, so an unbounded crawl
could not be asked for at any layer — the engine always stopped at a
number. A page cap is a policy decision (quota, budget, patience) that
belongs to whoever is paying for the crawl, not to the extraction
engine. Now `Option<usize>`, with `None` meaning "run until the frontier
is exhausted or the caller cancels".
Uncapped is opt-in everywhere; nothing changes for a caller who says
nothing:
- CrawlConfig::default() still caps at 50.
- CLI `--max-pages 0`, MCP `max_pages: 0`, OSS server `max_pages: 0`.
- OSS server's HARD_MAX_PAGES = 500 ceiling is gone. An omitted value
still defaults to 50 so a request that asks for nothing in
particular cannot run forever on a synchronous endpoint.
The frontier bound stays, and is now independent of the page cap. It is
a MEMORY rail, not a page limit: it bounds queued-but-unfetched URLs,
which are owned Strings held live for the whole crawl. Search listings,
tag clouds and faceted navigation emit thousands of links per page, and
a calendar-style URL space is effectively infinite — without this an
uncapped crawl dies on memory rather than finishing its work. Capped
crawls keep the existing 10x/5x behaviour; uncapped ones use a fixed
500k ceiling, roughly 50 MB of pending frontier.
The CLI and the OSS crawl route both render the full result at the end,
so they still collect pages; an uncapped crawl there grows with the
site. `stream_only` is what makes an uncapped crawl flat in memory, and
suits an async consumer. Also switches the resume-state page count off
`pages.len()` to `total`, which stays correct when nothing is retained.
BREAKING (minor) for downstream: `CrawlConfig.max_pages` is now
`Option<usize>`. Call sites need `Some(n)`.
Verified end to end: depth-1 uncapped crawl ran to 270 pages and stopped
when the frontier emptied, not at a ceiling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`FetchConfig.timeout` did not mean what it reads as. The HTTP client applies it as two independent budgets — one for the response head, then a fresh one armed when the head arrives for the body — so a single attempt takes up to 2x. The retry loop then runs two attempts with a 1s pause between them. A `timeout: 12s` config therefore permitted 49s, and nothing in the API said so. A request occupying a worker for 49s when the caller asked for 12 is a reliability problem, not a slow page. Adds `total_timeout`: a ceiling on one complete `fetch` covering every attempt, redirect and body read. Both retry loops now start a clock before the first attempt and run each attempt inside the remaining share, so the retry pause comes out of the same budget. An attempt is not started at all once the budget is spent, rather than being started and then abandoned. Defaults to `2 * timeout` (24s): a single slow-but-successful fetch keeps its full head+body allowance and is unaffected, while the retry can no longer extend past it. Retries exist for transient failures, which fail fast; retrying after a full-budget timeout rarely succeeds and always costs the caller another full budget. `None` restores the old compounding behaviour for anyone who wants it. `timeout`'s doc comment now states the 2x behaviour rather than leaving it to be discovered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`/v1/batch` refused more than 100 URLs. The cap existed because the aggregate response is built entirely in memory before anything is sent, so a large batch grew the process with the request — but that is a property of the response *shape*, not a sensible policy. Capping the count only moved the failure somewhere less obvious, and told callers with real work to go elsewhere. Adds `FetchClient::fetch_and_extract_batch_stream`, which yields each result as it completes. The existing batch fns spawn a task per URL up front and collect everything into a Vec, so both task count and memory scale with the input; this polls at most `concurrency` futures at a time via `buffer_unordered` and hands each result straight to the consumer. Results arrive in completion order — that is the point — so the ordered fns stay for callers that need order and can afford to buffer. `/v1/batch` gains `"stream": true`, which responds with NDJSON: one object per line, flushed as each URL lands. Nothing accumulates server-side, so HARD_MAX_URLS is gone. Verified against the self-host server: 150 URLs (past the old cap) all succeeded, and on a 100-URL batch the first line arrived at 0.10s of a 2.41s run — incremental, not buffered then dumped. `concurrency` stays bounded at 20. That one is politeness toward the sites being fetched, not a limit on the caller, and the doc comment now says so rather than implying it is about memory. A serialisation failure emits a parseable error line for that URL instead of tearing down the stream mid-batch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dget Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI's `cargo clippy --all -- -D warnings` rejected `Ok(builder...?)` in the NDJSON branch. The builder already returns the handler's error type, so the wrapper was pure noise. Missed locally because verification ran `cargo clippy --all --examples`, and `--examples` RESTRICTS target selection to examples rather than adding them to the default set — so the binaries, including the one that failed, were never linted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat: uncapped crawl, streaming batch, and one real fetch budget
CHANGELOG conflicted because both sides inserted at the same point in the [Unreleased] block: #98 added a Performance entry, this branch added the released [0.6.17] section. Both belong — the Performance note stays under [Unreleased] (it is not released yet) and [0.6.17] sits below it as a cut version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chore(release): 0.6.17 — drop competitor name from zh README, reconcile version
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
stagingtomain. Contains PR #98 and PR #97, both merged to staging with CI green.#98 — uncapped crawl, streaming batch, one fetch budget
CrawlConfig.max_pagesisOption<usize>;Nonecrawls until the site is exhausted.--max-pages 0on the CLI,max_pages: 0on the MCP tool and self-host server. Defaults unchanged, so a caller who says nothing still gets a bounded crawl.POST /v1/batchwith"stream": trueanswers NDJSON, one result per line as each URL finishes. Memory is flat regardless of batch size, and the maximum batch size is gone.stream_onlydrops each page after reporting it, so peak memory stays flat — this is what makes an unbounded crawl practical.timeouttwice per attempt and the retry loop ran two attempts, so a 12s timeout permitted 49s of wall clock.total_timeoutnow bounds the whole operation.#97 — 0.6.17 reconcile
The MCP registry has advertised 0.6.17 since 2026-07-22, but no tag was ever cut, so
0.6.16stayed the newest release while the registry pointed at a version that did not exist. This reconciles the repo and drops a competitor's name from the Chinese README so it matches the English one.Merge conflict, resolved
CHANGELOG.mdconflicted: both sides inserted at the same point in[Unreleased]. #98 added a Performance entry, #97 added the released[0.6.17]section. Both were kept — the Performance note stays under[Unreleased](not yet released) and[0.6.17]sits below it as a cut version. Verifiedcargo fmt --check --allclean and 682 lib tests passing on the merged tree before pushing.Not tagged
Cargo.tomlreads0.6.17, but the tag is deliberately not cut in this PR. Since #98 landed after the 0.6.17 notes were written, the next tag needs to contain it — so the release should be v0.6.18, decided separately.This matters downstream:
webclaw-serverPR #31 currently pins core by branch (branch = "feat/uncapped-crawl"), which breaks the tag-pinning rule and cannot ship. It gets repinned once a tag exists.🤖 Generated with Claude Code