fix(core): unblock backpressured sitemap load on persistState#3863
Open
anxkhn wants to merge 1 commit into
Open
fix(core): unblock backpressured sitemap load on persistState#3863anxkhn wants to merge 1 commit into
anxkhn wants to merge 1 commit into
Conversation
SitemapRequestList.pushNextUrl() blocks under backpressure by waiting for a 'readdata' event on the current urlQueueStream. persistState() drains that stream and replaces it with a new one, but 'readdata' is only ever emitted by readNextUrl() on the current stream. After the swap, the waiter parked on the old stream is never notified, so the background load() stalls, the sitemap is never marked fully loaded, and isFinished() never becomes true (the crawler hangs). Re-emit 'readdata' on the previous stream after swapping so any pending pushNextUrl() waiter is released; the pending URL has already been transferred to the new stream, so nothing is lost. Adds a regression test that persists state while the loader is backpressured (maxBufferSize: 1) and asserts the list finishes with all URLs. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
SitemapRequestListcan deadlock its background sitemap loader whenpersistState()runs while the loader is parked on stream backpressure.pushNextUrl()pushes a URL intothis.urlQueueStreamand, when the stream isfull, blocks by waiting for a custom
readdataevent on that stream object:That
readdataevent is only ever emitted byreadNextUrl(), on whateverthis.urlQueueStreamcurrently is.persistState()drains the current stream,creates a fresh one, transfers the drained URLs, and swaps it in:
If a
pushNextUrl()call was backpressured on the previous stream when the swaphappens, its
once('readdata')listener stays on the now-orphaned old stream.Nothing ever emits
readdataon that old stream again, so the promise neverresolves. The background
load()task is stuck atawait this.pushNextUrl(...),inProgressSitemapUrlis never cleared,isSitemapFullyLoaded()staysfalse,and
isFinished()never becomestrue, so the crawler hangs.persistState()is timer-driven (registered on thePERSIST_STATEevent), sothis can trigger on any long-running sitemap load, not only on manual calls.
Fix
Re-emit
readdataon the previous stream right after the swap, releasing anypending
pushNextUrl()waiter. The URL that waiter pushed has already beendrained and transferred into the new stream, so resolving it loses nothing and
introduces no duplicate.
emit('readdata')with no listener is a safe no-op.This mirrors the existing unblock idiom already used in
teardown(), whichemits
readdatafor the same reason (to unblock a waitingpushNextUrl()).Testing
Added a regression test to
test/core/sitemap_request_list.test.ts:maxBufferSize: 1forces the loader to block on backpressure right after thefirst URL,
persistState()is called at that moment, and the test asserts thelist drains completely, reaches
isSitemapFullyLoaded()/isFinished(), andyields every URL. It times out (deadlocks) on
masterand passes with the fix.yarn vitest run test/core/sitemap_request_list.test.ts: 18/18 pass (the newtest times out at 10s without the source change).
yarn eslintandyarn biome formaton both changed files: clean.tsc --noEmitfor the core build config andtsc-check-tests: both pass.Well-behaved loads are unaffected:
readNextUrl()still emitsreaddataon thecurrent stream as before, and the extra emit on the previous stream only
resolves an already-satisfied waiter.