Skip to content

fix: avoid duplicate final crawler persistence#3866

Open
AkhilTrivediX wants to merge 1 commit into
apify:masterfrom
AkhilTrivediX:agent/deduplicate-final-crawler-persistence
Open

fix: avoid duplicate final crawler persistence#3866
AkhilTrivediX wants to merge 1 commit into
apify:masterfrom
AkhilTrivediX:agent/deduplicate-final-crawler-persistence

Conversation

@AkhilTrivediX

Copy link
Copy Markdown

Summary

Avoid duplicate final persistence during BasicCrawler.run() shutdown.

  • Stop statistics before crawler teardown so the final statistics record is written once with its completion timestamp.
  • Let a crawler-owned event manager emit the final persistence event after the session pool listener is removed.
  • Keep the explicit persistence event for shared event managers, while preventing SessionPool.teardown() from writing the same state a second time.

Root cause

The shutdown path emitted PERSIST_STATE, then directly persisted the session pool and statistics again. A crawler-owned event manager emitted another final event as it closed, which added a second write for statistics.

Validation

  • corepack yarn vitest run test/core/crawlers/basic_crawler.test.ts --silent (70 passed)
  • corepack yarn eslint packages/core/src/session_pool/session_pool.ts packages/basic-crawler/src/internals/basic-crawler.ts test/core/crawlers/basic_crawler.test.ts
  • corepack yarn tsc-check-tests
  • corepack yarn biome format --write ...
  • git diff --check

Fixes #3167

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Double persisting Crawlee's SessionPool and Statistics at the end of crawler.run

2 participants