RFC: Automatic pagination detection for crawls (reduce manual “next page” rules)

[vigneshwarrvenkat]

Feature description

Detect common pagination patterns (e.g. rel="next", “next”/localized link text, numbered pages, query params like ?page=) and optionally emit follow-up requests with configurable confidence / overrides.

Motivation

Most crawls repeat the same boilerplate; mistakes here waste bandwidth and time.

Proposal (high level)

• Opt-in API at spider or LinkExtractor level; verbose logging when a pattern is chosen.
• Hook naturally into CrawlSpider / LinkExtractor.

Related

• ROADMAP.md: “Add functionality to automatically detect pagination URLs”

Open questions

• How to handle JS-only pagination (document as unsupported vs hook for manual session)?
• False-positive policy: conservative defaults vs “aggressive mode”?

(I searched existing feature requests for this topic.)

[vigneshwarrvenkat]

Following up on this RFC with a clearer two-phase plan. The roadmap item is “auto-detect pagination URLs".
I think the cleanest way to deliver that is in two steps, not one big feature.

Phase 1 — find the next link in HTML

Many sites already put the next page in the markup. Today every spider writes the same logic by hand. Phase 1 would be a small helper that looks at a response and returns the next page URL if it finds one.

It would check common patterns in order: rel="next", obvious “next” navigation, link text like “Next” or “›”, and simple URL patterns like /page/2 or ?page=2. If nothing matches, it returns nothing. This stays lightweight

Phase 2 — pagination that needs a browser

Some sites don’t expose a next URL in the HTML. Content loads when you scroll, click “Load more”, or wait for JavaScript. That’s a different problem which needs a live browser, not HTML parsing alone.

Scrapling already supports browser automation via DynamicFetcher, StealthyFetcher, and page_action. Phase 2 would add a reusable helper for the common cases:

click a “next” control and wait for the page to update
click “Load more” until it stops appearing
scroll down until no new content loads
It would stop when pagination is exhausted, or when safe limits are hit (max clicks, timeout, no change in content). When done, the fetcher returns the final HTML as usual with no new response type.

Please provide your feedback. Happy to start working on this.
@D4Vinci

[D4Vinci]

Hi, this can be added easily on the user side via 'LinkExtractor', so we are not planning to add it. The roadmap file is very old and needs an update.