Stealthy fetcher mode triggers unwanted ad/analytics requests impacting proxy usage

[rubiagul280]

When loading a page in stealthy fetcher mode, the request unexpectedly triggers dozens of additional calls to ad networks, analytics trackers, and third‑party resources.
Even after disabling resource loading, these ad/analytics requests still occur and inflate proxy usage—sometimes 80–100 extra requests for what should be a single page fetch. This makes proxy bandwidth consumption and request counts much higher than expected.

Question:
How can we effectively block or filter these unwanted third‑party requests so that only the main domain traffic is counted and proxied? Are there recommended approaches (e.g., request interception, domain whitelisting, or proxy‑side filtering) to ensure stealthy fetcher mode doesn’t include ads/analytics calls?

[D4Vinci]

Hi mate, is playing around with load_dom=False and network_idle=True not helping in your case? If that's the case, I would recommend some more advanced solutions till the next release is out.

[rubiagul280]

No, it didn't work for me. Here is the code

async def _fetch_first_page(self, search_url: str) -> Optional[str]:
    """Fetch first page using StealthyFetcher with load_dom=False to prevent JS execution."""
    try:
        logger.info(f"Starting page fetch: {search_url}")
        proxy = self._get_proxy_config()

        logger.info("Calling StealthyFetcher.async_fetch with load_dom=False...")
        response = await StealthyFetcher.async_fetch(
            url=search_url,
            headless=True,
            disable_resources=True,  # Block fonts, images, media, stylesheets, etc.
            load_dom=False,          # Don't wait for JS to execute - prevents tracking scripts
            network_idle=True,       # Wait for network to settle
            timeout=60000,
            wait=2000,
            proxy=proxy,
            # Anti-detection
            google_search=True,
            block_webrtc=True,
            hide_canvas=True,
            allow_webgl=True,
        )

        logger.info(f"Response received: {type(response)}")
        if not response:
            logger.error("Response is None")
            return None

        # Get HTML content from response
        html_content = getattr(response, 'html', None) or getattr(response, 'text', None) or str(response)
        logger.info(f"Got HTML content, length: {len(html_content) if html_content else 0}")

        if not html_content:
            logger.error("No HTML content found")
            return None

        return html_content

    except Exception as e:
        logger.error(f"Error fetching first page: {e}")
        return None

[D4Vinci]

Hey @rubiagul280

Before trying complicated solutions, can you try the beta of the next version and let me know if it resolves this issue for you?
Use this command to install pip install "git+https://github.com/D4Vinci/Scrapling.git@dev#egg=scrapling[all]" -U

This update includes several changes, some of which aim to further optimize browser sessions. Like automatically preventing autoplay media, such as ads, from autoplaying as they do now.

The browser fetchers/sessions, such as StealthyFetcher, now have a new argument called blocked_domains, which takes a set of domain names (strings) and prevents requests to those domains. This will be very helpful with ads.

Checkout the new readme for it here for some explanation: https://github.com/D4Vinci/Scrapling/tree/dev
This version is currently being discussed on the Discord server before release.

[rubiagul280]

@D4Vinci Thanks for the update. I will test it and get back to you

[Nooruddin87]

Use request interception in tools like Puppeteer or Playwright to whitelist only the target domain and abort anything matching ad/analytics patterns (e.g., Google Analytics, DoubleClick). You can also add proxy-side rules (like in mitmproxy) to block known tracker domains so only first-party traffic goes through and your proxy usage stays clean.