We evaluated five AI answer engines as crawl targets: ChatGPT, Gemini, Perplexity, Claude, and Google AI Overview. Only ChatGPT made it into the default configuration. This note records what we tried on each engine and why the others were dropped.
Perplexity's front-end is gated by Cloudflare Turnstile, which runs a two-layer bot check:
- Fingerprint layer —
navigator.webdriver,Runtime.enable, Canvas/WebGL fingerprints, TLS fingerprint.playwright-extra+puppeteer-extra-plugin-stealthdefeats most of it. - Behavioral layer — mouse trajectory, dwell time, IP reputation, browsing history. The stealth plugin does nothing at this layer.
What we observed:
- Headless: blocked before
perplexity.aifinished loading. - Visible (real Chromium window): the "verifying you are human" spinner never clears. Cloudflare has already classified the session as a bot; it just holds the page in a soft-loading state instead of returning an error.
- CDP attach (launch real Chrome, solve manually, then connect via
--cdp): works for the first query or two, then the same profile gets blocked.
The behavioral layer can't be bypassed from inside the browser. Residential-IP proxy services (Bright Data, Oxylabs) exist specifically because this is unsolved at the open-source level.
claude.ai requires an authenticated session, which means cookies need to be seeded and refreshed periodically. It is also behind Cloudflare.
Beyond the access problem, Claude does not perform real-time web search by default — answers come from training data, not a live RAG pipeline. For a visibility monitor, an answer that doesn't reflect the current web isn't useful.
Google search's AI Overview is protected by reCAPTCHA; neither headless nor visible mode consistently cleared the challenge.
A second problem: AI Overview only appears for some queries, determined by a Google-side heuristic we can't control. That makes it a poor monitoring surface — absence of data is ambiguous between "brand wasn't mentioned" and "Google didn't render an overview".
Gemini runs headless reliably, with one caveat. Playwright's bundled Chrome-for-Testing binary is fingerprinted as a bot: form submissions silently fail with Something went wrong (13) and no error is surfaced to the page.
Workaround: override Playwright's launch channel to 'chrome' so the crawler uses the user's installed /Applications/Google Chrome.app instead of the bundled build. With that change Gemini submissions succeed and anonymous queries work — no login needed.
We removed Gemini from the default config to keep the OSS release focused on one engine, but the launchChannel pattern is preserved in BaseCrawler for anyone re-adding it.
| Engine | Status | Blocker |
|---|---|---|
| ChatGPT | Default | — |
| Gemini | Architecture supported, not default | Needs real Chrome (launchChannel='chrome') |
| Perplexity | Removed | Cloudflare Turnstile behavioral layer |
| Claude | Removed | Login + no real-time search |
| Google AI Overview | Removed | reCAPTCHA + non-deterministic surface |
Adding any of the removed engines is a subclass of BaseCrawler and a one-line addition to LLM_PLATFORMS — see the "Adding a platform" section in the README.