WebRunner: persistent Chrome profile + interactive-login mode for Scholar

Google's bot detection rejects WebRunner requests too once an IP is
flagged from earlier scraping. The reliable workaround is to seed a
real Google sign-in into a Chrome profile dir once; subsequent
headless runs reuse the session cookie and Google trusts the
authenticated request.

Two new env vars in sources/scholar/webrunner_backend.py:

  AUTOPAPERTOPPT_CHROME_PROFILE_DIR
    When set, passes --user-data-dir=<path> to Chrome so cookies,
    login state, and captcha clearance survive across CLI runs.

  AUTOPAPERTOPPT_CHROME_HEADLESS
    Default '1' (headless). Set '0' for the one-time interactive
    sign-in that seeds the profile dir. When non-headless the
    Chrome window holds open for 60 s (vs the 3 s headless wait)
    so the user has time to log in / accept consent banners /
    solve any captcha.

_build_chrome_args extracted from _drive_chrome_sync so the args
construction is unit-testable without spinning up Chrome.

Tests
-----
+3 tests covering: default headless + no profile, profile dir
passed as --user-data-dir, HEADLESS=0 drops the headless flag.

477 tests pass, ruff + bandit clean.

Docs
----
configuration.md gains two env var rows + a new 'Suppressing
Scholar captchas with a persistent Chrome profile' recipe walking
through the one-time interactive setup and the steady-state
headless usage, with caveats about profile-dir locking, cookie
expiry, and treating the profile dir as a secret.

Author: JE-Chen

docs/configuration.md

@@ -32,6 +32,8 @@ each value into `os.environ` before any fetcher initialises.
 | `AUTOPAPERTOPPT_CROSSREF_PLUS_TOKEN` | unset | Crossref Plus subscriber token. Attached to requests as `Crossref-Plus-API-Token: Bearer <token>`. Raises rate limits and improves cache freshness on the `acm` and `crossref` plugins. |
 | `AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING` | unset | **Scholar plugin is now default-ON.** Set `=1` to opt out. Google's ToS forbids automated access; default-on for coverage, opt-out if you'd rather not take the captcha / IP-block risk. |
 | `AUTOPAPERTOPPT_DISABLE_WEBRUNNER` | unset | The Scholar plugin defaults to driving a real Chrome through **WebRunner** (`je_web_runner` is a default dependency) — Google's bot-detection is far less aggressive on real browsers. Set `=1` to force the httpx scrape path even when WebRunner is available (useful for CI / Docker containers without a Chrome binary, or latency-sensitive runs where Chrome's 5-10 s boot isn't worth it). When set, the plugin's existing captcha-cooldown logic kicks in for the httpx path. |
+| `AUTOPAPERTOPPT_CHROME_PROFILE_DIR` | unset | When set, passes `--user-data-dir=<path>` to Chrome so cookies / login state / captcha clearance survive across CLI invocations. A one-time interactive Google sign-in seeded into this profile dir reliably suppresses Scholar captchas on subsequent headless runs. See the **Suppressing Scholar captchas with a persistent profile** recipe below. |
+| `AUTOPAPERTOPPT_CHROME_HEADLESS` | `1` (headless) | Set `=0` to open Chrome visibly. Used ONLY for the one-time interactive Google sign-in that seeds `AUTOPAPERTOPPT_CHROME_PROFILE_DIR`. Subsequent runs should drop this env var so Chrome stays headless. |
 | `AUTOPAPERTOPPT_CORE_API_KEY` | unset | Free key from <https://core.ac.uk/services/api>. Enables the OA resolver's CORE.ac.uk lookup step (200M+ institutional / regional OA repository items). Skipped silently when unset (the other OA strategies — Unpaywall, Semantic Scholar, arXiv — still run). |
 | `AUTOPAPERTOPPT_CONTACT_EMAIL` | unset | Sent to Crossref / OpenAlex as the `mailto=` parameter (entry into their polite pool), to NCBI as `tool` / `email` headers, **and to Unpaywall as `email=`** for the post-dedup OA PDF resolver. Highly recommended — without it the resolver skips Unpaywall lookups entirely, which is the single biggest PDF coverage win for IEEE / ACM / Springer / Elsevier paywalled papers (typical lift 40-70%). |
 
@@ -164,6 +166,50 @@ Override via:
 Clear the cache by deleting the directory; AutoPaperToPPT
 re-creates it on demand.
 
+## Suppressing Scholar captchas with a persistent Chrome profile
+
+Google flags an IP after a few automated Scholar requests even with
+WebRunner's real-browser path. The reliable workaround is to seed a
+persistent Chrome profile with a real Google sign-in once; subsequent
+headless runs reuse the same session cookies, which Google trusts.
+
+**One-time setup:**
+
+```powershell
+# 1. Pick a directory anywhere on disk
+$env:AUTOPAPERTOPPT_CHROME_PROFILE_DIR = "D:\autopapertoppt-scholar-profile"
+
+# 2. Open Chrome visibly and trigger one Scholar request
+$env:AUTOPAPERTOPPT_CHROME_HEADLESS = "0"
+autopapertoppt --query "any keywords" --source scholar --max 1 --out .\tmp\
+
+# Chrome opens. Sign into your Google account, accept any consent
+# banners, complete any captcha. The window holds open for 60s.
+```
+
+**Every run after that:**
+
+```powershell
+$env:AUTOPAPERTOPPT_CHROME_PROFILE_DIR = "D:\autopapertoppt-scholar-profile"
+Remove-Item Env:\AUTOPAPERTOPPT_CHROME_HEADLESS   # back to headless
+autopapertoppt --query "..." --out .\exports\
+```
+
+Chrome boots headless but loads the same profile dir, sends your
+authenticated Google session cookie, and Scholar serves real results
+instead of a captcha page.
+
+**Caveats:**
+
+- Only one Chrome process can hold the profile dir at a time. If you
+  have a regular Chrome open on the same profile path, the
+  WebRunner instance will fail to start. Use a dedicated path.
+- The session cookie is a real authentication credential. Treat the
+  profile directory like a secret — back it up if you re-image the
+  machine, restrict file permissions.
+- Cookie eventually expires (~1-2 months for Google). Re-do the
+  interactive sign-in then.
+
 ## Settings the project explicitly does NOT have
 
 By design — listing them so a contributor doesn't accidentally

sources/scholar/webrunner_backend.py

@@ -37,7 +37,19 @@
 _LOG = get_logger(__name__)
 _SEARCH_URL_BASE = "https://scholar.google.com/scholar"
 _DISABLE_ENV = "AUTOPAPERTOPPT_DISABLE_WEBRUNNER"
+#: Chrome user-data directory to persist between runs. When set, the
+#: session cookies from any prior login (Google account, captcha
+#: clearance, etc.) survive across CLI invocations — a one-time
+#: interactive login in this profile dramatically reduces captcha hits.
+_PROFILE_DIR_ENV = "AUTOPAPERTOPPT_CHROME_PROFILE_DIR"
+#: Set to ``0`` to disable headless mode (needed for the one-time
+#: interactive login into Google when seeding the profile dir).
+_HEADLESS_ENV = "AUTOPAPERTOPPT_CHROME_HEADLESS"
 _PAGE_LOAD_WAIT_SECONDS = 3.0
+#: When the headless flag is OFF (interactive login mode) we hold the
+#: window open longer so the user has time to complete the Google
+#: sign-in flow before the search returns.
+_INTERACTIVE_WAIT_SECONDS = 60.0
 
 
 def is_available() -> bool:
@@ -86,29 +98,27 @@ def _drive_chrome_sync(url: str) -> str:
     # path before it ever runs.
     from je_web_runner import webdriver_wrapper_instance
 
-    chrome_args = [
-        "--disable-blink-features=AutomationControlled",
-        "--lang=en-US",
-        # Headless new mode passes more of Google's automation
-        # detection than the legacy --headless flag.
-        "--headless=new",
-        # Reduce fingerprint surface.
-        "--disable-gpu",
-        "--no-sandbox",
-        "--window-size=1280,720",
-    ]
+    chrome_args, headless = _build_chrome_args()
     try:
         webdriver_wrapper_instance.set_driver("chrome", options=chrome_args)
     except Exception as err:  # noqa: BLE001 — Selenium raises many types
         raise RuntimeError(f"WebRunner cannot start chrome: {err}") from err
 
     try:
         webdriver_wrapper_instance.to_url(url)
-        # Page needs a beat to render results client-side. We don't use
-        # a smart-wait helper here because the SERP layout has no single
-        # stable readiness signal that survives captcha vs. real results.
+        # Headless: tight wait, only need the SERP HTML. Non-headless
+        # (interactive login mode): hold the window open so the user
+        # has time to complete the Google sign-in flow.
+        wait = _PAGE_LOAD_WAIT_SECONDS if headless else _INTERACTIVE_WAIT_SECONDS
+        if not headless:
+            _LOG.warning(
+                "Chrome opened in interactive mode for %.0fs — sign into "
+                "Google in the window now. Session cookies will persist "
+                "in the profile dir for subsequent headless runs.",
+                wait,
+            )
         import time
-        time.sleep(_PAGE_LOAD_WAIT_SECONDS)
+        time.sleep(wait)
         return webdriver_wrapper_instance.current_webdriver.page_source
     except Exception as err:  # noqa: BLE001 — best-effort
         raise RuntimeError(f"WebRunner page-load failed: {err}") from err
@@ -117,3 +127,35 @@ def _drive_chrome_sync(url: str) -> str:
             webdriver_wrapper_instance.quit()
         except Exception as err:  # noqa: BLE001  # nosec B110 — best-effort cleanup
             _LOG.debug("WebRunner cleanup failed: %s", err)
+
+
+def _build_chrome_args() -> tuple[list[str], bool]:
+    """Return ``(chrome_args, is_headless)`` based on env-var overrides.
+
+    Layered on top of the always-applied anti-detection flags:
+
+    - ``AUTOPAPERTOPPT_CHROME_PROFILE_DIR=<path>``: pass
+      ``--user-data-dir=<path>`` so cookies / login state persist.
+    - ``AUTOPAPERTOPPT_CHROME_HEADLESS=0``: drop ``--headless=new`` so
+      the user can interact with the Chrome window (required for the
+      one-time Google sign-in that seeds the profile dir).
+    """
+    chrome_args = [
+        "--disable-blink-features=AutomationControlled",
+        "--lang=en-US",
+        # Reduce fingerprint surface; safe in both headless and visible
+        # modes.
+        "--disable-gpu",
+        "--no-sandbox",
+        "--window-size=1280,720",
+    ]
+    headless = os.environ.get(_HEADLESS_ENV, "1") != "0"
+    if headless:
+        # Headless 'new' passes more of Google's automation detection
+        # than legacy --headless.
+        chrome_args.append("--headless=new")
+    profile_dir = os.environ.get(_PROFILE_DIR_ENV, "").strip()
+    if profile_dir:
+        chrome_args.append(f"--user-data-dir={profile_dir}")
+        _LOG.info("Chrome using persistent profile at %s", profile_dir)
+    return chrome_args, headless

tests/sources/test_scholar.py

@@ -205,3 +205,36 @@ def test_webrunner_is_available_respects_disable_env(monkeypatch):
 
     monkeypatch.setenv("AUTOPAPERTOPPT_DISABLE_WEBRUNNER", "1")
     assert webrunner_backend.is_available() is False
+
+
+def test_chrome_args_default_headless_with_no_profile(monkeypatch):
+    """No env vars → headless + anti-detection flags, no profile dir."""
+    from scholar import webrunner_backend
+
+    for var in ("AUTOPAPERTOPPT_CHROME_PROFILE_DIR", "AUTOPAPERTOPPT_CHROME_HEADLESS"):
+        monkeypatch.delenv(var, raising=False)
+    args, headless = webrunner_backend._build_chrome_args()  # noqa: SLF001
+    assert headless is True
+    assert "--headless=new" in args
+    assert "--disable-blink-features=AutomationControlled" in args
+    assert not any(a.startswith("--user-data-dir=") for a in args)
+
+
+def test_chrome_args_with_profile_dir_passes_user_data_dir(monkeypatch):
+    from scholar import webrunner_backend
+
+    monkeypatch.setenv("AUTOPAPERTOPPT_CHROME_PROFILE_DIR", "D:/scholar-profile")
+    monkeypatch.delenv("AUTOPAPERTOPPT_CHROME_HEADLESS", raising=False)
+    args, headless = webrunner_backend._build_chrome_args()  # noqa: SLF001
+    assert headless is True
+    assert "--user-data-dir=D:/scholar-profile" in args
+
+
+def test_chrome_args_headless_zero_drops_headless_flag(monkeypatch):
+    """AUTOPAPERTOPPT_CHROME_HEADLESS=0 enables the interactive-login mode."""
+    from scholar import webrunner_backend
+
+    monkeypatch.setenv("AUTOPAPERTOPPT_CHROME_HEADLESS", "0")
+    args, headless = webrunner_backend._build_chrome_args()  # noqa: SLF001
+    assert headless is False
+    assert "--headless=new" not in args