Merge pull request #278 from autoscrape-labs/fix/multicontext-proxies

Fix problem when trying to use private proxies in new browser contexts

Author: thalissonvs

public/docs/api/commands/target.md

@@ -100,6 +100,9 @@ incognito_tab = await create_target(
 )
 ```
 
+!!! info "Headless vs Headed: how contexts show up"
+    Browser contexts are isolated logical environments. In headed mode, the first page created inside a new context will usually open in a new OS window. In headless mode, no window is shown — the isolation remains purely logical (cookies, storage, cache and auth state are still separate per context). Prefer contexts in headless/CI pipelines for performance and clean isolation.
+
 ## Advanced Features
 
 ### Target Events

public/docs/deep-dive/browser-domain.md

@@ -327,6 +327,85 @@ Browser contexts are essential for several automation scenarios:
 4. **Session Isolation**: Prevent cross-contamination between test scenarios
 5. **Parallel Scraping**: Scrape multiple sites with different configurations
 
+### Headless vs Headed: Windows and Best Practices
+
+Browser contexts are a logical isolation layer. What you actually see is the page created inside a context:
+
+- In headed mode (visible UI), creating the first page inside a new browser context will typically open a new OS window. The context is the isolated environment; the page is what renders in a tab or window.
+- In headless mode (no visible UI), no windows appear. The isolation still exists logically in the background, keeping cookies, storage, cache and auth state fully separate per context.
+
+Recommendations:
+
+- Prefer using multiple contexts in headless environments (e.g., CI/CD) for cleaner isolation, faster startup, and lower resource usage compared to launching multiple browser processes.
+- Use contexts to simulate multiple users or sessions in parallel without cross-contamination.
+
+Why contexts are efficient:
+
+- Creating a new browser context is significantly faster and lighter than starting a whole new browser instance. This makes test suites and scraping jobs more reliable and scalable.
+
+### CDP Hierarchy and Context Window Semantics (Advanced)
+
+To reason precisely about contexts, it's useful to map Pydoll concepts to CDP:
+
+- Browser (process): single Chromium process running the DevTools endpoint.
+- BrowserContext: isolated profile inside that process (cookies, storage, cache, permissions).
+- Target/Page: an individual top-level page, popup, or background target that you control.
+
+CDP and `browserContextId`:
+
+- When creating a page via `Target.createTarget`, passing `browserContextId` tells the browser which isolated profile the new page should belong to. Without this ID, the target is created in the default context.
+- The ID is essential for isolation — it binds the new target to the correct storage/auth/permission boundary.
+
+Why the first page in a context opens a window (headed):
+
+- In headed mode, a page needs a top-level native window to render. A freshly created context initially has no window associated with it — it exists only in memory.
+- The first page created in that context implicitly materializes a window for that context. Subsequent pages can open as tabs within that window.
+
+Implications for `new_window`/`newWindow` semantics:
+
+- If you attempt to create a page with "tab-like" behavior (no new top-level window) in a context that has no existing window (first page), the browser may error because there is no host window to attach the tab to.
+- Practically: treat the first page in a new context (headed) as requiring a top-level window. Afterwards, you can create additional pages as tabs.
+
+Headless mode makes this distinction moot:
+
+- With no visible UI, windows vs tabs are logical constructs only. Context isolation is enforced the same way, but nothing is rendered, so there is no requirement to bootstrap a native window for the first page.
+
+### Context-specific Proxy: sanitize + auth via Fetch events
+
+When creating a browser context with a private proxy (credentials embedded in the URL), Pydoll follows a two-step strategy to avoid leaking credentials and reliably authenticate:
+
+1) Sanitize the proxy server in the CDP command
+
+- If you pass `proxy_server='http://user:pass@host:port'`, only the credential-free URL is sent to CDP (`http://host:port`).
+- Internally, Pydoll extracts and stores the credentials keyed by `browserContextId`.
+
+2) Attach per-context auth handlers on first tab
+
+- When you open a `Tab` inside that context, Pydoll enables Fetch events for that tab and registers two temporary listeners:
+  - `Fetch.requestPaused`: continues normal requests.
+  - `Fetch.authRequired`: automatically responds with the stored `user`/`pass`, then disables Fetch to avoid intercepting further requests.
+
+Why this design?
+
+- Prevents credential exposure in command logs and CDP parameters.
+- Keeps the auth scope strictly limited to the context that requested the proxy.
+- Works in both headed and headless modes (the auth flow is network-level, not UI-dependent).
+
+Code flow highlights (simplified):
+
+```python
+# On context creation
+context_id = await browser.create_browser_context(proxy_server='user:pwd@host:port')
+# => sends Target.createBrowserContext with 'http://host:port'
+# => stores {'context_id': ('user', 'pwd')} internally
+
+# On first tab in that context
+tab = await browser.new_tab(browser_context_id=context_id)
+# => tab.enable_fetch_events(handle_auth=True)
+# => tab.on('Fetch.requestPaused', continue_request)
+# => tab.on('Fetch.authRequired', continue_with_auth(user, pwd))
+```
+
 ### Creating and Managing Contexts
 
 ```python

public/docs/zh/api/commands/target.md

@@ -101,6 +101,9 @@ incognito_tab = await create_target(
 )
 ```
 
+!!! info "Headless 与 Headed：上下文如何呈现"
+    浏览器上下文是逻辑上的隔离环境。在 Headed 模式下，在新的上下文中创建的第一个页面通常会打开一个新的系统窗口。 在 Headless 模式下不会显示窗口——隔离依然存在于后台（cookies、storage、缓存与认证状态仍按上下文分离）。在 CI/Headless 环境中优先使用上下文以获得更高性能与更干净的隔离。
+
 ## 高级特性
 
 ### 目标事件

public/docs/zh/deep-dive/browser-domain.md

@@ -325,6 +325,85 @@ graph TB
 4. **会话隔离**：防止测试场景间的交叉污染
 5. **并行抓取**：使用不同配置同时抓取多个网站
 
+### Headless 与 Headed：窗口表现与最佳实践
+
+浏览器上下文是一个逻辑上的隔离环境。实际显示在屏幕上的，是在该上下文内创建的页面（page）：
+
+- 在 Headed 模式（可见 UI）下，在新的浏览器上下文内创建第一个页面通常会打开一个新的系统窗口。上下文是隔离的环境；页面才是会在标签页或窗口中渲染的对象。
+- 在 Headless 模式（无界面）下，不会出现可见窗口。上下文的隔离仍然存在于后台，确保 cookies、storage、缓存与认证状态在不同上下文之间完全分离。
+
+建议：
+
+- 在 CI/CD 等无界面环境中，优先使用多个上下文来实现隔离。相比启动多个浏览器进程，创建新上下文更快、资源占用更低。
+- 使用上下文来并行模拟多个用户或会话，避免相互污染。
+
+为什么上下文更高效：
+
+- 创建浏览器上下文远比启动一个新的浏览器实例更轻量、更迅速。这将使测试套件与抓取任务更稳定、更具可扩展性。
+
+### CDP 层级与上下文窗口语义（高级）
+
+将 Pydoll 概念映射到 CDP，有助于精确理解上下文：
+
+- 浏览器（进程）：运行 DevTools 端点的单个 Chromium 进程。
+- 浏览器上下文（BrowserContext）：该进程内的隔离“用户配置文件”（cookies、存储、缓存、权限相互独立）。
+- 目标/页面（Target/Page）：可控制的顶层页面、弹窗或后台目标。
+
+CDP 与 `browserContextId`：
+
+- 使用 `Target.createTarget` 创建页面时传入 `browserContextId`，告诉浏览器将新页面归属到指定的隔离配置文件。未传入时，目标将创建在默认上下文中。
+- 该 ID 是实现隔离的关键——它将新目标绑定到正确的存储/认证/权限边界。
+
+为何上下文中的“第一个页面”在 Headed 模式下会打开窗口：
+
+- 在 Headed 模式中，页面需要一个顶层的原生窗口来渲染。新创建的上下文起初只存在于内存中，并没有关联的窗口。
+- 在该上下文中创建的第一个页面会隐式“实体化”一个窗口。之后再创建的页面可以作为该窗口中的标签页加入。
+
+对 `new_window`/`newWindow` 语义的影响：
+
+- 如果你希望以“仅新标签”的方式创建页面（不新建顶层窗口），但目标上下文尚无窗口（即第一个页面），浏览器可能会报错，因为没有可附着的宿主窗口。
+- 实践上：在新的上下文（Headed）里，首个页面可视为需要一个顶层窗口；随后你就可以创建额外页面作为标签页。
+
+在 Headless 模式下，这个区分不再重要：
+
+- 没有可见 UI 时，“窗口 vs 标签”的区别只是逻辑概念。隔离照常生效，但无需为首个页面引导原生窗口。
+
+### 上下文专属代理：URL 净化 + 通过 Fetch 事件进行认证
+
+当你为某个浏览器上下文配置带凭证的私有代理（在 URL 中嵌入用户名/密码）时，Pydoll 采用“两步法”以避免凭证泄漏并实现可靠认证：
+
+1）在 CDP 命令中净化代理地址
+
+- 若传入 `proxy_server='http://user:pass@host:port'`，发送给 CDP 的仅为去除凭证的 URL（`http://host:port`）。
+- 在内部，Pydoll 会提取并按 `browserContextId` 存储凭证。
+
+2）在该上下文的首个 Tab 上附加认证处理器
+
+- 在该上下文内打开 Tab 时，Pydoll 会为该 Tab 启用 Fetch 事件，并注册两个临时监听器：
+  - `Fetch.requestPaused`：继续普通请求。
+  - `Fetch.authRequired`：自动使用存储的 `user`/`pass` 响应，然后关闭 Fetch 以免继续拦截。
+
+设计动机：
+
+- 防止凭证出现在命令日志和 CDP 参数中。
+- 将认证作用域限制在请求该代理的上下文中。
+- 在 Headed/Headless 场景下均可工作（认证流程在网络层，不依赖 UI）。
+
+流程（简化）：
+
+```python
+# 创建上下文
+context_id = await browser.create_browser_context(proxy_server='user:pwd@host:port')
+# => 发送 Target.createBrowserContext 时使用 'http://host:port'
+# => 内部存储 {'context_id': ('user', 'pwd')}
+
+# 在该上下文打开第一个 Tab
+tab = await browser.new_tab(browser_context_id=context_id)
+# => tab.enable_fetch_events(handle_auth=True)
+# => tab.on('Fetch.requestPaused', continue_request)
+# => tab.on('Fetch.authRequired', continue_with_auth(user, pwd))
+```
+
 ### 创建与管理上下文
 
 ```python

pydoll/browser/chromium/base.py

@@ -9,6 +9,7 @@
 from random import randint
 from tempfile import TemporaryDirectory
 from typing import Any, Awaitable, Callable, Optional, overload
+from urllib.parse import urlsplit, urlunsplit
 
 from pydoll.browser.interfaces import BrowserOptionsManager
 from pydoll.browser.managers import (
@@ -93,6 +94,7 @@ def __init__(
         self._connection_handler = ConnectionHandler(self._connection_port)
         self._backup_preferences_dir = ''
         self._tabs_opened: dict[str, Tab] = {}
+        self._context_proxy_auth: dict[str, tuple[str, str]] = {}
 
     async def __aenter__(self) -> 'Browser':
         """Async context manager entry."""
@@ -203,13 +205,22 @@ async def create_browser_context(
         Returns:
             Browser context ID for use with other methods.
         """
+        # If proxy_server contains credentials, strip them and store per-context auth
+        sanitized_proxy = proxy_server
+        extracted_auth: Optional[tuple[str, str]] = None
+        if proxy_server:
+            sanitized_proxy, extracted_auth = self._sanitize_proxy_and_extract_auth(proxy_server)
+
         response: CreateBrowserContextResponse = await self._execute_command(
             TargetCommands.create_browser_context(
-                proxy_server=proxy_server,
+                proxy_server=sanitized_proxy,
                 proxy_bypass_list=proxy_bypass_list,
             )
         )
-        return response['result']['browserContextId']
+        context_id = response['result']['browserContextId']
+        if extracted_auth:
+            self._context_proxy_auth[context_id] = extracted_auth
+        return context_id
 
     async def delete_browser_context(self, browser_context_id: str):
         """
@@ -251,8 +262,8 @@ async def new_tab(self, url: str = '', browser_context_id: Optional[str] = None)
         target_id = response['result']['targetId']
         tab = Tab(self, **self._get_tab_kwargs(target_id, browser_context_id))
         self._tabs_opened[target_id] = tab
-        if url:
-            await tab.go_to(url)
+        await self._setup_context_proxy_auth_for_tab(tab, browser_context_id)
+        if url: await tab.go_to(url)
         return tab
 
     async def get_targets(self) -> list[TargetInfo]:
@@ -577,6 +588,60 @@ async def _continue_request_with_auth_callback(
         await self.disable_fetch_events()
         return response
 
+    @staticmethod
+    async def _tab_continue_request_callback(event: RequestPausedEvent, tab: Tab):
+        """Internal callback to continue paused requests at Tab level."""
+        request_id = event['params']['requestId']
+        return await tab.continue_request(request_id)
+
+    @staticmethod
+    async def _tab_continue_request_with_auth_callback(
+        event: RequestPausedEvent,
+        tab: Tab,
+        proxy_username: Optional[str],
+        proxy_password: Optional[str],
+    ):
+        """Internal callback for proxy/server authentication at Tab level."""
+        request_id = event['params']['requestId']
+        response: Response = await tab.continue_with_auth(
+            request_id=request_id,
+            auth_challenge_response=AuthChallengeResponseType.PROVIDE_CREDENTIALS,
+            proxy_username=proxy_username,
+            proxy_password=proxy_password,
+        )
+        await tab.disable_fetch_events()
+        return response
+
+    async def _setup_context_proxy_auth_for_tab(
+        self, tab: Tab, browser_context_id: Optional[str]
+    ) -> None:
+        """Enable proxy auth handling for a Tab if its context has credentials stored."""
+        if not browser_context_id:
+            return
+        creds = self._context_proxy_auth.get(browser_context_id)
+        if not creds:
+            return
+        username, password = creds
+        await tab.enable_fetch_events(handle_auth=True)
+        await tab.on(
+            FetchEvent.REQUEST_PAUSED,
+            partial(
+                self._tab_continue_request_callback,
+                tab=tab,
+            ),
+            temporary=True,
+        )
+        await tab.on(
+            FetchEvent.AUTH_REQUIRED,
+            partial(
+                self._tab_continue_request_with_auth_callback,
+                tab=tab,
+                proxy_username=username,
+                proxy_password=password,
+            ),
+            temporary=True,
+        )
+
     async def _verify_browser_running(self):
         """
         Verify browser started successfully.
@@ -763,6 +828,49 @@ def _get_tab_ws_address(self, tab_id: str) -> str:
         ws_domain = '/'.join(self._ws_address.split('/')[:3])
         return f'{ws_domain}/devtools/page/{tab_id}'
 
+    @staticmethod
+    def _sanitize_proxy_and_extract_auth(
+        proxy_server: str,
+    ) -> tuple[str, Optional[tuple[str, str]]]:
+        """Strip credentials from a proxy URL and return sanitized URL plus (user, pass).
+
+        Accepts inputs like:
+        - username:password@host:port
+        - http://username:password@host:port
+        - socks5://username:password@host:port
+        - host:port (no credentials)
+        Returns a (sanitized_proxy, (user, pass) | None).
+        Ensures scheme is present in the sanitized URL (defaults to http).
+        """
+        base = proxy_server if '://' in proxy_server else f'http://{proxy_server}'
+        parts = urlsplit(base)
+        netloc = parts.netloc
+        creds: Optional[tuple[str, str]] = None
+        if '@' in netloc:
+            cred_part, host_part = netloc.split('@', 1)
+            if ':' in cred_part:
+                user, pwd = cred_part.split(':', 1)
+            else:
+                user, pwd = cred_part, ''
+            creds = (user, pwd)
+            sanitized = urlunsplit((
+                parts.scheme,
+                host_part,
+                parts.path,
+                parts.query,
+                parts.fragment,
+            ))
+        else:
+            # No creds; ensure scheme
+            sanitized = urlunsplit((
+                parts.scheme,
+                parts.netloc,
+                parts.path,
+                parts.query,
+                parts.fragment,
+            ))
+        return sanitized, creds
+
     @abstractmethod
     def _get_default_binary_location(self) -> str:
         """Get default browser executable path (implemented by subclasses)."""

pydoll/browser/tab.py

@@ -56,7 +56,7 @@
     DownloadWillBeginEvent,
 )
 from pydoll.protocol.browser.types import DownloadBehavior, DownloadProgressState
-from pydoll.protocol.fetch.types import HeaderEntry, RequestStage
+from pydoll.protocol.fetch.types import AuthChallengeResponseType, HeaderEntry, RequestStage
 from pydoll.protocol.network.events import RequestWillBeSentEvent
 from pydoll.protocol.network.types import (
     Cookie,
@@ -709,6 +709,27 @@ async def fulfill_request(
             )
         )
 
+    async def continue_with_auth(
+        self,
+        request_id: str,
+        auth_challenge_response: AuthChallengeResponseType,
+        proxy_username: Optional[str] = None,
+        proxy_password: Optional[str] = None,
+    ):
+        """Continue a paused request replying to an authentication challenge.
+
+        Useful for proxy auth (407) or server auth (401) when Fetch is enabled
+        with handle_auth=True.
+        """
+        return await self._execute_command(
+            FetchCommands.continue_request_with_auth(
+                request_id=request_id,
+                auth_challenge_response=auth_challenge_response,
+                proxy_username=proxy_username,
+                proxy_password=proxy_password,
+            )
+        )
+
     @asynccontextmanager
     async def expect_file_chooser(
         self, files: Union[str, Path, list[Union[str, Path]]]