Merge branch 'main' into feat/tab-request-api-issue-171

Author: thalissonvs

.github/ISSUE_TEMPLATE/config.yml

@@ -1,4 +1,4 @@
-blank_issues_enabled: false
+blank_issues_enabled: true
 contact_links:
   - name: Questions & Discussions
     url: https://github.com/thalissonvs/pydoll/discussions

CHANGELOG.md

@@ -1,3 +1,26 @@
+## 2.4.0 (2025-08-01)
+
+### Feat
+
+- changing bool prefs to properties and adding support to user-data-dir preferences
+- adding prefs options customization
+- add overloads for find and query methods in FindElementsMixin
+- add method to retrieve parent element and its attributes
+- implements start_timeout option
+
+### Fix
+
+- adding typehint and fixing some codes
+- removing options preferences private attributes
+- set default URL to 'about:blank' in create_target method
+- change navigation when creating a new tab
+- add type hinting support and update project description
+
+### Refactor
+
+- remove redundant asterisk from find method overloads and reorganize query method overloads
+- refine type hint for response parameter and improve key check
+
 ## 2.3.1 (2025-07-12)
 
 ### Fix

README.md

@@ -4,6 +4,7 @@
 <h1 align="center">Pydoll: Automate the Web, Naturally</h1>
 
 <p align="center">
+    <a href="https://github.com/autoscrape-labs/pydoll/stargazers"><img src="https://img.shields.io/github/stars/autoscrape-labs/pydoll?style=social"></a>
     <a href="https://codecov.io/gh/autoscrape-labs/pydoll" >
         <img src="https://codecov.io/gh/autoscrape-labs/pydoll/graph/badge.svg?token=40I938OGM9"/>
     </a>
@@ -31,15 +32,64 @@ Built from scratch with a different philosophy, Pydoll connects directly to the
 
 We believe that powerful automation shouldn't require you to become an expert in configuration or constantly fight with bot protection systems. With Pydoll, you can focus on what really matters: your automation logic, not the underlying complexity or protection systems.
 
+<div>
+  <h4>Be a good human. Give it a star ⭐</h4> 
+    No stars, no bugs fixed. Just kidding (maybe)
+</div>
+
 ## 🌟 What makes Pydoll special?
 
 - **Zero Webdrivers**: Say goodbye to webdriver compatibility issues
-- **Native Captcha Bypass**: Handles Cloudflare Turnstile and reCAPTCHA v3*
+- **Human-like Interaction Engine**: Capable of passing behavioral CAPTCHAs like reCAPTCHA v3 or Turnstile, depending on IP reputation and interaction patterns
 - **Asynchronous Performance**: For high-speed automation and multiple simultaneous tasks
 - **Humanized Interactions**: Mimic real user behavior
 - **Simplicity**: With Pydoll, you install and you're ready to automate.
 
->⚠️ The effectiveness of captcha bypass depends on various factors, such as IP address reputation. Pydoll can achieve scores comparable to real users, but cannot handle restrictive configurations or IP blocks.
+## What's New in 2.4.0
+
+### Advanced browser preferences support (thanks to [@LucasAlvws](https://github.com/LucasAlvws))
+You can now customize Chromium browser preferences through the `browser_preferences` dict in ChromiumOptions.<br><br>
+Set things like download directory, language, notification blocking, PDF handling, and more.
+Helper properties like `set_default_download_directory`, `set_accept_languages`, and `prompt_for_download` were added for convenience.
+Preferences are merged automatically, no need to redefine everything.<br><br>
+Here's an example:
+
+```python
+options = ChromiumOptions()
+options.browser_preferences = {  # you can set the entire dict
+    'download': {
+        'default_directory': '/tmp/downloads',
+        'prompt_for_download': False
+    },
+    'intl': {
+        'accept_languages': 'en-US,en,pt-BR'
+    },
+    'profile': {
+        'default_content_setting_values': {
+            'notifications': 2  # Block notifications
+        }
+    }
+}
+
+options.set_default_download_directory('/tmp/downloads')   # or just the individual properties
+options.set_accept_languages('en-US,en,pt-BR')
+options.prompt_for_download = False
+```
+See [docs/features.md](docs/features.md#custom-browser-preferences) for more details.
+
+### New `get_parent_element()` method
+Retrieve the parent of any WebElement, making it easier to navigate the DOM structure:
+```python
+element = await tab.find(id='button')
+parent = await element.get_parent_parent()
+```
+### New start_timeout option (thanks to [@j0j1j2](https://github.com/j0j1j2))
+Added to ChromiumOptions to control how long the browser can take to start. Useful on slower machines or CI environments.
+
+```python
+options = ChromiumOptions()
+options.start_timeout = 20  # wait 20 seconds
+```
 
 ## 📦 Installation
 
@@ -360,13 +410,17 @@ Please make sure to:
 If you find Pydoll useful, consider [supporting me on GitHub](https://github.com/sponsors/thalissonvs).  
 You'll get access to exclusive benefits like priority support, custom features and much more!
 
-Can't sponsor right now? No problem — you can still help a lot by:
+Can't sponsor right now? No problem, you can still help a lot by:
 - Starring the repository
 - Sharing on social media
 - Writing posts or tutorials
 - Giving feedback or reporting issues
 
-Every bit of support makes a difference — thank you!
+Every bit of support makes a difference/
+
+## 💬 Spread the word
+
+If Pydoll saved you time, mental health, or a keyboard from being smashed, give it a ⭐, share it, or tell your weird dev friends.
 
 ## 📄 License
 

cz.yaml

@@ -2,4 +2,4 @@
 commitizen:
   name: cz_conventional_commits
   tag_format: $version
-  version: 2.3.1
+  version: 2.4.0

docs/deep-dive/browser-preferences.md

@@ -0,0 +1,36 @@
+# Deep Dive: Custom Browser Preferences in Pydoll
+
+## Overview
+The `browser_preferences` feature (PR #204) enables direct, fine-grained control over Chromium browser settings via the `ChromiumOptions` API. This is essential for advanced automation, testing, and scraping scenarios where default browser behavior must be customized.
+
+## How It Works
+- `ChromiumOptions.browser_preferences` is a dictionary that maps directly to Chromium's internal preferences structure.
+- Preferences are merged: setting new keys updates only those keys, preserving others.
+- Helper methods (`set_default_download_directory`, `set_accept_languages`, etc.) are provided for common scenarios.
+- Preferences are applied before browser launch, ensuring all settings take effect from the start of the session.
+- Validation ensures only dictionaries are accepted; invalid structures raise clear errors.
+
+## Example
+```python
+options = ChromiumOptions()
+options.browser_preferences = {
+    'download': {'default_directory': '/tmp', 'prompt_for_download': False},
+    'intl': {'accept_languages': 'en-US,en'},
+    'profile': {'default_content_setting_values': {'notifications': 2}}
+}
+```
+
+## Advanced Usage
+- **Merging:** Multiple assignments merge keys, so you can incrementally build your preferences.
+- **Validation:** If you pass a non-dict or use the reserved 'prefs' key, an error is raised.
+- **Internals:** Preferences are set via a recursive setter that creates nested dictionaries as needed.
+- **Integration:** Used by the browser process manager to initialize the user data directory with your custom settings.
+
+## Best Practices
+- Use helper methods for common patterns; set `browser_preferences` directly for advanced needs.
+- Check Chromium documentation for available preferences: https://chromium.googlesource.com/chromium/src/+/4aaa9f29d8fe5eac55b8632fa8fcb05a68d9005b/chrome/common/pref_names.cc
+- Avoid setting experimental or undocumented preferences unless you know their effects.
+
+## References
+- See `pydoll/browser/options.py` for implementation details.
+- See tests in `tests/test_browser/test_browser_chrome.py` for usage examples.

docs/features.md

@@ -662,6 +662,55 @@ These network analysis capabilities make Pydoll ideal for:
 - **Debugging**: Identify failed requests and network issues
 - **Security Testing**: Analyze request/response patterns
 
+## Custom Browser Preferences
+
+Pydoll now supports advanced browser customization through the `ChromiumOptions.browser_preferences` property. This allows you to set any Chromium browser preference for your automation session.
+
+### What You Can Customize
+- Download directory and prompt behavior
+- Accepted languages
+- Notification blocking
+- PDF handling
+- Any other Chromium-supported preference
+
+### Example: Setting Preferences
+```python
+from pydoll.browser.chromium import Chrome
+from pydoll.browser.options import ChromiumOptions
+
+options = ChromiumOptions()
+options.browser_preferences = {
+    'download': {
+        'default_directory': '/tmp/downloads',
+        'prompt_for_download': False
+    },
+    'intl': {
+        'accept_languages': 'en-US,en,pt-BR'
+    },
+    'profile': {
+        'default_content_setting_values': {
+            'notifications': 2  # Block notifications
+        }
+    }
+}
+
+# Helper methods for common preferences
+options.set_default_download_directory('/tmp/downloads')
+options.set_accept_languages('en-US,en,pt-BR')
+options.prompt_for_download = False
+
+browser = Chrome(options=options)
+```
+
+You can call `browser_preferences` multiple times—new keys will be merged, not replaced.
+
+### Why is this useful?
+- Fine-grained control for scraping, testing, or automation
+- Avoid popups or unwanted prompts
+- Match user locale or automate downloads
+
+---
+
 ## File Upload Support
 
 Seamlessly handle file uploads in your automation:

pydoll/browser/chromium/base.py

@@ -1,7 +1,12 @@
 import asyncio
+import json
+import os
+import shutil
 from abc import ABC, abstractmethod
+from contextlib import suppress
 from functools import partial
 from random import randint
+from tempfile import TemporaryDirectory
 from typing import Any, Callable, Optional
 
 from pydoll.browser.interfaces import BrowserOptionsManager
@@ -77,13 +82,20 @@ def __init__(
         self._browser_process_manager = BrowserProcessManager()
         self._temp_directory_manager = TempDirectoryManager()
         self._connection_handler = ConnectionHandler(self._connection_port)
+        self._backup_preferences_dir = ''
 
     async def __aenter__(self) -> 'Browser':
         """Async context manager entry."""
         return self
 
     async def __aexit__(self, exc_type, exc_val, exc_tb):
         """Async context manager exit with cleanup."""
+        if self._backup_preferences_dir:
+            user_data_dir = self._get_user_data_dir()
+            shutil.copy2(
+                self._backup_preferences_dir,
+                os.path.join(user_data_dir, 'Default', 'Preferences'),
+            )
         if await self._is_browser_running(timeout=2):
             await self.stop()
 
@@ -113,9 +125,7 @@ async def start(self, headless: bool = False) -> Tab:
         proxy_config = self._proxy_manager.get_proxy_credentials()
 
         self._browser_process_manager.start_browser_process(
-            binary_location,
-            self._connection_port,
-            self.options.arguments,
+            binary_location, self._connection_port, self.options.arguments
         )
         await self._verify_browser_running()
         await self._configure_proxy(proxy_config[0], proxy_config[1])
@@ -199,12 +209,13 @@ async def new_tab(self, url: str = '', browser_context_id: Optional[str] = None)
         """
         response: CreateTargetResponse = await self._execute_command(
             TargetCommands.create_target(
-                url=url,
                 browser_context_id=browser_context_id,
             )
         )
         target_id = response['result']['targetId']
-        return Tab(self, self._connection_port, target_id, browser_context_id)
+        tab = Tab(self, self._connection_port, target_id, browser_context_id)
+        if url: await tab.go_to(url)
+        return tab
 
     async def get_targets(self) -> list[TargetInfo]:
         """
@@ -584,10 +595,60 @@ async def _execute_command(
 
     def _setup_user_dir(self):
         """Setup temporary user data directory if not specified in options."""
-        if '--user-data-dir' not in [arg.split('=')[0] for arg in self.options.arguments]:
-            # For all browsers, use a temporary directory
+        user_data_dir = self._get_user_data_dir()
+        if user_data_dir and self.options.browser_preferences:
+            self._set_browser_preferences_in_user_data_dir(user_data_dir)
+        elif not user_data_dir:
             temp_dir = self._temp_directory_manager.create_temp_dir()
+            # For all browsers, use a temporary directory
             self.options.arguments.append(f'--user-data-dir={temp_dir.name}')
+            if self.options.browser_preferences:
+                self._set_browser_preferences_in_temp_dir(temp_dir)
+
+    def _set_browser_preferences_in_temp_dir(self, temp_dir: TemporaryDirectory):
+        os.mkdir(os.path.join(temp_dir.name, 'Default'))
+        preferences = self.options.browser_preferences
+        with open(
+            os.path.join(temp_dir.name, 'Default', 'Preferences'), 'w', encoding='utf-8'
+        ) as json_file:
+            json.dump(preferences, json_file)
+
+    def _set_browser_preferences_in_user_data_dir(self, user_data_dir: str):
+        """
+        Set browser preferences in the user data directory.
+
+        This function will:
+        1. Create a backup of the existing Preferences file if it exists
+        2. Create Default directory if it doesn't exist
+        3. Write the new preferences to the Preferences file
+
+        Args:
+            user_data_dir: Path to the user data directory
+        """
+        default_dir = os.path.join(user_data_dir, 'Default')
+        os.makedirs(default_dir, exist_ok=True)
+
+        preferences_path = os.path.join(default_dir, 'Preferences')
+        self._backup_preferences_dir = os.path.join(default_dir, 'Preferences.backup')
+
+        if os.path.exists(preferences_path):
+            # Backup existing Preferences file
+            shutil.copy2(preferences_path, self._backup_preferences_dir)
+
+        preferences = {}
+        if os.path.exists(preferences_path):
+            with suppress(json.JSONDecodeError):
+                with open(preferences_path, 'r', encoding='utf-8') as preferences_file:
+                    preferences = json.load(preferences_file)
+        preferences.update(self.options.browser_preferences)
+        with open(preferences_path, 'w', encoding='utf-8') as json_file:
+            json.dump(preferences, json_file, indent=2)
+
+    def _get_user_data_dir(self) -> Optional[str]:
+        for arg in self.options.arguments:
+            if arg.startswith('--user-data-dir='):
+                return arg.split('=', 1)[1]
+        return None
 
     @abstractmethod
     def _get_default_binary_location(self) -> str:

pydoll/browser/interfaces.py

@@ -21,6 +21,11 @@ def start_timeout(self) -> int:
     def add_argument(self, argument: str):
         pass
 
+    @property
+    @abstractmethod
+    def browser_preferences(self) -> dict:
+        pass
+
 
 class BrowserOptionsManager(ABC):
     @abstractmethod