Skip to content

Commit 196ca0b

Browse files
IdanvilenskiIdanvilenski
committed
Complete Python SDK restructure and packaging
- Restructure project as proper Python package 'brightdata' - Add comprehensive packaging configuration (pyproject.toml, setup.py) - Implement modular architecture with api/, utils/, exceptions/ modules - Add GitHub Actions workflows for testing and automated publishing - Include examples and comprehensive documentation - Set up proper distribution files for PyPI publishing - Remove old demo files and restructure codebase - Add comprehensive error handling and logging system - Support for both scraping and search APIs with parallel processing
1 parent 3c80c32 commit 196ca0b

31 files changed

+2201
-613
lines changed

‎.github/workflows/publish.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: Build and Publish
2+
3+
on:
4+
push:
5+
tags:
6+
- 'v*'
7+
release:
8+
types: [published]
9+
workflow_dispatch:
10+
11+
jobs:
12+
build:
13+
runs-on: ubuntu-latest
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Set up Python
19+
uses: actions/setup-python@v4
20+
with:
21+
python-version: '3.8'
22+
23+
- name: Install dependencies
24+
run: |
25+
python -m pip install --upgrade pip
26+
pip install build twine
27+
pip install -r requirements.txt
28+
29+
- name: Build package
30+
run: python -m build
31+
32+
- name: Upload build artifacts
33+
uses: actions/upload-artifact@v3
34+
with:
35+
name: dist-files
36+
path: dist/
37+
38+
- name: Publish to PyPI
39+
if: github.event_name == 'release'
40+
env:
41+
TWINE_USERNAME: __token__
42+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
43+
run: |
44+
twine upload dist/*
45+
46+
test-install:
47+
runs-on: ubuntu-latest
48+
needs: build
49+
50+
steps:
51+
- name: Set up Python
52+
uses: actions/setup-python@v4
53+
with:
54+
python-version: '3.8'
55+
56+
- name: Download build artifacts
57+
uses: actions/download-artifact@v3
58+
with:
59+
name: dist-files
60+
path: dist/
61+
62+
- name: Test wheel installation
63+
run: |
64+
pip install dist/*.whl
65+
python -c "import brightdata; print('✅ Package imported successfully')"

‎.github/workflows/test.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: Tests
2+
3+
on:
4+
push:
5+
branches: [ main, develop ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
strategy:
13+
matrix:
14+
python-version: ['3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
15+
16+
steps:
17+
- uses: actions/checkout@v4
18+
19+
- name: Set up Python ${{ matrix.python-version }}
20+
uses: actions/setup-python@v4
21+
with:
22+
python-version: ${{ matrix.python-version }}
23+
24+
- name: Install dependencies
25+
run: |
26+
python -m pip install --upgrade pip
27+
pip install -r requirements.txt
28+
pip install pytest pytest-cov
29+
30+
- name: Test package import
31+
run: |
32+
python -c "import brightdata; print('✅ Import successful')"
33+
34+
- name: Run tests
35+
run: |
36+
python -m pytest tests/ -v --cov=brightdata --cov-report=xml
37+
38+
- name: Upload coverage to Codecov
39+
if: matrix.python-version == '3.8'
40+
uses: codecov/codecov-action@v3
41+
with:
42+
file: ./coverage.xml

‎.gitignore‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
pip-wheel-metadata/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
*.manifest
32+
*.spec
33+
34+
# Installer logs
35+
pip-log.txt
36+
pip-delete-this-directory.txt
37+
38+
# Unit test / coverage reports
39+
htmlcov/
40+
.tox/
41+
.nox/
42+
.coverage
43+
.coverage.*
44+
.cache
45+
nosetests.xml
46+
coverage.xml
47+
*.cover
48+
*.py,cover
49+
.hypothesis/
50+
.pytest_cache/
51+
52+
# Translations
53+
*.mo
54+
*.pot
55+
56+
# Django stuff:
57+
*.log
58+
local_settings.py
59+
db.sqlite3
60+
db.sqlite3-journal
61+
62+
# Flask stuff:
63+
instance/
64+
.webassets-cache
65+
66+
# Scrapy stuff:
67+
.scrapy
68+
69+
# Sphinx documentation
70+
docs/_build/
71+
72+
# PyBuilder
73+
target/
74+
75+
# Jupyter Notebook
76+
.ipynb_checkpoints
77+
78+
# IPython
79+
profile_default/
80+
ipython_config.py
81+
82+
# pyenv
83+
.python-version
84+
85+
# pipenv
86+
Pipfile.lock
87+
88+
# PEP 582
89+
__pypackages__/
90+
91+
# Celery stuff
92+
celerybeat-schedule
93+
celerybeat.pid
94+
95+
# SageMath parsed files
96+
*.sage.py
97+
98+
# Environments
99+
.env
100+
.venv
101+
env/
102+
venv/
103+
ENV/
104+
env.bak/
105+
venv.bak/
106+
107+
# Spyder project settings
108+
.spyderproject
109+
.spyproject
110+
111+
# Rope project settings
112+
.ropeproject
113+
114+
# mkdocs documentation
115+
/site
116+
117+
# mypy
118+
.mypy_cache/
119+
.dmypy.json
120+
dmypy.json
121+
122+
# Pyre type checker
123+
.pyre/
124+
125+
# IDE
126+
.vscode/
127+
.idea/
128+
*.swp
129+
*.swo
130+
*~
131+
132+
# OS
133+
.DS_Store
134+
Thumbs.db

‎.pypirc.example‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
[distutils]
2+
index-servers =
3+
pypi
4+
testpypi
5+
6+
[pypi]
7+
repository: https://upload.pypi.org/legacy/
8+
username: __token__
9+
password: pypi-your-api-token-here
10+
11+
[testpypi]
12+
repository: https://test.pypi.org/legacy/
13+
username: __token__
14+
password: pypi-your-test-api-token-here

‎CHANGELOG.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [1.0.0] - 2024-08-10
9+
10+
### Added
11+
- Initial release of Bright Data Python SDK
12+
- Web scraping functionality using Bright Data Web Unlocker API
13+
- Search engine results using Bright Data SERP API
14+
- Support for multiple search engines (Google, Bing, Yandex)
15+
- Parallel processing for multiple URLs and queries
16+
- Comprehensive error handling with retry logic
17+
- Input validation for URLs, zones, and parameters
18+
- Automatic zone creation and management
19+
- Multiple output formats (JSON, raw HTML, markdown)
20+
- Content download functionality
21+
- Zone management utilities
22+
- Comprehensive logging system
23+
- Built-in connection pooling
24+
- Environment variable configuration support
25+
26+
### Features
27+
- `bdclient` main client class
28+
- `scrape()` method for web scraping
29+
- `search()` method for SERP API
30+
- `download_content()` for saving results
31+
- `list_zones()` for zone management
32+
- Automatic retry with exponential backoff
33+
- Structured logging support
34+
- Configuration via environment variables or direct parameters
35+
36+
### Dependencies
37+
- `requests>=2.25.0`
38+
- `python-dotenv>=0.19.0`
39+
40+
### Python Support
41+
- Python 3.7+
42+
- Cross-platform compatibility (Windows, macOS, Linux)

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2025 Bright Data
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎MANIFEST.in‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
include README.md
2+
include LICENSE
3+
include requirements.txt
4+
recursive-include brightdata *.py
5+
recursive-exclude * __pycache__
6+
recursive-exclude * *.py[co]

0 commit comments

Comments
 (0)