Skip to content

Commit 438e272

Browse files
author
arman-bd
committed
wip
1 parent c156360 commit 438e272

File tree

12 files changed

+8747
-140
lines changed

12 files changed

+8747
-140
lines changed

.github/workflows/_benchmark.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ jobs:
3232
3333
- name: Install system dependencies (macOS)
3434
if: runner.os == 'macOS'
35-
run: brew install cmake ninja openssl@3 libnghttp2
35+
run: brew install cmake ninja libnghttp2
3636

3737
- name: Cache vendor dependencies
3838
uses: actions/cache@v4

.github/workflows/_test.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ jobs:
8888
- name: Install dependencies via vcpkg (Windows)
8989
if: runner.os == 'Windows'
9090
run: |
91-
vcpkg install openssl:x64-windows nghttp2:x64-windows zlib:x64-windows --clean-after-build
91+
vcpkg install nghttp2:x64-windows zlib:x64-windows --clean-after-build
9292
shell: bash
9393

9494
- name: Cache vendor dependencies

README.md

Lines changed: 63 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,9 @@ pip install httpmorph
2020

2121
### Requirements
2222

23-
- Python 3.7+
24-
- macOS or Linux (Windows support in progress)
25-
- OpenSSL/BoringSSL
23+
- Python 3.8+
24+
- Windows, macOS, or Linux
25+
- BoringSSL (built automatically from source)
2626
- libnghttp2 (for HTTP/2)
2727

2828
## Quick Start
@@ -142,17 +142,17 @@ httpmorph aims for high compatibility with Python's `requests` library:
142142

143143
| Feature | Status |
144144
|---------|--------|
145-
| GET, POST, PUT, DELETE, HEAD, PATCH, OPTIONS | Supported |
146-
| JSON request/response | Supported |
147-
| Form data & file uploads | Supported |
148-
| Custom headers | Supported |
149-
| Authentication | Supported |
150-
| Cookies & sessions | Supported |
151-
| Redirects with history | Supported |
152-
| Timeout control | Supported |
153-
| SSL verification | Supported |
154-
| Streaming responses | Supported |
155-
| Exception hierarchy | Supported |
145+
| GET, POST, PUT, DELETE, HEAD, PATCH, OPTIONS | Supported |
146+
| JSON request/response | Supported |
147+
| Form data & file uploads | Supported |
148+
| Custom headers | Supported |
149+
| Authentication | Supported |
150+
| Cookies & sessions | Supported |
151+
| Redirects with history | Supported |
152+
| Timeout control | Supported |
153+
| SSL verification | Supported |
154+
| Streaming responses | Supported |
155+
| Exception hierarchy | Supported |
156156

157157
## Response Object
158158

@@ -218,54 +218,85 @@ Benchmarks show httpmorph matching or exceeding the performance of curl and othe
218218

219219
| Platform | Status |
220220
|----------|--------|
221-
| macOS (Intel & Apple Silicon) | Fully supported |
222-
| Linux (x86_64, ARM64) | Fully supported |
223-
| Windows | In progress |
221+
| Windows | Fully supported |
222+
| macOS (Intel & Apple Silicon) | Fully supported |
223+
| Linux (x86_64, ARM64) | ✅ Fully supported |
224224

225-
Windows support is actively being developed. Follow the [GitHub issues](https://github.com/anthropics/httpmorph/issues) for updates.
225+
All platforms use **BoringSSL** (Google's fork of OpenSSL) for consistent TLS behavior and advanced fingerprinting capabilities.
226226

227227
## Building from Source
228228

229-
### macOS
229+
httpmorph uses **BoringSSL** (built from source) on all platforms for consistent TLS fingerprinting.
230230

231+
### Prerequisites
232+
233+
**Windows:**
234+
```bash
235+
# Install build tools
236+
choco install cmake golang nasm visualstudio2022buildtools -y
237+
238+
# Or install manually:
239+
# - Visual Studio 2019+ (with C++ tools)
240+
# - CMake 3.15+
241+
# - Go 1.18+
242+
# - NASM (for BoringSSL assembly optimizations)
243+
```
244+
245+
**macOS:**
231246
```bash
232247
# Install dependencies
233-
brew install openssl@3 libnghttp2
248+
brew install cmake ninja libnghttp2
249+
```
234250

235-
# Build and install
236-
python setup.py build_ext --inplace
237-
pip install -e .
251+
**Linux:**
252+
```bash
253+
# Ubuntu/Debian
254+
sudo apt-get install cmake ninja-build libssl-dev pkg-config autoconf automake libtool
255+
256+
# Fedora/RHEL
257+
sudo dnf install cmake ninja-build openssl-devel pkg-config autoconf automake libtool
238258
```
239259

240-
### Linux
260+
### Build Steps
241261

242262
```bash
243-
# Install dependencies (Ubuntu/Debian)
244-
sudo apt-get install libssl-dev libnghttp2-dev
263+
# 1. Clone the repository
264+
git clone https://github.com/yourusername/httpmorph.git
265+
cd httpmorph
245266

246-
# Or (Fedora/RHEL)
247-
sudo dnf install openssl-devel libnghttp2-devel
267+
# 2. Build vendor dependencies (BoringSSL, nghttp2, zlib)
268+
./scripts/setup_vendors.sh # On Windows: bash scripts/setup_vendors.sh
248269

249-
# Build and install
270+
# 3. Build Python extensions
250271
python setup.py build_ext --inplace
251-
pip install -e .
272+
273+
# 4. Install in development mode
274+
pip install -e ".[dev]"
252275
```
253276

277+
**Note:** The first build takes 5-10 minutes as it compiles BoringSSL from source. Subsequent builds are much faster (~30 seconds) as the vendor libraries are cached.
278+
254279
## Development
255280

256281
```bash
257-
# Clone repository
282+
# Clone and setup (includes building BoringSSL)
258283
git clone https://github.com/yourusername/httpmorph.git
259284
cd httpmorph
285+
./scripts/setup_vendors.sh
260286

261287
# Install development dependencies
262288
pip install -e ".[dev]"
263289

264290
# Run tests
265-
pytest tests/
291+
pytest tests/ -v
266292

267293
# Run with coverage
268294
pytest tests/ --cov=httpmorph --cov-report=html
295+
296+
# Run specific test markers
297+
pytest tests/ -m "not slow" # Skip slow tests
298+
pytest tests/ -m integration # Only integration tests
299+
pytest tests/ -m fingerprint # Only fingerprinting tests
269300
```
270301

271302
## Architecture
@@ -310,10 +341,6 @@ Run the test suite:
310341
pytest tests/ -v
311342
```
312343

313-
## License
314-
315-
[License information to be added]
316-
317344
## Acknowledgments
318345

319346
- Built on BoringSSL (Google)

include/boringssl_compat.h

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
/**
2+
* boringssl_compat.h - Windows compatibility layer for BoringSSL
3+
*
4+
* This header must be included before any BoringSSL headers on Windows
5+
* to avoid conflicts with Windows Cryptography API (wincrypt.h) macros.
6+
*/
7+
8+
#ifndef BORINGSSL_COMPAT_H
9+
#define BORINGSSL_COMPAT_H
10+
11+
#ifdef _WIN32
12+
13+
/* Windows headers included by Python.h may define these macros
14+
* which conflict with BoringSSL type names. Undefine them before
15+
* including BoringSSL headers. */
16+
#ifdef X509_NAME
17+
#undef X509_NAME
18+
#endif
19+
20+
#ifdef X509_CERT_PAIR
21+
#undef X509_CERT_PAIR
22+
#endif
23+
24+
#ifdef X509_EXTENSIONS
25+
#undef X509_EXTENSIONS
26+
#endif
27+
28+
#ifdef PKCS7_SIGNER_INFO
29+
#undef PKCS7_SIGNER_INFO
30+
#endif
31+
32+
#ifdef OCSP_REQUEST
33+
#undef OCSP_REQUEST
34+
#endif
35+
36+
#ifdef OCSP_RESPONSE
37+
#undef OCSP_RESPONSE
38+
#endif
39+
40+
#endif /* _WIN32 */
41+
42+
#endif /* BORINGSSL_COMPAT_H */

scripts/setup_vendors.sh

Lines changed: 48 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -40,60 +40,70 @@ echo "Detected OS: $OS"
4040
echo ""
4141

4242
#
43-
# 1. BoringSSL / OpenSSL
43+
# 1. BoringSSL
4444
#
45-
echo "==> Setting up SSL library..."
45+
echo "==> Setting up BoringSSL..."
4646

47-
# On Windows, use vcpkg's OpenSSL instead of building BoringSSL
48-
if [ "$OS" = "Windows" ] && [ -n "$VCPKG_ROOT" ] && [ -d "$VCPKG_ROOT/installed/x64-windows" ]; then
49-
echo "✓ Using vcpkg's OpenSSL (Windows)"
50-
echo " Location: $VCPKG_ROOT/installed/x64-windows"
51-
else
52-
# On Linux/macOS, build BoringSSL
53-
if [ ! -d "boringssl" ]; then
54-
echo "Cloning BoringSSL..."
55-
git clone --depth 1 https://boringssl.googlesource.com/boringssl
56-
fi
47+
# Build BoringSSL on all platforms (including Windows)
48+
if [ ! -d "boringssl" ]; then
49+
echo "Cloning BoringSSL..."
50+
git clone --depth 1 https://boringssl.googlesource.com/boringssl
51+
fi
5752

58-
cd boringssl
53+
cd boringssl
5954

60-
# Check if already built
61-
if [ ! -f "build/ssl/libssl.a" ]; then
62-
echo "Building BoringSSL..."
55+
# Check if already built (Windows uses .lib files instead of .a)
56+
if [ "$OS" = "Windows" ]; then
57+
SSL_LIB_FILE="build/ssl/ssl.lib"
58+
else
59+
SSL_LIB_FILE="build/ssl/libssl.a"
60+
fi
6361

64-
# BoringSSL requires CMake and Go
65-
if ! command -v cmake &> /dev/null; then
66-
echo "ERROR: cmake not found. Please install cmake."
67-
exit 1
68-
fi
62+
if [ ! -f "$SSL_LIB_FILE" ]; then
63+
echo "Building BoringSSL..."
6964

70-
if ! command -v go &> /dev/null; then
71-
echo "WARNING: go not found. BoringSSL will build without some tests."
72-
fi
65+
# BoringSSL requires CMake and Go
66+
if ! command -v cmake &> /dev/null; then
67+
echo "ERROR: cmake not found. Please install cmake."
68+
exit 1
69+
fi
70+
71+
if ! command -v go &> /dev/null; then
72+
echo "WARNING: go not found. BoringSSL will build without some tests."
73+
fi
7374

74-
mkdir -p build
75-
cd build
75+
mkdir -p build
76+
cd build
7677

78+
# Platform-specific CMake configuration
79+
if [ "$OS" = "Windows" ]; then
80+
# Use Visual Studio generator on Windows for proper .lib output
81+
cmake -DCMAKE_BUILD_TYPE=Release \
82+
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
83+
-DBUILD_SHARED_LIBS=OFF \
84+
..
85+
cmake --build . --config Release
86+
else
7787
cmake -DCMAKE_BUILD_TYPE=Release \
7888
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
7989
..
8090
make -j$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
91+
fi
8192

82-
echo "✓ BoringSSL built successfully"
93+
echo "✓ BoringSSL built successfully"
8394

84-
# Clean up .git directory to save cache space
85-
cd "$VENDOR_DIR/boringssl"
86-
if [ -d ".git" ]; then
87-
echo "Cleaning up .git directory to save cache space..."
88-
rm -rf .git
89-
fi
90-
else
91-
echo "✓ BoringSSL already built"
95+
# Clean up .git directory to save cache space
96+
cd "$VENDOR_DIR/boringssl"
97+
if [ -d ".git" ]; then
98+
echo "Cleaning up .git directory to save cache space..."
99+
rm -rf .git
92100
fi
93-
94-
cd "$VENDOR_DIR"
101+
else
102+
echo "✓ BoringSSL already built"
95103
fi
96104

105+
cd "$VENDOR_DIR"
106+
97107
#
98108
# 2. liburing (Linux only)
99109
#
@@ -203,11 +213,7 @@ echo "Vendor Setup Complete!"
203213
echo "================================"
204214
echo ""
205215
echo "Built libraries:"
206-
if [ "$OS" = "Windows" ] && [ -n "$VCPKG_ROOT" ] && [ -d "$VCPKG_ROOT/installed/x64-windows" ]; then
207-
echo " ✓ OpenSSL: $VCPKG_ROOT/installed/x64-windows (vcpkg)"
208-
else
209-
echo " ✓ BoringSSL: $VENDOR_DIR/boringssl/build"
210-
fi
216+
echo " ✓ BoringSSL: $VENDOR_DIR/boringssl/build"
211217
if [ "$OS" = "Linux" ]; then
212218
echo " ✓ liburing: $VENDOR_DIR/liburing/install"
213219
fi

0 commit comments

Comments
 (0)