Skip to content

Commit 88b5c5f

Browse files
Improve benchmark with statistical analysis and fair comparison
Changes: - Move benchmark files to dedicated /benchmark directory - Add run_benchmark.sh script for easy execution with uvx - Improve benchmark.py with: - Statistical analysis: 10 runs with mean, median, std dev, min, max - Fair comparison: Both libraries test equivalent configuration complexity - System information: Display Python version, platform, library versions - Better warmup: 50 iterations before timing - Raw data output: All timing data for verification - Comprehensive documentation Benchmark improvements: - Test realistic configuration with 18 fields across multiple categories - msgspec-ext: Flat structure with __ delimiter (optimized for .env) - pydantic-settings: Nested structure with BaseSettings classes - Both load from identical .env file Updated README with accurate benchmark results: - 2.7x faster than pydantic-settings (down from 3.8x claim) - Based on 10 runs × 1000 iterations with statistical validation - More honest and reproducible results Run: ./benchmark/run_benchmark.sh to reproduce results 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 7e6ab1e commit 88b5c5f

File tree

4 files changed

+328
-179
lines changed

4 files changed

+328
-179
lines changed

README.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -142,26 +142,26 @@ class AppSettings(BaseSettings):
142142

143143
msgspec-ext leverages msgspec's high-performance serialization with bulk JSON decoding for maximum speed.
144144

145-
**Benchmark Results** (1000 iterations, Python 3.12):
145+
**Benchmark Results** (10 runs × 1000 iterations, Python 3.12):
146146

147147
| Library | Time per load | Relative Performance |
148148
|---------|---------------|---------------------|
149-
| msgspec-ext | 0.702ms | Baseline ⚡ |
150-
| pydantic-settings | 2.694ms | 3.8x slower |
149+
| msgspec-ext | 2.271ms | Baseline ⚡ |
150+
| pydantic-settings | 6.157ms | 2.7x slower |
151151

152-
msgspec-ext is **3.8x faster** than pydantic-settings while providing the same level of type safety and validation.
152+
msgspec-ext is **2.7x faster** than pydantic-settings while providing the same level of type safety and validation.
153153

154154
**Key optimizations:**
155155
- Bulk JSON decoding in C (via msgspec)
156156
- Cached encoders and decoders
157157
- Automatic field ordering
158158
- Zero Python loops for validation
159159

160-
*Benchmark measures complete settings initialization including .env file parsing and type validation. Run `python benchmark.py` to reproduce.*
160+
*Benchmark measures complete settings initialization with complex configuration (app settings, database, redis, feature flags) including .env file parsing and type validation. Run `./benchmark/run_benchmark.sh` to reproduce.*
161161

162162
## Why msgspec-ext?
163163

164-
- **Performance** - 3.8x faster than pydantic-settings
164+
- **Performance** - 2.7x faster than pydantic-settings
165165
- **Lightweight** - 4x smaller package size (0.49 MB vs 1.95 MB)
166166
- **Type safety** - Full type validation with modern Python type checkers
167167
- **Minimal dependencies** - Only msgspec and python-dotenv
@@ -172,7 +172,7 @@ msgspec-ext is **3.8x faster** than pydantic-settings while providing the same l
172172
|---------|------------|-------------------|
173173
| .env support |||
174174
| Type validation |||
175-
| Performance | **3.8x faster**| Baseline |
175+
| Performance | **2.7x faster**| Baseline |
176176
| Package size | 0.49 MB | 1.95 MB |
177177
| Nested config |||
178178
| Field aliases |||

benchmark.py

Lines changed: 0 additions & 172 deletions
This file was deleted.

0 commit comments

Comments
 (0)