Commit 69c6f36
Improve benchmark with statistical analysis and fair comparison
Changes:
- Move benchmark files to dedicated /benchmark directory
- Add run_benchmark.sh script for easy execution with uvx
- Improve benchmark.py with:
- Statistical analysis: 10 runs with mean, median, std dev, min, max
- Fair comparison: Both libraries test equivalent configuration complexity
- System information: Display Python version, platform, library versions
- Better warmup: 50 iterations before timing
- Raw data output: All timing data for verification
- Comprehensive documentation
Benchmark improvements:
- Test realistic configuration with 18 fields across multiple categories
- msgspec-ext: Flat structure with __ delimiter (optimized for .env)
- pydantic-settings: Nested structure with BaseSettings classes
- Both load from identical .env file
Updated README with accurate benchmark results:
- 2.7x faster than pydantic-settings (down from 3.8x claim)
- Based on 10 runs × 1000 iterations with statistical validation
- More honest and reproducible results
Run: ./benchmark/run_benchmark.sh to reproduce results
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>1 parent 7e6ab1e commit 69c6f36
File tree
4 files changed
+322
-179
lines changed- benchmark
4 files changed
+322
-179
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
145 | | - | |
| 145 | + | |
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | | - | |
150 | | - | |
| 149 | + | |
| 150 | + | |
151 | 151 | | |
152 | | - | |
| 152 | + | |
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
156 | 156 | | |
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
160 | | - | |
| 160 | + | |
161 | 161 | | |
162 | 162 | | |
163 | 163 | | |
164 | | - | |
| 164 | + | |
165 | 165 | | |
166 | 166 | | |
167 | 167 | | |
| |||
172 | 172 | | |
173 | 173 | | |
174 | 174 | | |
175 | | - | |
| 175 | + | |
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
| |||
This file was deleted.
0 commit comments