Skip to content

Commit a8d7163

Browse files
evelynmitchellevelynmitchell
committed
intial commit
0 parents  commit a8d7163

22 files changed

+3153
-0
lines changed

‎EXAMPLE_OUTPUT.md‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
# Example Benchmark Output
2+
3+
This document shows example output from the benchmarking suite to help you understand what to expect.
4+
5+
## Running Benchmarks
6+
7+
```bash
8+
$ python benchmarks/run_benchmarks.py --categories numpy memory async --output-dir /tmp/example_results
9+
```
10+
11+
## Console Output
12+
13+
```
14+
================================================================================
15+
Python Performance Benchmark Suite
16+
================================================================================
17+
18+
Python: 3.12.3 (CPython)
19+
Platform: Linux-6.11.0-1018-azure-x86_64-with-glibc2.39
20+
Processor: x86_64
21+
22+
Categories: numpy, memory, async
23+
Results will be saved to: /tmp/example_results/benchmarks_py3123_20251212_154343.json
24+
25+
================================================================================
26+
27+
============================= test session starts ==============================
28+
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /usr/bin/python
29+
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5...)
30+
collecting ... collected 79 items
31+
32+
benchmarks/numpy/test_array_operations.py::TestArrayCreation::test_benchmark_array_zeros PASSED [ 1%]
33+
benchmarks/numpy/test_array_operations.py::TestArrayCreation::test_benchmark_array_ones PASSED [ 2%]
34+
benchmarks/numpy/test_array_operations.py::TestArrayCreation::test_benchmark_array_random PASSED [ 3%]
35+
...
36+
benchmarks/async/test_async_operations.py::TestAsyncContextManagers::test_benchmark_async_context_manager PASSED [100%]
37+
38+
--------------------------------------------------------------------------------------------------------------------- benchmark: 79 tests ---------------------------------------------------------------------------------------------------------------------
39+
Name (time in ns) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
40+
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
41+
test_benchmark_memoryview_creation 101.7900 (1.0) 472.3800 (1.0) 104.8514 (1.0) 8.5320 (1.0) 103.9000 (1.0) 0.8900 (1.10) 1249;2265 9,537,306.9001 (1.0) 97003 100
42+
test_benchmark_memoryview_slicing 115.7100 (1.14) 600.5400 (1.27) 120.1996 (1.15) 16.0398 (1.88) 117.7200 (1.13) 0.8100 (1.0) 2322;3157 8,319,492.3590 (0.87) 85970 100
43+
test_benchmark_basic_indexing 148.7800 (1.46) 624.0700 (1.32) 156.2725 (1.49) 11.3937 (1.34) 156.1900 (1.50) 5.2100 (6.43) 1302;1390 6,399,077.3746 (0.67) 67627 100
44+
...
45+
test_benchmark_matrix_inverse 24,451,620.0000 (>1000.0) 26,837,460.0000 (>1000.0) 25,023,254.9583 (>1000.0) 507,854.1733 (>1000.0) 24,794,725.0000 (>1000.0) 423,817.5000 (>1000.0) 59;15 39.9628 (0.00) 124 1
46+
test_benchmark_eigenvalues 32,166,045.0001 (>1000.0) 35,146,420.0000 (>1000.0) 32,787,383.5147 (>1000.0) 632,673.5280 (>1000.0) 32,632,055.0000 (>1000.0) 433,702.5000 (>1000.0) 52;21 30.4995 (0.00) 102 1
47+
48+
============================== 79 passed in 157.32s (0:02:37) ===============================
49+
```
50+
51+
## Summary Output
52+
53+
After benchmarks complete, you'll see a summary:
54+
55+
```
56+
================================================================================
57+
BENCHMARK SUMMARY
58+
================================================================================
59+
60+
NumPy Benchmarks (28 tests)
61+
--------------------------------------------------------------------------------
62+
Average time: 2854.3421 ms
63+
Min time: 0.1048 ms
64+
Max time: 35146.4200 ms
65+
66+
Top 5 slowest tests:
67+
1. test_benchmark_eigenvalues 32787.3835 ms
68+
2. test_benchmark_matrix_inverse 25023.2550 ms
69+
3. test_benchmark_svd 21456.8920 ms
70+
4. test_benchmark_matrix_multiply 15234.1234 ms
71+
5. test_benchmark_broadcast_outer_product 12876.5432 ms
72+
73+
Memory Benchmarks (28 tests)
74+
--------------------------------------------------------------------------------
75+
Average time: 1234.5678 ms
76+
Min time: 0.1049 ms
77+
Max time: 4898.5567 ms
78+
79+
Top 5 slowest tests:
80+
1. test_benchmark_gc_disabled 4898.5567 ms
81+
2. test_benchmark_reference_cycles 4167.6484 ms
82+
3. test_benchmark_gc_collect 3767.8965 ms
83+
4. test_benchmark_dict_preallocated 579.5765 ms
84+
5. test_benchmark_dict_creation 497.5206 ms
85+
86+
Async Benchmarks (23 tests)
87+
--------------------------------------------------------------------------------
88+
Average time: 345.6789 ms
89+
Min time: 0.1049 ms
90+
Max time: 2345.6789 ms
91+
92+
Top 5 slowest tests:
93+
1. test_benchmark_concurrent_sleeps 1234.5678 ms
94+
2. test_benchmark_producer_consumer 987.6543 ms
95+
3. test_benchmark_pipeline 876.5432 ms
96+
4. test_benchmark_fan_out_fan_in 765.4321 ms
97+
5. test_benchmark_gather_1000_tasks 654.3210 ms
98+
99+
================================================================================
100+
Total benchmarks: 79
101+
Total time: 157.32 seconds
102+
Average per benchmark: 1991.8987 ms
103+
================================================================================
104+
105+
✅ Benchmarks completed successfully!
106+
Results saved to: /tmp/example_results/benchmarks_py3123_20251212_154343.json
107+
108+
To compare with another run:
109+
python benchmarks/utils/compare_results.py <baseline.json> /tmp/example_results/benchmarks_py3123_20251212_154343.json
110+
```
111+
112+
## Comparing Python Versions
113+
114+
Example output when comparing Python 3.11 vs 3.12:
115+
116+
```bash
117+
$ python benchmarks/utils/compare_results.py results_311.json results_312.json
118+
```
119+
120+
```
121+
================================================================================
122+
Benchmark Comparison: results_311 vs results_312
123+
================================================================================
124+
125+
Benchmark Baseline (ms) Comparison (ms) Change
126+
---------------------------------------------------------------------------------------------------------
127+
test_benchmark_array_zeros 145.2340 139.6760 -3.83%
128+
test_benchmark_empty_coroutine 112.3456 105.5675 -6.03%
129+
test_benchmark_list_creation 138.9012 131.5573 -5.29%
130+
test_benchmark_string_formatting 115.6789 104.9458 -9.27%
131+
test_benchmark_matrix_multiply 15876.5432 15234.1234 -4.05%
132+
...
133+
134+
================================================================================
135+
SUMMARY
136+
================================================================================
137+
138+
✅ Top Improvements (faster in results_312):
139+
-9.27% - test_benchmark_string_formatting
140+
-6.03% - test_benchmark_empty_coroutine
141+
-5.29% - test_benchmark_list_creation
142+
-4.05% - test_benchmark_matrix_multiply
143+
-3.83% - test_benchmark_array_zeros
144+
145+
⚠️ Top Regressions (slower in results_312):
146+
+2.15% - test_benchmark_gc_collect
147+
+1.23% - test_benchmark_import_asyncio
148+
149+
Total benchmarks: 79
150+
Improved: 67 (84.8%)
151+
Regressed: 8 (10.1%)
152+
Neutral: 4 (5.1%)
153+
```
154+
155+
## Memory Profiling Output
156+
157+
Example output from memory profiling:
158+
159+
```bash
160+
$ python benchmarks/utils/profile_memory.py
161+
```
162+
163+
```
164+
Memory Profiling Utilities
165+
============================================================
166+
167+
Profiling example function...
168+
169+
Result: 499500
170+
Current memory: 0.78 MB
171+
Peak memory: 1.23 MB
172+
173+
Top allocations:
174+
0.4500 MB - 100 allocations
175+
0.2300 MB - 1000 allocations
176+
0.1200 MB - 100 allocations
177+
0.0890 MB - 1 allocations
178+
0.0567 MB - 10 allocations
179+
180+
Memory Snapshot:
181+
------------------------------------------------------------
182+
183+
Top 20 object types by count:
184+
dict 12,345
185+
list 8,901
186+
tuple 6,789
187+
function 4,567
188+
type 3,456
189+
...
190+
```
191+
192+
## Key Insights from Benchmarks
193+
194+
### What You Can Learn
195+
196+
1. **NumPy Performance**
197+
- Array creation: ~100-150 microseconds for 1M elements
198+
- Matrix multiplication: ~15ms for 500x500 matrices
199+
- Linear algebra operations show significant improvements in newer Python versions
200+
201+
2. **Memory Management**
202+
- List creation: ~130 microseconds for 10K elements
203+
- GC collection: ~3-5 milliseconds
204+
- Preallocated structures are 10-15% faster
205+
206+
3. **Async Performance**
207+
- Empty coroutine overhead: ~105 microseconds
208+
- Task creation: ~50 microseconds
209+
- 1000 concurrent tasks: ~650 milliseconds
210+
211+
4. **Process Startup**
212+
- Module import: 50-200 microseconds (cached)
213+
- Subprocess creation: 5-10 milliseconds
214+
- Import time varies significantly by module complexity
215+
216+
## Next Steps
217+
218+
1. Run your own benchmarks: `python benchmarks/run_benchmarks.py`
219+
2. Compare different Python versions
220+
3. Identify performance bottlenecks in your specific use case
221+
4. Track performance improvements over time

‎QUICKSTART.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Quick Start Guide
2+
3+
This guide will help you get started with the Python Performance Benchmarking Suite.
4+
5+
## Installation
6+
7+
1. **Install the benchmarking dependencies:**
8+
9+
```bash
10+
cd benchmarks
11+
pip install -r requirements.txt
12+
```
13+
14+
Note: If you encounter issues installing PyTorch, you can skip it and run benchmarks without the PyTorch category.
15+
16+
## Running Your First Benchmark
17+
18+
### Option 1: Run All Benchmarks
19+
20+
```bash
21+
python benchmarks/run_benchmarks.py
22+
```
23+
24+
This will run all benchmark categories and save results to `benchmark_results/`.
25+
26+
### Option 2: Run Specific Categories
27+
28+
```bash
29+
# Run only NumPy benchmarks
30+
python benchmarks/run_benchmarks.py --categories numpy
31+
32+
# Run multiple categories
33+
python benchmarks/run_benchmarks.py --categories numpy memory async
34+
```
35+
36+
### Option 3: Run Individual Tests
37+
38+
```bash
39+
# Run a specific test
40+
cd /path/to/repository
41+
python -m pytest benchmarks/numpy/test_array_operations.py::TestArrayCreation --benchmark-only --no-cov -v
42+
43+
# Run all NumPy tests
44+
python -m pytest benchmarks/numpy/ --benchmark-only --no-cov -v
45+
```
46+
47+
## Understanding the Output
48+
49+
After running benchmarks, you'll see output like:
50+
51+
```
52+
-------------------------------------------------------- benchmark: 5 tests -------------------------------------------------------
53+
Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS (Kops/s) Rounds Iterations
54+
------------------------------------------------------------------------------------------------------------------------------
55+
test_benchmark_array_zeros 102.80 554.47 139.68 10.11 137.62 8.87 3420;1587 7.16 40823 1
56+
```
57+
58+
Key metrics:
59+
- **Min/Max**: Fastest and slowest execution times
60+
- **Mean**: Average execution time
61+
- **Median**: Middle value (less affected by outliers)
62+
- **StdDev**: Standard deviation (consistency of results)
63+
- **OPS**: Operations per second (higher is better)
64+
65+
## Analyzing Results
66+
67+
### 1. View Summary
68+
69+
The run_benchmarks.py script automatically generates a summary showing:
70+
- Total benchmarks run
71+
- Category-wise statistics
72+
- Top 5 slowest tests per category
73+
74+
### 2. Compare Python Versions
75+
76+
```bash
77+
# Run benchmarks with Python 3.11
78+
python3.11 benchmarks/run_benchmarks.py --output-dir results/
79+
80+
# Run benchmarks with Python 3.12
81+
python3.12 benchmarks/run_benchmarks.py --output-dir results/
82+
83+
# Compare the results
84+
python benchmarks/utils/compare_results.py results/benchmarks_py311*.json results/benchmarks_py312*.json
85+
```
86+
87+
### 3. Profile Memory Usage
88+
89+
```bash
90+
# Profile a specific operation
91+
python benchmarks/utils/profile_memory.py
92+
```
93+
94+
## Common Use Cases
95+
96+
### Test Array Operations Performance
97+
98+
```bash
99+
python -m pytest benchmarks/numpy/test_array_operations.py --benchmark-only --no-cov -v
100+
```
101+
102+
### Test Async Performance
103+
104+
```bash
105+
python -m pytest benchmarks/async/test_async_operations.py --benchmark-only --no-cov -v
106+
```
107+
108+
### Test Memory Management
109+
110+
```bash
111+
python -m pytest benchmarks/memory/test_memory_operations.py --benchmark-only --no-cov -v
112+
```
113+
114+
### Test Import/Startup Time
115+
116+
```bash
117+
python -m pytest benchmarks/startup/test_startup_operations.py --benchmark-only --no-cov -v
118+
```
119+
120+
## Tips
121+
122+
1. **Warm-up runs**: The benchmark framework automatically performs warm-up runs to ensure fair measurements.
123+
124+
2. **Minimize system load**: For most accurate results, close other applications and run benchmarks when the system is idle.
125+
126+
3. **Multiple runs**: Run benchmarks multiple times to account for system variability:
127+
```bash
128+
for i in {1..3}; do
129+
python benchmarks/run_benchmarks.py --output-dir results/run_$i
130+
done
131+
```
132+
133+
4. **Focus on relative changes**: When comparing Python versions, focus on relative performance changes rather than absolute values.
134+
135+
## Troubleshooting
136+
137+
### PyTorch Not Available
138+
139+
If PyTorch is not installed, the PyTorch benchmarks will be skipped automatically. To install PyTorch:
140+
141+
```bash
142+
# CPU version
143+
pip install torch --index-url https://download.pytorch.org/whl/cpu
144+
145+
# CUDA version (if you have NVIDIA GPU)
146+
pip install torch
147+
```
148+
149+
### Coverage Errors
150+
151+
If you see coverage-related errors, make sure to use the `--no-cov` flag:
152+
153+
```bash
154+
python -m pytest benchmarks/ --benchmark-only --no-cov -v
155+
```
156+
157+
### Slow Benchmarks
158+
159+
Some benchmarks (especially GC and complex operations) can take a long time. You can:
160+
- Skip slow tests: `pytest -m "not slow"`
161+
- Reduce min-rounds: `pytest --benchmark-min-rounds=3`
162+
- Focus on specific categories
163+
164+
## Next Steps
165+
166+
- Read the full [README.md](README.md) for detailed information
167+
- Explore [benchmark categories](README.md#benchmark-categories)
168+
- Learn about [comparing Python versions](README.md#comparing-python-versions)
169+
- Check out the [example comparison script](compare_versions.sh)

0 commit comments

Comments
 (0)