Skip to content

Commit 68df66f

Browse files
Add benchmark results to README.md (#828)
* Add benchmark results to top-level README * mypy * Push down WIP example * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Spelling * Link rtfd within `Internal Documentation` for now Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 6aee778 commit 68df66f

File tree

7 files changed

+214
-84
lines changed

7 files changed

+214
-84
lines changed

README.md

Lines changed: 20 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -174,11 +174,21 @@
174174
[200] 100000 responses
175175
```
176176

177-
PS: `proxy.py` and benchmark tools are running on the same machine during the above load test.
178-
Checkout the repo and try it for yourself. See [Benchmarks](#benchmarks) for more details.
177+
- `1 Million` requests benchmark comparison @ `8000 QPS`
178+
179+
| Server | Throughput (request/sec) |
180+
| ------ | ------------ |
181+
| `proxy.py` | 30,351 |
182+
| `blacksheep` | 7,358 |
183+
| `aiohttp` | 6,615 |
184+
| `tornado` | 3,301 |
185+
| `Flask` | 830 |
186+
187+
See [Benchmark](https://github.com/abhinavsingh/proxy.py/tree/develop/benchmark#readme) for more details and how to run them locally.
179188

180189
- Lightweight
181-
- Uses only `~5-20MB` RAM
190+
- Uses `~5-20 MB` RAM
191+
- Compressed containers size is `~18.04 MB`
182192
- No external dependency other than standard Python library
183193
- Programmable
184194
- Customize proxy behavior using [Proxy Server Plugins](#http-proxy-plugins). Example:
@@ -1627,21 +1637,11 @@ optional arguments:
16271637

16281638
## Internal Documentation
16291639

1630-
Code is well documented. Browse through internal class hierarchy and documentation using `pydoc3`
1631-
1632-
```console
1633-
pydoc3 proxy
1640+
Code is well documented. You have a few options to browse the internal class hierarchy and documentation:
16341641

1635-
PACKAGE CONTENTS
1636-
__main__
1637-
common (package)
1638-
core (package)
1639-
http (package)
1640-
main
1641-
1642-
FILE
1643-
/Users/abhinav/Dev/proxy.py/proxy/__init__.py
1644-
```
1642+
1. Visit [proxypy.readthedocs.io](https://proxypy.readthedocs.io/)
1643+
2. Build and open docs locally using `make lib-doc`
1644+
2. Use `pydoc3` locally using `pydoc3 proxy`
16451645

16461646
# Run Dashboard
16471647

@@ -2028,7 +2028,9 @@ for list of tests.
20282028

20292029
# Benchmarks
20302030

2031-
Simply run the following command from repo root to start benchmark
2031+
See [Benchmark](https://github.com/abhinavsingh/proxy.py/tree/develop/benchmark) directory on how to run benchmark comparisons with other OSS web servers.
2032+
2033+
To run standalone benchmark for `proxy.py`, use the following command from repo root:
20322034

20332035
```console
20342036
./helper/benchmark.sh

benchmark/README.md

Lines changed: 109 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,30 @@
11
# Benchmark
22

33
# Table of Contents
4+
- [TL;DR](#tldr)
45
- [Usage](#usage)
56
- [Results](#results)
67
- [Proxy.py](#proxypy)
7-
- [Tornado](#tornado)
8+
- [Blacksheep](#blacksheep)
89
- [AioHttp](#aiohttp)
10+
- [Tornado](#tornado)
911
- [Flask](#flask)
1012

13+
## TL;DR
14+
15+
1 Million requests benchmark results @ 8000 QPS
16+
17+
| Server | Throughput (request/sec) |
18+
| ------ | ------------ |
19+
| `proxy.py` | 30,351 |
20+
| `blacksheep` | 7,358 |
21+
| `aiohttp` | 6,615 |
22+
| `tornado` | 3,301 |
23+
| `Flask` | 830 |
24+
25+
- On a single core, `proxy.py` yields `7829 req/sec` throughput
26+
- Reference: [Add benchmarks comparison for `proxy.py`, `tornado`, `aiohttp`, `flask`](https://github.com/abhinavsingh/proxy.py/pull/827)
27+
1128
## Usage
1229

1330
```console
@@ -24,7 +41,7 @@
2441
CONCURRENCY: 100 workers, QPS: 8000 req/sec, TOTAL DURATION: 1m, TIMEOUT: 1 sec
2542
```
2643

27-
### Proxy.py
44+
### `Proxy.py`
2845

2946
```console
3047
=============================
@@ -79,64 +96,63 @@ Server gracefully shutdown
7996
=============================
8097
```
8198

82-
### Tornado
99+
### `Blacksheep`
83100

84101
```console
85102
=============================
86-
Benchmarking Tornado
87-
Server (pid:31161) running
103+
Benchmarking Blacksheep
104+
Server (pid:42973) running
88105

89106
Summary:
90-
Total: 60.0208 secs
91-
Slowest: 0.1234 secs
92-
Fastest: 0.0050 secs
93-
Average: 0.0323 secs
94-
Requests/sec: 3301.1874
107+
Total: 60.0112 secs
108+
Slowest: 0.0357 secs
109+
Fastest: 0.0047 secs
110+
Average: 0.0136 secs
111+
Requests/sec: 7358.7628
95112

96-
Total data: 3515190 bytes
113+
Total data: 8390552 bytes
97114
Size/request: 19 bytes
98115

99116
Response time histogram:
100117
0.005 [1] |
101-
0.017 [6] |
102-
0.029 [13657] |■■■
103-
0.040 [168114] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
104-
0.052 [2458] |■
105-
0.064 [411] |
106-
0.076 [194] |
107-
0.088 [72] |
108-
0.100 [6] |
109-
0.112 [34] |
110-
0.123 [57] |
118+
0.008 [3] |
119+
0.011 [4] |
120+
0.014 [310108] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
121+
0.017 [123192] |■■■■■■■■■■■■■■■
122+
0.020 [6795] |
123+
0.023 [1008] |
124+
0.026 [222] |
125+
0.029 [180] |
126+
0.033 [58] |
127+
0.036 [37] |
111128

112129

113130
Latency distribution:
114-
10% in 0.0288 secs
115-
25% in 0.0299 secs
116-
50% in 0.0317 secs
117-
75% in 0.0338 secs
118-
90% in 0.0360 secs
119-
95% in 0.0378 secs
120-
99% in 0.0426 secs
131+
10% in 0.0123 secs
132+
25% in 0.0127 secs
133+
50% in 0.0133 secs
134+
75% in 0.0142 secs
135+
90% in 0.0152 secs
136+
95% in 0.0158 secs
137+
99% in 0.0181 secs
121138

122139
Details (average, fastest, slowest):
123-
DNS+dialup: 0.0000 secs, 0.0050 secs, 0.1234 secs
140+
DNS+dialup: 0.0000 secs, 0.0047 secs, 0.0357 secs
124141
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
125-
req write: 0.0000 secs, 0.0000 secs, 0.0005 secs
126-
resp wait: 0.0322 secs, 0.0048 secs, 0.0960 secs
142+
req write: 0.0000 secs, 0.0000 secs, 0.0006 secs
143+
resp wait: 0.0136 secs, 0.0030 secs, 0.0356 secs
127144
resp read: 0.0000 secs, 0.0000 secs, 0.0010 secs
128145

129146
Status code distribution:
130-
[200] 185010 responses
147+
[200] 441608 responses
148+
131149

132-
Error distribution:
133-
[13130] Get "http://127.0.0.1:8888/http-route-example": dial tcp 127.0.0.1:8888: connect: connection refused
134150

135151
Server gracefully shutdown
136152
=============================
137153
```
138154

139-
### AioHttp
155+
### `AioHttp`
140156

141157
```console
142158
=============================
@@ -194,7 +210,64 @@ Server gracefully shutdown
194210
=============================
195211
```
196212

197-
### Flask
213+
### `Tornado`
214+
215+
```console
216+
=============================
217+
Benchmarking Tornado
218+
Server (pid:31161) running
219+
220+
Summary:
221+
Total: 60.0208 secs
222+
Slowest: 0.1234 secs
223+
Fastest: 0.0050 secs
224+
Average: 0.0323 secs
225+
Requests/sec: 3301.1874
226+
227+
Total data: 3515190 bytes
228+
Size/request: 19 bytes
229+
230+
Response time histogram:
231+
0.005 [1] |
232+
0.017 [6] |
233+
0.029 [13657] |■■■
234+
0.040 [168114] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
235+
0.052 [2458] |■
236+
0.064 [411] |
237+
0.076 [194] |
238+
0.088 [72] |
239+
0.100 [6] |
240+
0.112 [34] |
241+
0.123 [57] |
242+
243+
244+
Latency distribution:
245+
10% in 0.0288 secs
246+
25% in 0.0299 secs
247+
50% in 0.0317 secs
248+
75% in 0.0338 secs
249+
90% in 0.0360 secs
250+
95% in 0.0378 secs
251+
99% in 0.0426 secs
252+
253+
Details (average, fastest, slowest):
254+
DNS+dialup: 0.0000 secs, 0.0050 secs, 0.1234 secs
255+
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
256+
req write: 0.0000 secs, 0.0000 secs, 0.0005 secs
257+
resp wait: 0.0322 secs, 0.0048 secs, 0.0960 secs
258+
resp read: 0.0000 secs, 0.0000 secs, 0.0010 secs
259+
260+
Status code distribution:
261+
[200] 185010 responses
262+
263+
Error distribution:
264+
[13130] Get "http://127.0.0.1:8888/http-route-example": dial tcp 127.0.0.1:8888: connect: connection refused
265+
266+
Server gracefully shutdown
267+
=============================
268+
```
269+
270+
### `Flask`
198271

199272
```console
200273
=============================

benchmark/blacksheep/__init__.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# -*- coding: utf-8 -*-
2+
"""
3+
proxy.py
4+
~~~~~~~~
5+
⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
6+
Network monitoring, controls & Application development, testing, debugging.
7+
8+
:copyright: (c) 2013-present by Abhinav Singh and contributors.
9+
:license: BSD, see LICENSE for more details.
10+
"""

benchmark/blacksheep/server.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# -*- coding: utf-8 -*-
2+
"""
3+
proxy.py
4+
~~~~~~~~
5+
⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
6+
Network monitoring, controls & Application development, testing, debugging.
7+
8+
:copyright: (c) 2013-present by Abhinav Singh and contributors.
9+
:license: BSD, see LICENSE for more details.
10+
"""
11+
from blacksheep.server import Application
12+
from blacksheep.server.responses import text
13+
14+
15+
app = Application()
16+
17+
18+
@app.route('/http-route-example')
19+
async def home(request): # type: ignore[no-untyped-def]
20+
return text('HTTP route response')

0 commit comments

Comments
 (0)