Skip to content

Commit e473eb1

Browse files
abhinavsinghdependabot[bot]pre-commit-ci[bot]webknjaz
authored
v2.4.0rc2 (#838)
* Merge pull request #825 from abhinavsingh/dependabot/pip/pylint-2.12.1 pip prod(deps): bump pylint from 2.11.1 to 2.12.1 * Bump ncipollo/release-action from 1.8.10 to 1.9.0 (#826) Bumps [ncipollo/release-action](https://github.com/ncipollo/release-action) from 1.8.10 to 1.9.0. - [Release notes](https://github.com/ncipollo/release-action/releases) - [Commits](ncipollo/release-action@v1.8.10...v1.9.0) --- updated-dependencies: - dependency-name: ncipollo/release-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <[email protected]> * npm: bump rollup-plugin-copy from 3.3.0 to 3.4.0 in /dashboard (#824) Bumps [rollup-plugin-copy](https://github.com/vladshcherbin/rollup-plugin-copy) from 3.3.0 to 3.4.0. - [Release notes](https://github.com/vladshcherbin/rollup-plugin-copy/releases) - [Commits](vladshcherbin/rollup-plugin-copy@3.3.0...3.4.0) --- updated-dependencies: - dependency-name: rollup-plugin-copy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <[email protected]> * Add benchmarks comparison for `proxy.py`, `tornado`, `aiohttp`, `flask` (#827) * Add benchmarks for `proxy.py`, `tornado`, `aiohttp`, `flask` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Lint happy * Disable W0223 for tornado Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Integrate Towncrier configuration (#823) * Move changelog out of README into a separate file * Expose the changelog to Sphinx * Integrate Towncrier configuration into the repo * Include the change fragment docs in Sphinx * Add a config for the Chronographer GitHub App * Add a change note for PR #823 * Update CHANGELOG.md Co-authored-by: Abhinav Singh <[email protected]> * Add `benchmark` results to `README.md` (#828) * Add benchmark results to top-level README * mypy * Push down WIP example * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Spelling * Link rtfd within `Internal Documentation` for now Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * pip prod(deps): bump types-paramiko from 2.8.1 to 2.8.2 (#831) Bumps [types-paramiko](https://github.com/python/typeshed) from 2.8.1 to 2.8.2. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-paramiko dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump js-cookie from 2.2.1 to 3.0.1 in /dashboard (#830) Bumps [js-cookie](https://github.com/js-cookie/js-cookie) from 2.2.1 to 3.0.1. - [Release notes](https://github.com/js-cookie/js-cookie/releases) - [Changelog](https://github.com/js-cookie/js-cookie/blob/master/.release-it.json) - [Commits](js-cookie/js-cookie@v2.2.1...v3.0.1) --- updated-dependencies: - dependency-name: js-cookie dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <[email protected]> * Optimizations & Update Benchmark Results (#832) * Bail out of lock earlier * Avoid calling `get_events` for unfinished work tasks. Use `cached_property` for `HttpParser` optimization * Add `type: ignore[no-any-return]`, odd scenario * We dont have to rebuild response packet repeatedly within `WebServerPlugin` * Parse line and header in one invocation * Minor optimizations and update benchmark to use `oha` instead of `hey` * Remove `flask` from benchmark, only benchmark `asgi` or `async` based libraries. Use `uvicorn` and 10 workers for `blacksheep`. Use `oha` instead of `hey` * Add benchmark for `starlette` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add missing dep * pre-commit Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [ProxyPoolPlugin] Avoid remote proxy of private IP requests (#833) * Avoid proxy of requests to private IP within `ProxyPoolPlugin` * Fix tests * spell fix * No content length or chunked case can occur with `HTTP/1.1` too (#834) * No content length or chunked case can occur with `HTTP/1.1` too * `WPS331` false-positive * npm: bump eslint-plugin-standard from 4.1.0 to 5.0.0 in /dashboard (#835) Bumps [eslint-plugin-standard](https://github.com/standard/eslint-plugin-standard) from 4.1.0 to 5.0.0. - [Release notes](https://github.com/standard/eslint-plugin-standard/releases) - [Commits](standard/eslint-plugin-standard@v4.1.0...v5.0.0) --- updated-dependencies: - dependency-name: eslint-plugin-standard dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * pip prod(deps): bump twine from 3.5.0 to 3.6.0 (#836) Bumps [twine](https://github.com/pypa/twine) from 3.5.0 to 3.6.0. - [Release notes](https://github.com/pypa/twine/releases) - [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst) - [Commits](pypa/twine@3.5.0...3.6.0) --- updated-dependencies: - dependency-name: twine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * `task.result` can throw (#837) * `task.result` can raise exception * Remove redundant `except` * Fix doc spell. We need to fix it well, currently strategy feels dumb and PITA * Move common words within `spelling_wordlist.txt` Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <[email protected]>
2 parents a8e4d03 + 257d237 commit e473eb1

File tree

91 files changed

+1107
-503
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

91 files changed

+1107
-503
lines changed

.github/chronographer.yml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
enforce_name:
3+
suffix: .md
4+
exclude:
5+
bots:
6+
- dependabot-preview
7+
- dependabot
8+
- patchback
9+
humans:
10+
- pyup-bot
11+
...

.github/workflows/test-library.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -820,7 +820,7 @@ jobs:
820820
- name: >-
821821
Publish a GitHub Release for
822822
${{ needs.pre-setup.outputs.git-tag }}
823-
uses: ncipollo/release-action@v1.8.10
823+
uses: ncipollo/release-action@v1.9.0
824824
with:
825825
allowUpdates: false
826826
artifactErrorsFailBuild: false

.pre-commit-config.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,7 @@ repos:
176176
- --show-error-context
177177
- --strict
178178
- --strict-optional
179+
- benchmark/
179180
- examples/
180181
- proxy/
181182
- tests/

CHANGELOG.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<!-- markdownlint-disable no-duplicate-heading -->
2+
# Changelog
3+
4+
All notable changes to this project will be documented in this file.
5+
6+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
7+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
8+
9+
<!-- towncrier release notes start -->
10+
11+
12+
## v2.x
13+
14+
- No longer ~~a single file module~~.
15+
- Added support for threadless execution.
16+
- Added dashboard app.
17+
- Added support for unit testing.
18+
19+
## v1.x
20+
21+
- `Python3` only.
22+
- Deprecated support for ~~Python 2.x~~.
23+
- Added support multi core accept.
24+
- Added plugin support.
25+
26+
## v0.x
27+
28+
- Single file.
29+
- Single threaded server.
30+
31+
For detailed changelog refer to release PRs or commit history.

Makefile

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ endif
2929
.PHONY: all https-certificates sign-https-certificates ca-certificates
3030
.PHONY: lib-check lib-clean lib-test lib-package lib-coverage lib-lint lib-pytest
3131
.PHONY: lib-release-test lib-release lib-profile lib-doc
32-
.PHONY: lib-dep lib-flake8 lib-mypy
32+
.PHONY: lib-dep lib-flake8 lib-mypy lib-speedscope
3333
.PHONY: container container-run container-release container-build container-buildx
3434
.PHONY: devtools dashboard dashboard-clean
3535

@@ -138,9 +138,26 @@ lib-profile:
138138
-o profile.svg \
139139
-t -F -s -- \
140140
python -m proxy \
141+
--hostname 127.0.0.1 \
142+
--num-acceptors 1 \
143+
--num-workers 1 \
144+
--enable-web-server \
145+
--plugin proxy.plugin.WebServerPlugin \
146+
--local-executor \
147+
--backlog 65536 \
148+
--open-file-limit 65536 \
149+
--log-file /dev/null
150+
151+
lib-speedscope:
152+
ulimit -n 65536 && \
153+
sudo py-spy record \
154+
-o profile.speedscope.json \
155+
-f speedscope \
156+
-t -F -s -- \
157+
python -m proxy \
158+
--hostname 127.0.0.1 \
141159
--num-acceptors 1 \
142160
--num-workers 1 \
143-
--disable-http-proxy \
144161
--enable-web-server \
145162
--plugin proxy.plugin.WebServerPlugin \
146163
--local-executor \

README.md

Lines changed: 45 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -111,10 +111,10 @@
111111
- [Sending a Pull Request](#sending-a-pull-request)
112112
- [Benchmarks](#benchmarks)
113113
- [Flags](#flags)
114-
- [Changelog](#changelog)
115-
- [v2.x](#v2x)
116-
- [v1.x](#v1x)
117-
- [v0.x](#v0x)
114+
- [Changelog](https://proxypy.rtfd.io/en/latest/changelog)
115+
- [v2.x](https://proxypy.rtfd.io/en/latest/changelog#v2x)
116+
- [v1.x](https://proxypy.rtfd.io/en/latest/changelog#v1x)
117+
- [v0.x](https://proxypy.rtfd.io/en/latest/changelog#v0x)
118118

119119
[//]: # (DO-NOT-REMOVE-docs-badges-END)
120120

@@ -128,57 +128,55 @@
128128
```console
129129
# On Macbook Pro 2019 / 2.4 GHz 8-Core Intel Core i9 / 32 GB RAM
130130
❯ ./helper/benchmark.sh
131-
CONCURRENCY: 100 workers, TOTAL REQUESTS: 100000 req, QPS: 8000 req/sec, TIMEOUT: 1 sec
131+
CONCURRENCY: 100 workers, TOTAL REQUESTS: 100000 req
132132

133133
Summary:
134-
Total: 3.1217 secs
135-
Slowest: 0.0499 secs
136-
Fastest: 0.0004 secs
137-
Average: 0.0030 secs
138-
Requests/sec: 32033.7261
134+
Success rate: 1.0000
135+
Total: 2.5489 secs
136+
Slowest: 0.0443 secs
137+
Fastest: 0.0006 secs
138+
Average: 0.0025 secs
139+
Requests/sec: 39232.6572
139140

140-
Total data: 1900000 bytes
141-
Size/request: 19 bytes
141+
Total data: 1.81 MiB
142+
Size/request: 19 B
143+
Size/sec: 727.95 KiB
142144

143145
Response time histogram:
144-
0.000 [1] |
145-
0.005 [92268] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
146-
0.010 [7264] |■■■
147-
0.015 [318] |
148-
0.020 [102] |
149-
0.025 [32] |
150-
0.030 [6] |
151-
0.035 [4] |
152-
0.040 [1] |
153-
0.045 [2] |
154-
0.050 [2] |
155-
146+
0.001 [5006] |■■■■■
147+
0.001 [19740] |■■■■■■■■■■■■■■■■■■■■■
148+
0.002 [29701] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
149+
0.002 [21278] |■■■■■■■■■■■■■■■■■■■■■■
150+
0.003 [15376] |■■■■■■■■■■■■■■■■
151+
0.004 [6644] |■■■■■■■
152+
0.004 [1609] |■
153+
0.005 [434] |
154+
0.006 [83] |
155+
0.006 [29] |
156+
0.007 [100] |
156157

157158
Latency distribution:
158-
10% in 0.0017 secs
159-
25% in 0.0020 secs
160-
50% in 0.0025 secs
161-
75% in 0.0036 secs
162-
90% in 0.0050 secs
163-
95% in 0.0060 secs
164-
99% in 0.0087 secs
159+
10% in 0.0014 secs
160+
25% in 0.0018 secs
161+
50% in 0.0023 secs
162+
75% in 0.0030 secs
163+
90% in 0.0036 secs
164+
95% in 0.0040 secs
165+
99% in 0.0047 secs
165166

166167
Details (average, fastest, slowest):
167-
DNS+dialup: 0.0000 secs, 0.0004 secs, 0.0499 secs
168-
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
169-
req write: 0.0000 secs, 0.0000 secs, 0.0020 secs
170-
resp wait: 0.0030 secs, 0.0004 secs, 0.0462 secs
171-
resp read: 0.0000 secs, 0.0000 secs, 0.0027 secs
168+
DNS+dialup: 0.0025 secs, 0.0015 secs, 0.0030 secs
169+
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0001 secs
172170

173171
Status code distribution:
174-
[200] 100000 responses
172+
[200] 100000 responses
175173
```
176174

177-
PS: `proxy.py` and benchmark tools are running on the same machine during the above load test.
178-
Checkout the repo and try it for yourself. See [Benchmarks](#benchmarks) for more details.
175+
- See [Benchmark](https://github.com/abhinavsingh/proxy.py/tree/develop/benchmark#readme) for more details and how to run them locally.
179176

180177
- Lightweight
181-
- Uses only `~5-20MB` RAM
178+
- Uses `~5-20 MB` RAM
179+
- Compressed containers size is `~18.04 MB`
182180
- No external dependency other than standard Python library
183181
- Programmable
184182
- Customize proxy behavior using [Proxy Server Plugins](#http-proxy-plugins). Example:
@@ -1627,21 +1625,11 @@ optional arguments:
16271625

16281626
## Internal Documentation
16291627

1630-
Code is well documented. Browse through internal class hierarchy and documentation using `pydoc3`
1631-
1632-
```console
1633-
pydoc3 proxy
1634-
1635-
PACKAGE CONTENTS
1636-
__main__
1637-
common (package)
1638-
core (package)
1639-
http (package)
1640-
main
1628+
Code is well documented. You have a few options to browse the internal class hierarchy and documentation:
16411629

1642-
FILE
1643-
/Users/abhinav/Dev/proxy.py/proxy/__init__.py
1644-
```
1630+
1. Visit [proxypy.readthedocs.io](https://proxypy.readthedocs.io/)
1631+
2. Build and open docs locally using `make lib-doc`
1632+
2. Use `pydoc3` locally using `pydoc3 proxy`
16451633

16461634
# Run Dashboard
16471635

@@ -2028,7 +2016,9 @@ for list of tests.
20282016

20292017
# Benchmarks
20302018

2031-
Simply run the following command from repo root to start benchmark
2019+
See [Benchmark](https://github.com/abhinavsingh/proxy.py/tree/develop/benchmark) directory on how to run benchmark comparisons with other OSS web servers.
2020+
2021+
To run standalone benchmark for `proxy.py`, use the following command from repo root:
20322022

20332023
```console
20342024
./helper/benchmark.sh
@@ -2216,30 +2206,3 @@ options:
22162206
Proxy.py not working? Report at:
22172207
https://github.com/abhinavsingh/proxy.py/issues/new
22182208
```
2219-
2220-
# Changelog
2221-
2222-
## v2.4.0
2223-
2224-
- No longer support `Python 3.6` due to `asyncio.run` usage in the core.
2225-
2226-
## v2.x
2227-
2228-
- No longer ~~a single file module~~.
2229-
- Added support for threadless execution.
2230-
- Added dashboard app.
2231-
- Added support for unit testing.
2232-
2233-
## v1.x
2234-
2235-
- `Python3` only.
2236-
- Deprecated support for ~~Python 2.x~~.
2237-
- Added support multi core accept.
2238-
- Added plugin support.
2239-
2240-
## v0.x
2241-
2242-
- Single file.
2243-
- Single threaded server.
2244-
2245-
For detailed changelog refer to release PRs or commit history.

benchmark/README.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Benchmark
2+
3+
# Table of Contents
4+
- [TL;DR](#tldr)
5+
- [Usage](#usage)
6+
- [Results](#results)
7+
8+
## TL;DR
9+
10+
NOTE: On Macbook Pro 2019 / 2.4 GHz 8-Core Intel Core i9 / 32 GB RAM
11+
12+
| Server | Throughput (request/sec) | Num Workers | Runner |
13+
| ------ | ------------ | ------------------------| ------ |
14+
| `blacksheep` | 46,564 | 10 | uvicorn |
15+
| `starlette` | 44,102 | 10 | uvicorn |
16+
| `proxy.py` | 39,232 | 10 | - |
17+
| `aiohttp` | 6,615 | 1 | - |
18+
| `tornado` | 3,301 | 1 | - |
19+
20+
- On a single core, `proxy.py` yields `~9449 req/sec` throughput.
21+
- Try it using `--num-acceptors=1`
22+
23+
## Usage
24+
25+
```console
26+
git clone https://github.com/abhinavsingh/proxy.py.git
27+
cd proxy.py
28+
pip install -r benchmark/requirements.txt
29+
./benchmark/compare.sh > /tmp/compare.log 2>&1
30+
```
31+
32+
## Results
33+
34+
```console
35+
cat /tmp/compare.log
36+
```
File renamed without changes.

benchmark/aiohttp/__init__.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# -*- coding: utf-8 -*-
2+
"""
3+
proxy.py
4+
~~~~~~~~
5+
⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
6+
Network monitoring, controls & Application development, testing, debugging.
7+
8+
:copyright: (c) 2013-present by Abhinav Singh and contributors.
9+
:license: BSD, see LICENSE for more details.
10+
"""

benchmark/aiohttp/server.py

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# -*- coding: utf-8 -*-
2+
"""
3+
proxy.py
4+
~~~~~~~~
5+
⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
6+
Network monitoring, controls & Application development, testing, debugging.
7+
8+
:copyright: (c) 2013-present by Abhinav Singh and contributors.
9+
:license: BSD, see LICENSE for more details.
10+
"""
11+
from aiohttp import web
12+
13+
14+
async def handle(request: web.Request) -> web.StreamResponse:
15+
return web.Response(body=b'HTTP route response')
16+
17+
18+
app = web.Application()
19+
20+
app.add_routes(
21+
[
22+
web.get('/http-route-example', handle),
23+
],
24+
)
25+
26+
web.run_app(app, host='127.0.0.1', port=8080)

0 commit comments

Comments
 (0)