You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Popen works well and is now the default like initially intended.
Fixed readme.
AsyncServerHandler code fixed and simplified.
Added time.sleep(random.random()) to reduce number of spawned processes.
improved setstate and getstate
Copy file name to clipboardExpand all lines: README.md
+78-76Lines changed: 78 additions & 76 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,79 +7,42 @@ A python package to offload a function call to an http server running on localho
7
7
Imagine a case of a multi threaded or multiprocessing application where 1 or few functions are heavily resource (cpu or memory) intensive, but the other functions can run in parallel.\
8
8
Example - an api call followed by tokenization and classification using a large DL model followed by further API calls.\
9
9
In such a case, it would make sense to create a server (generally using torchserve or tfserving) to serve requests, and replace the function call with a post request to the server.\
10
-
ServerHandler does this automatically during runtime. (Although it might be slower than a 'proper' server)\
11
-
AsyncServerHandler is also available which uses [aiohttp](https://docs.aiohttp.org/) to make the requests async, for use with fastapi and other async use cases.
12
-
AsyncServerHandler has the same usage as ServerHandler, except calls need to have await before it.
13
-
14
-
## Reccomended Usage
15
-
When the somemodule does not have any expensive global loading.
16
-
```python
17
-
from auto_function_serving.ServerHandler import ServerHandler
from auto_function_serving.ServerHandler import ServerHandler
24
-
someheavyfunction = ServerHandler("""
25
-
from somemodule import someheavyfunction
26
-
""", "someheavyfunction")
27
-
```
28
-
Any calls to this new **someheavyfunction** will make requests to 1 instance of a process running a [http.server.HTTPServer](https://docs.python.org/3/library/http.server.html) which runs the function within it. Even calls made from different processes, threads, multiprocessing or servers like flask.\
29
-
It can also be used like a traditional decorator for functions with no dependencies outside the function.
30
-
```python
31
-
from auto_function_serving.ServerHandler import ServerHandler
32
-
@ServerHandler.decorator
33
-
defsomeheavyfunction(args,**kwargs):
34
-
for i inrange(big_number)
35
-
someexpensivecomputation
36
-
```
37
-
imports inside the function will work
38
-
```python
39
-
from auto_function_serving.ServerHandler import ServerHandler
40
-
@ServerHandler.decorator
41
-
defsomeheavyfunction(args,**kwargs):
42
-
import numpy as np
43
-
```
44
-
```python
45
-
from auto_function_serving.ServerHandler import ServerHandler
46
-
@ServerHandler.decorator
47
-
defsomeheavyfunction(args,**kwargs):
48
-
ifnothasattr(someheavyfunction,'RunOnce'):
49
-
global np
50
-
import numpy as np
51
-
setattr(someheavyfunction,'RunOnce',None)
52
-
... etc
53
-
```
54
-
but the better way to do it might be
55
-
```python
56
-
from auto_function_serving.ServerHandler import ServerHandler
57
-
someheavyfunction = ServerHandler("""
58
-
import numpy as np
59
-
def someheavyfunction(args,**kwargs):
60
-
np.ones(1000)
61
-
...
62
-
""", "someheavyfunction")
63
-
```
64
-
## Installation
10
+
ServerHandler creates a **synchronous** server and replaces any calls to the function automatically during runtime.\
11
+
Requests are made to 1 instance of a process running a [http.server.HTTPServer](https://docs.python.org/3/library/http.server.html) which runs the function within it. Even calls made from different processes, threads, multiprocessing, flask, FastApi and async code are made to the same server process.\
12
+
Drawback - It might be slower than a 'proper' server and is restricted to 1 worker.
13
+
### AsyncServerHandler
14
+
AsyncServerHandler is also available which uses [aiohttp](https://docs.aiohttp.org/) to make the requests async, for use with fastapi and other async use cases. \
15
+
AsyncServerHandler has the same usage as ServerHandler, except calls need to be awaited or used with asyncio.run() or with asyncio.get_event_loop().run_until_complete().\
16
+
Number of async calls can be limited by setting AsyncServerHandler.TCPConnector_limit which controls the [TCPconnector](https://docs.aiohttp.org/en/stable/client_reference.html?highlight=connector#aiohttp.TCPConnector) limit (default 100) or by using [Semaphore](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore)
65
17
66
-
Use the package manager pip to install [auto_function_serving](https://pypi.org/project/auto-function-serving/)
67
-
```bash
68
-
pip install auto_function_serving
69
-
```
18
+
## Usage
70
19
71
-
## More Usage
72
20
In general :
73
21
```
74
-
some independent code with a callable
22
+
some code with a callable
75
23
```
76
-
can be replaced with
24
+
can be replaced with an instance of Either ServerHandler or AsyncserverHandler that accepts the code as a string in it's first argument and the name of the callable as the second argument.
77
25
```python
78
26
from auto_function_serving.ServerHandler import ServerHandler
79
27
callable_name = ServerHandler("""
80
28
some independent code with a callable
81
29
""", "callable_name")
82
30
```
31
+
Complete arguments
32
+
```python
33
+
from auto_function_serving.ServerHandler import ServerHandler
* if None, then the input code is hashed and a port is chosen from 50000 to 60000 using the hash
40
+
* if int, then int is chosen
41
+
* otherwise, a random open port is chosen
42
+
2. backend - either 'Popen' or 'multiprocessing'
43
+
3. wait - approx max number of seconds to wait for the server to run. No waiting done if set to 0, default 100
44
+
4. backlog - max number of backlogged requests before returning errors, python default is 5, but default in ServerHandler is 1024.
45
+
83
46
84
47
Example :
85
48
```python
@@ -91,27 +54,30 @@ def functionname(someinput):
91
54
```
92
55
can be replaced with
93
56
```python
94
-
from auto_function_serving.ServerHandler importServerHandler
95
-
functionname =ServerHandler("""
57
+
from auto_function_serving.ServerHandler importAsyncserverHandler
58
+
functionname =AsyncServerHandler("""
96
59
import module1
97
60
import module2
98
61
def functionname(someinput):
99
62
a = module1.function1(someinput)
100
63
return module2.function2(a)
101
-
""", "functionname")
64
+
""", "functionname", port="Any")
102
65
```
103
-
# How does this work?
104
-
Code for the server is stored in ServerHandler.base_code, inspect.cleandoc and some string formatting is used to fill in the blanks.\
105
-
A port from 50000 to 60000 is chosen by hashing the input text to make it independent of the source of a function. Collisions are possible, but unlikely. The port can be specified if needed.
106
-
```python
107
-
from somemodule import someheavyfunction
108
-
from auto_function_serving.ServerHandler import ServerHandler
Decorators for functions (@AsyncserverHandler.decorator and @ServerHandler.decorator) are also provided, but don't work in some cases. Details in more usage.
67
+
68
+
## Installation
69
+
70
+
Use the package manager pip to install [auto_function_serving](https://pypi.org/project/auto-function-serving/)
71
+
```bash
72
+
pip install auto_function_serving
110
73
```
111
-
ServerHandler.ip_address is set as "127.0.0.1".\
112
-
The server process is started with Popen (or multiprocessing if specified), and the first thing it does is import socket and bind the port - if it's not available the code stops after an exception. Therefore only 1 instance of the server runs at a time on a machine.\
74
+
75
+
## How does this work?
76
+
Code for the server is stored in [ServerHandler](https://github.com/arrmansa/auto-function-serving/blob/main/src/auto_function_serving/ServerHandler.py).base_code and inspect.cleandoc and some string formatting is used to fill in the blanks.\
77
+
The server process is started with Popen (or multiprocessing if specified). The first thing it does is import socket and bind the port - if it's not available the code stops after an exception. Therefore only 1 instance of the server runs at a time on a machine.\
113
78
We know the function is ready after we can recieve a valid get request from the server.\
114
-
Inputs and outputs are sent as bytes, converted to and from objects using pickle.
79
+
Inputs and outputs are sent as bytes, converted to and from objects using pickle.\
80
+
If port is None in while initializing (default), a port from 50000 to 60000 is chosen by hashing the input code to make it independent of the source of a function. Collisions of different functions are possible, but unlikely. The collision of the same function in multiple processes is used to make sure only 1 server process runs at a time. The port can be specified if needed.
115
81
116
82
## Performance (On my machine)
117
83
overhead for small input and output (few bytes) - \
@@ -122,6 +88,42 @@ overhead for large input and output\
122
88
~60ms for 5 mb input and output (10 mb total transfer).\
123
89
~600ms for 50 mb input and output (100 mb total transfer).
124
90
91
+
## More Usage
92
+
93
+
It can also be used with the provided decorator for functions with no dependencies outside the function.
94
+
```python
95
+
from auto_function_serving.ServerHandler import ServerHandler
96
+
@ServerHandler.decorator
97
+
defsomeheavyfunction(args,**kwargs):
98
+
for i inrange(big_number)
99
+
someexpensivecomputation
100
+
```
101
+
imports inside the function will work
102
+
```python
103
+
from auto_function_serving.ServerHandler import ServerHandler
104
+
@ServerHandler.decorator
105
+
defsomeheavyfunction(args,**kwargs):
106
+
import numpy as np
107
+
```
108
+
```python
109
+
from auto_function_serving.ServerHandler import ServerHandler
110
+
@ServerHandler.decorator
111
+
defsomeheavyfunction(args,**kwargs):
112
+
ifnothasattr(someheavyfunction,'RunOnce'):
113
+
global np
114
+
import numpy as np
115
+
setattr(someheavyfunction,'RunOnce',None)
116
+
... etc
117
+
```
118
+
119
+
When the somemodule does not have any expensive global loading.
120
+
```python
121
+
from auto_function_serving.ServerHandler import ServerHandler
0 commit comments