|
| 1 | +Metadata-Version: 2.1 |
| 2 | +Name: auto-function-serving |
| 3 | +Version: 0.1.6 |
| 4 | +Summary: A python package to offload a function call to an http server automatically using a decorator. |
| 5 | +Author-email: Arrman Anicket Saha <arrmansa99+430@gmail.com> |
| 6 | +Project-URL: Homepage, https://github.com/arrmansa/auto-function-serving |
| 7 | +Project-URL: Bug Tracker, https://github.com/arrmansa/auto-function-serving/issues |
| 8 | +Classifier: Programming Language :: Python :: 3 |
| 9 | +Classifier: License :: OSI Approved :: Apache Software License |
| 10 | +Classifier: Operating System :: OS Independent |
| 11 | +Requires-Python: >=3.7 |
| 12 | +Description-Content-Type: text/markdown |
| 13 | +License-File: LICENSE |
| 14 | + |
| 15 | +# auto-function-serving |
| 16 | + |
| 17 | +A python package to offload a function call to an http server running on localhost automatically using a decorator. Compatible with multiprocessing, pickle, flask, fastapi, async etc.. |
| 18 | + |
| 19 | +## Why |
| 20 | + |
| 21 | +Imagine a case of a multi threaded or multiprocessing application where 1 or few functions are heavily resource (cpu or memory) intensive, but the other functions can run in parallel.\ |
| 22 | +Example - an api call followed by tokenization and classification using a large DL model followed by further API calls.\ |
| 23 | +In such a case, it would make sense to create a server (generally using torchserve or tfserving) to serve requests, and replace the function call with a post request to the server.\ |
| 24 | +ServerHandler creates a **synchronous** server and replaces any calls to the function automatically during runtime.\ |
| 25 | +Requests are made to 1 instance of a process running a [http.server.HTTPServer](https://docs.python.org/3/library/http.server.html) which runs the function within it.\ |
| 26 | +AsyncServerHandler is also available which makes the requests asynchronously.\ |
| 27 | +Even calls made from different processes, threads, multiprocessing, flask, FastApi and async event loops are made to the same server process. |
| 28 | + |
| 29 | +## Usage |
| 30 | + |
| 31 | +In general : |
| 32 | +``` |
| 33 | +some code with a callable |
| 34 | +``` |
| 35 | +can be replaced with an instance of Either ServerHandler or AsyncserverHandler that accepts the code as a string in it's first argument and the name of the callable as the second argument. |
| 36 | +```python |
| 37 | +from auto_function_serving.ServerHandler import ServerHandler |
| 38 | +callable_name = ServerHandler(""" |
| 39 | +some independent code with a callable |
| 40 | +""", "callable_name") |
| 41 | +``` |
| 42 | +Example : |
| 43 | +```python |
| 44 | +import module1 |
| 45 | +import module2 |
| 46 | +def functionname(someinput): |
| 47 | + a = module1.function1(someinput) |
| 48 | + return module2.function2(a) |
| 49 | +``` |
| 50 | +can be replaced with |
| 51 | +```python |
| 52 | +from auto_function_serving.ServerHandler import AsyncserverHandler |
| 53 | +functionname = AsyncServerHandler(""" |
| 54 | +import module1 |
| 55 | +import module2 |
| 56 | +def functionname(someinput): |
| 57 | + a = module1.function1(someinput) |
| 58 | + return module2.function2(a) |
| 59 | +""", "functionname", port="Any") |
| 60 | +``` |
| 61 | +Decorators (@AsyncserverHandler.decorator and @ServerHandler.decorator) and AsyncServerHandler details in more usage. |
| 62 | + |
| 63 | +## Arguments |
| 64 | + |
| 65 | +```python |
| 66 | +from auto_function_serving.ServerHandler import ServerHandler |
| 67 | +callable_name = ServerHandler(""" |
| 68 | +some independent code with a callable |
| 69 | +""", "callable_name", port=None, backend='Popen', wait=100, backlog = 1024)) |
| 70 | +``` |
| 71 | +1. port |
| 72 | + * if None, then the input code is hashed and a port is chosen from 50000 to 60000 using the hash |
| 73 | + * if int, then int is chosen |
| 74 | + * otherwise, a random open port is chosen |
| 75 | +2. backend - either 'Popen' or 'multiprocessing'. Popen Should be used in general. |
| 76 | +3. wait - approx max number of seconds to wait for the server to run. No waiting done if set to 0, default 100 |
| 77 | +4. backlog - max number of backlogged requests before returning errors, python default is 5, but default in ServerHandler is 1024. |
| 78 | + |
| 79 | +## Features |
| 80 | + |
| 81 | +runs [http.server.HTTPServer](https://docs.python.org/3/library/http.server.html).\ |
| 82 | +ServerHandler and AsyncServerHandler objects can be loaded and unloaded with pickle.\ |
| 83 | +Uses Popen or multiprocessing to run the server.\ |
| 84 | +Uses only a single external dependency (aiohttp), and only for async.\ |
| 85 | +http, not https.\ |
| 86 | +chooses a port based on hash of input. (unless specified otherwise) |
| 87 | + |
| 88 | +### Advantages |
| 89 | + |
| 90 | +Minimal code changes.\ |
| 91 | +Should be compatible with almost all functions in almost all CPython envs. (Not sure where it could fail? Please add an issue if you find one.)\ |
| 92 | +Memory leaks or errors (from the server) are extremely unlikely since it is minimal, single threaded, single process and a default component of python stdlib.\ |
| 93 | +Exceptions cause 5xx errors without closing the server.\ |
| 94 | +Even Separate Processes will make requests to 1 instance of the same server unless specified otherwise. (Because it's looking for a server on a specific port.).\ |
| 95 | +Can specify otherwise by set the port to any free port so that a new ServerHandler object starts a new server.\ |
| 96 | +http post requests : lightweight, few ms overhead, reliable.\ |
| 97 | +Async is a good feature.\ |
| 98 | +now with tests. |
| 99 | + |
| 100 | +### Disadvatages |
| 101 | + |
| 102 | +Having a string of code as an argument to a class is not pythonic, unless the decorator is used.\ |
| 103 | +Importing inside functions is not ideal, even when the decorator is used.\ |
| 104 | +http post requests : insecure, few ms overhead.\ |
| 105 | +Exceptions inside the server are not sent back.\ |
| 106 | +No batching.\ |
| 107 | +No inbuilt logging. (Could be added). |
| 108 | +Initialization delay of upto few seconds to start the server. |
| 109 | +Async functions will not work on the server. |
| 110 | + |
| 111 | +#### Possible Edge cases |
| 112 | +No auto server restart in case server closes.\ |
| 113 | +May leave some resources locked for a while (<1min) if not closed properly.\ |
| 114 | +Problems might occur if Popen or multiprocessing are not available.\ |
| 115 | +Possible nested async errors with jupyter or other? Please look into [nest-asyncio](https://pypi.org/project/nest-asyncio/) and the [iss](https://github.com/python/cpython/issues/93462)[ues](https://github.com/python/cpython/issues/66435).\ |
| 116 | +Warnings from somewhat hacky (but legit and completely functional) workarounds. \ |
| 117 | +Closing of server process in __del__ and atexit.redister(__del__) fail for some reason (tested and unlikely). \ |
| 118 | + |
| 119 | + |
| 120 | +## Installation |
| 121 | + |
| 122 | +Use the package manager pip to install [auto_function_serving](https://pypi.org/project/auto-function-serving/) |
| 123 | +```bash |
| 124 | +pip install auto_function_serving |
| 125 | +``` |
| 126 | + |
| 127 | +## How does this work? |
| 128 | +Code for the server is stored in [ServerHandler](https://github.com/arrmansa/auto-function-serving/blob/main/src/auto_function_serving/ServerHandler.py).base_code and some string formatting is used to fill in the blanks.\ |
| 129 | +The server process is started with Popen (or multiprocessing if specified). The first thing it does is import socket and bind the port - if it's not available the code stops after an exception. Therefore only 1 instance of the server runs at a time on a machine.\ |
| 130 | +We know the function is ready after we can receive a valid get request from the server.\ |
| 131 | +Inputs and outputs are sent as bytes, converted to and from objects using pickle.\ |
| 132 | +If port is None in while initializing (default), a port from 50000 to 60000 is chosen by hashing the input code to make it independent of the source of a function. Collisions of different functions are possible, but unlikely. The collision of the same function in multiple processes is used to make sure only 1 server process runs at a time. The port can be specified if needed. |
| 133 | + |
| 134 | +## Performance (On my machine) |
| 135 | + |
| 136 | +overhead for small input and output (few bytes) - \ |
| 137 | +~2ms for requests with urllib.request\ |
| 138 | +~4ms for async requests with aiohttp.ClientSession \ |
| 139 | +overhead for large input and output\ |
| 140 | +~10ms for 0.5 mb input and output (1mb total transfer).\ |
| 141 | +~60ms for 5 mb input and output (10 mb total transfer).\ |
| 142 | +~600ms for 50 mb input and output (100 mb total transfer). |
| 143 | + |
| 144 | +## More Usage |
| 145 | + |
| 146 | +It can also be used with the provided decorator for functions with no dependencies outside the function. |
| 147 | +```python |
| 148 | +from auto_function_serving.ServerHandler import ServerHandler |
| 149 | +@ServerHandler.decorator |
| 150 | +def someheavyfunction(args,**kwargs): |
| 151 | + for i in range(big_number) |
| 152 | + someexpensivecomputation |
| 153 | +``` |
| 154 | +imports inside the function will work |
| 155 | +```python |
| 156 | +from auto_function_serving.ServerHandler import ServerHandler |
| 157 | +@ServerHandler.decorator |
| 158 | +def someheavyfunction(args,**kwargs): |
| 159 | + import numpy as np |
| 160 | +``` |
| 161 | +```python |
| 162 | +from auto_function_serving.ServerHandler import ServerHandler |
| 163 | +@ServerHandler.decorator |
| 164 | +def someheavyfunction(args,**kwargs): |
| 165 | + if not hasattr(someheavyfunction,'RunOnce'): |
| 166 | + global np |
| 167 | + import numpy as np |
| 168 | + setattr(someheavyfunction,'RunOnce',None) |
| 169 | + ... etc |
| 170 | +``` |
| 171 | + |
| 172 | +When the somemodule does not have any expensive global loading. |
| 173 | +```python |
| 174 | +from auto_function_serving.ServerHandler import ServerHandler |
| 175 | +from somemodule import someheavyfunction |
| 176 | +someheavyfunction = ServerHandler.decorator(someheavyfunction) |
| 177 | +``` |
| 178 | +Ip address can be changed by setting ServerHandler.ip_address (default "127.0.0.1") before creating a new instance. |
| 179 | + |
| 180 | +### AsyncServerHandler |
| 181 | +AsyncServerHandler is also available which uses [aiohttp](https://docs.aiohttp.org/) to make the requests asynchronously, for use with fastapi and other async use cases. \ |
| 182 | +AsyncServerHandler has the same usage as ServerHandler, except calls need to be awaited or used with asyncio.run() or with asyncio.get_event_loop().run_until_complete().\ |
| 183 | +Number of async calls can be limited by setting AsyncServerHandler.TCPConnector_limit which controls the [TCPconnector](https://docs.aiohttp.org/en/stable/client_reference.html?highlight=connector#aiohttp.TCPConnector) limit (default 100). Using [Semaphore](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore) is also something to consider. |
| 184 | + |
| 185 | +## Other things to look into |
| 186 | +Libraries : Celery, Tfserving, Torchserve, Flask\ |
| 187 | +Sending globals and locals to exec\ |
| 188 | +ast trees |
| 189 | + |
| 190 | +## Contributing |
| 191 | +Pull requests are welcome. |
| 192 | + |
| 193 | +## License |
| 194 | +[Apache License 2.0](https://choosealicense.com/licenses/apache-2.0/) |
0 commit comments