You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Pyper is a generalized framework for concurrent data-processing, based on functional programming patterns. Used for 🌐 **Data Collection**, 🔀 **ETL systems**, and general-purpose 🛠️ **Python Scripting**
12
28
13
-
See the [Documentation]()
29
+
See the [Documentation](https://pyper-dev.github.io/pyper/)
14
30
15
31
Key features:
16
32
17
-
* 💡**Intuitive Code**: Easy to learn, easy to think about. Implements intuitive abstractions to seamlessly unify threaded and asynchronous work.
18
-
* 🚀 **Functional Paradigm**: Python functions are the building blocks of data pipelines. Lets you write clean, reusable code without effort.
19
-
* 🛡️ **Safety**: Hides the heavy lifting of underlying task creation/execution. No more worrying about race conditions, memory leaks, and thread-level error handling.
33
+
* 💡**Intuitive API**: Easy to learn, easy to think about. Implements clean abstractions to seamlessly unify threaded and asynchronous work.
34
+
* 🚀 **Functional Paradigm**: Python functions are the building blocks of data pipelines. Let's you write clean, reusable code naturally.
35
+
* 🛡️ **Safety**: Hides the heavy lifting of underlying task creation and execution. No more worrying about race conditions, memory leaks, and thread-level error handling.
20
36
* ⚡ **Efficiency**: Designed from the ground up for lazy execution, using queues, workers, and generators.
21
37
* ✨ **Pure Python**: Lightweight, with zero sub-dependencies.
22
38
@@ -25,9 +41,11 @@ Key features:
25
41
Install the latest version using `pip`:
26
42
27
43
```console
28
-
$ pip install pyper
44
+
$ pip install python-pyper
29
45
```
30
46
47
+
(Note that `python-pyper` is the pypi registered package)
48
+
31
49
## Example
32
50
33
51
Let's simulate a pipeline that performs a series of transformations on some data.
@@ -68,6 +86,7 @@ async def print_sum(data):
68
86
69
87
70
88
asyncdefmain():
89
+
# Define a pipeline of tasks using `pyper.task`
71
90
run = task(step1) \
72
91
>> task(step2, concurrency=20) \
73
92
>> task(step3, concurrency=20) \
@@ -89,7 +108,9 @@ In our pipeline:
89
108
90
109
*`task(step3, concurrency=20)` spins up 20 threaded workers, taking each value as input and returning an output
91
110
92
-
The script therefore takes ~2 seconds to complete, as stage 2 and 3 in the pipeline only take the 1 second of sleep time, performed concurrently. If you'd like, experiment with tweaking the `limit` and `concurrency` values for yourself.
111
+
The script therefore takes ~2 seconds to complete, as `step2` and `step3` in the pipeline only take the 1 second of sleep time, performed concurrently. If you'd like, experiment with tweaking the `limit` and `concurrency` values for yourself.
112
+
113
+
---
93
114
94
115
<detailsmarkdown="1">
95
116
<summary><u>What does the logic translate to in non-concurrent code?</u></summary>
@@ -202,9 +223,9 @@ async def main():
202
223
await run(20) # takes ~2 seconds
203
224
```
204
225
205
-
This implementation achieves the basic desired concurrent data flow, but still lacks some quality-of-life features that Pyper takes care of, like thread-level error handling.
226
+
This implementation achieves the basic desired concurrent data flow, but still lacks some quality-of-life features that Pyper takes care of, like error handling within threads.
206
227
207
-
Pyper abstracts away the complexities of managing queues and workers, so that this code can be reduced to the two-line main function in the example above.
228
+
Pyper handles the complexities of managing queues and workers, so that this code can be reduced to the two-line main function in the example above.
208
229
209
230
</details>
210
231
@@ -248,22 +269,23 @@ def main():
248
269
>> task(step2, concurrency=20) \
249
270
>> task(step3, concurrency=20) \
250
271
& print_sum
272
+
# Run synchronously
251
273
run(limit=20)
252
274
253
275
254
276
if__name__=="__main__":
255
277
main() # takes ~2 seconds
256
278
```
257
279
258
-
A pipeline consisting of _at least one asynchronous function_ becomes an `AsyncPipeline`, which exposes the same logical function, provided `async` and `await` syntax in all of the obvious places. This makes it effortless to unify synchronously defined and asynchronously defined funcions where need be.
280
+
A pipeline consisting of _at least one asynchronous function_ becomes an `AsyncPipeline`, which exposes the same logical function, provided `async` and `await` syntax in all of the obvious places. This makes it effortless to unify synchronously defined and asynchronously defined functions where need be.
259
281
260
282
</details>
261
283
262
284
## Examples
263
285
264
286
To explore more of Pyper's features, see some real-world examples below:
0 commit comments