Skip to content

Commit 9f7655d

Browse files
authored
Added testig and FastAPI sections in docs. (#83)
1 parent 76bb6c7 commit 9f7655d

File tree

11 files changed

+2075
-1371
lines changed

11 files changed

+2075
-1371
lines changed

docs/.vuepress/public/favicon.ico

9.54 KB
Binary file not shown.

docs/.vuepress/styles/index.scss

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,17 @@
22
visibility: hidden;
33
}
44

5-
.hero{
6-
img{
7-
max-width: 55% !important;
8-
padding: 2rem;
5+
.hero-info-wrapper {
6+
img {
7+
max-width: 60% !important;
8+
padding: 1rem;
99
}
1010
}
1111

12+
.actions {
13+
align-items: flex-end;
14+
}
15+
1216
span.site-name {
1317
visibility: hidden;
1418
}

docs/examples/introduction/inmemory_run.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ async def add_one(value: int) -> int:
1212

1313

1414
async def main() -> None:
15+
# Never forget to call startup in the beginning.
1516
await broker.startup()
1617
# Send the task to the broker.
1718
task = await add_one.kiq(1)

docs/examples/testing/main_file.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import os
2+
3+
from taskiq import AsyncBroker, InMemoryBroker, ZeroMQBroker
4+
5+
env = os.environ.get("ENVIRONMENT")
6+
7+
broker: AsyncBroker = ZeroMQBroker()
8+
9+
if env and env == "pytest":
10+
broker = InMemoryBroker()

docs/guide/README.md

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,14 @@ Taskiq is a library that helps you send and process python functions in a distri
1212
For example, you have many heavy to calculate functions you want to execute on another server.
1313
You can implement interservice communication by yourself, or you can use Taskiq to make the job done easily.
1414

15-
The core library has only two brokers. It provides CLI, basic functionality for creating tasks, and abstractions to extend functionality. The main idea is to make taskiq modular with a lot of
16-
replaceable parts, such as brokers, result backends, and middlewares.
15+
The core library doesn't have much functionality. It provides two built-in brokers, CLI, basic functionality for creating distributed tasks, and abstractions to extend the taskiq. The main idea of taskiq is to make it modular and easy to extend. We have libraries for many
16+
possible use cases, but if you lack something, you can adopt taskiq to fit your needs.
1717

1818
## Why not use existing libraries?
1919

20-
We couldn't find a solution like Celery or Dramatiq that can run async code and send tasks asynchronously.
21-
It was the main reason we created this project.
20+
We created this project because we couldn't find any project that can execute and send async functions using distributed queues like RabbitMQ.
2221

23-
You might have seen projects built on top of asyncio that solve similar problem, but here's a comparasion table of taskiq and other projects.
22+
You might have seen projects built on top of asyncio that solve a similar problem, but here's a comparison table of the taskiq and other projects.
2423

2524
| Feature name | Taskiq | Arq | AioTasks |
2625
| --------------------------: | :----: | :---: | :------: |

docs/guide/getting-started.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ After installation of the core library, you need to find the broker that fits yo
2929

3030
::: info Cool tip!
3131

32-
We highly recommend [taskiq-aio-pika](https://pypi.org/project/taskiq-aio-pika/) as the broker and [taskiq-redis](https://pypi.org/project/taskiq-redis/) as the result backend for production use.
32+
We highly recommend [taskiq-aio-pika](https://pypi.org/project/taskiq-aio-pika/) or [taskiq-nats](https://pypi.org/project/taskiq-nats/) as the broker and [taskiq-redis](https://pypi.org/project/taskiq-redis/) as the result backend for production use.
3333

3434
:::
3535

@@ -57,9 +57,9 @@ And that's it. Now let's add some tasks and the main function. You can add tasks
5757

5858
@[code python](../examples/introduction/inmemory_run.py)
5959

60-
::: tip Cool tip!
60+
::: warning Cool warning!
6161

62-
Calling the `startup` method is not required, but we strongly recommend you do so.
62+
Calling the `startup` method is necessary. If you don't call it, you may get an undefined behaviour.
6363

6464
:::
6565

@@ -74,7 +74,7 @@ Returned value: 2
7474
Ok, the code of the task execution is a little bit fancier than an ordinary function call, but it's still relatively simple to understand. To send a task to the broker,
7575
you need to call the `.kiq` method on the function,
7676
it returns the `TaskiqTask` object that can check whether the result is ready
77-
or it can wait for it to become available.
77+
or not. Also it has methods to wait for the result to become available.
7878

7979
You can get more information about taskiq types, CLI and internal structure in the "[Architecture overview](./architecture-overview.md)" section.
8080

docs/guide/scheduling-tasks.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@ order: 8
55
# Scheduling tasks
66

77
Sometimes you may want to execute some tasks according to some schedule.
8-
For example, you want to call a function every day at 2 pm.
8+
For example, you maybe want to call a function every day at 2 pm.
99

10-
It's easy to do with taskiq. We have primitives that can help you to solve your problems.
10+
That's not a problem if you use taskiq. We have primitives that can help you to solve your problems.
1111

1212
Let's imagine we have a module, as shown below, and we want to execute the `heavy_task` every 5 minutes.
1313
What should we do?

docs/guide/taskiq-with-fastapi.md

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
---
2+
order: 10
3+
---
4+
5+
# Taskiq + FastAPI
6+
7+
FastAPI is one of the most popular async web frameworks in python. It has gained it's popularity because of two things:
8+
1. It's easy to use;
9+
2. It has a cool dependency injection.
10+
11+
In taskiq we try to make our libraries easy to use and we too have a depenndency injection. But our dependencies
12+
are not compatible with FastAPI's dependencies by default. That is why we have created a library "[taskiq-fastapi](https://pypi.org/project/taskiq-fastapi/)" to make integration with
13+
FastAPI as smooth as possible.
14+
15+
Let's see what we got here. In this library, we provide you with only one public function called `init`. It takes a broker and a string path (as in uvicorn) to the fastapi application (or factory function). This function must be called in your main broker file.
16+
17+
18+
```python
19+
from taskiq import ZeroMQBroker
20+
import taskiq_fastapi
21+
22+
broker = ZeroMQBroker()
23+
24+
taskiq_fastapi.init(broker, "my_package.application:app")
25+
26+
```
27+
28+
There are two rules to make everything work as you expect:
29+
1. Add `TaskiqDepends` as a default value for every parameter with `Request` or `HTTPConnection` types in base dependencies.
30+
2. Use only `TaskiqDepends` in tasks.
31+
32+
33+
::: tip Cool and important note!
34+
35+
The Request or HTTPConnection that you'll get injected in your task is not the same request or connection you have had in your handler when you were sending the task!
36+
37+
:::
38+
39+
Many fastapi dependency functions are depend on `fastapi.Request`. We provide a mocked request to such dependencies. But taskiq cannot resolve dependencies until you explicitly specify that this parameter must be injected.
40+
41+
As an example. If you previously had a dependency like this:
42+
43+
```python
44+
from fastapi import Request
45+
from typing import Any
46+
47+
def get_redis_pool(request: Request) -> Any:
48+
return request.app.state.redis_pool
49+
50+
```
51+
52+
To make it resolvable in taskiq, you need to make it clear that Request object must be injected. Like this:
53+
54+
```python
55+
from fastapi import Request
56+
from taskiq import TaskiqDepends
57+
58+
59+
async def get_redis_pool(request: Request = TaskiqDepends()):
60+
return request.app.state.redis_pool
61+
62+
```
63+
64+
65+
Also you want to call startup of your brokers somewhere.
66+
67+
```python
68+
from fastapi import FastAPI
69+
from your_project.taskiq import broker
70+
71+
app = FastAPI()
72+
73+
74+
@app.on_event("startup")
75+
async def app_startup():
76+
if not broker.is_worker_process:
77+
await broker.startup()
78+
79+
80+
@app.on_event("shutdown")
81+
async def app_shutdown():
82+
if not broker.is_worker_process:
83+
await broker.shutdown()
84+
85+
```
86+
87+
And that's it. Now you can use your taskiq tasks with functions and classes that depend on FastAPI dependenices. You can find bigger examples in the [taskiq-fastapi repo](https://github.com/taskiq-python/taskiq-fastapi).

docs/guide/testing-taskiq.md

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
---
2+
order: 9
3+
---
4+
5+
# Testing with taskiq
6+
7+
Everytime we write programs, we want them to be correct. To achieve this, we use tests.
8+
Taskiq allows you to write tests easily as if tasks were normal functions.
9+
10+
Let's dive into examples.
11+
12+
## Preparations
13+
14+
### Environment setup
15+
For testing you maybe don't want to use actual distributed broker. But still you want to validate your logic.
16+
Since python is an interpreted language, you can easily replace you broker with another one if the expression is correct.
17+
18+
We can set an environment variable, that indicates that currently we're running in testing environment.
19+
20+
::: tabs
21+
22+
@tab linux|macos
23+
24+
25+
```bash
26+
export ENVIRONMENT="pytest"
27+
pytest -vv
28+
```
29+
30+
@tab windows
31+
32+
```powershell
33+
$env:ENVIRONMENT = 'pytest'
34+
pytest -vv
35+
```
36+
37+
:::
38+
39+
40+
Or we can even tell pytest to set this environment for us, just before executing tests using [pytest-env](https://pypi.org/project/pytest-env/) plugin.
41+
42+
::: tabs
43+
44+
@tab pytest.ini
45+
46+
```ini
47+
[pytest]
48+
env =
49+
ENVIRONMENT=pytest
50+
```
51+
52+
@tab pyproject.toml
53+
54+
```toml
55+
[tool.pytest.ini_options]
56+
env = [
57+
"ENVIRONMENT=pytest",
58+
]
59+
```
60+
61+
:::
62+
63+
### Async tests
64+
65+
Since taskiq is fully async, we suggest using [anyio](https://anyio.readthedocs.io/en/stable/testing.html) to run async functions in pytest. Install the [lib](https://pypi.org/project/anyio/) and place this fixture somewhere in your root `conftest.py` file.
66+
67+
```python
68+
@pytest.fixture
69+
def anyio_backend():
70+
return 'asyncio'
71+
```
72+
73+
After the preparations are done, we need to modify the broker's file in your project.
74+
75+
@[code python](../examples/testing/main_file.py)
76+
77+
As you can see, we added an `if` statement. If the expression is true, we replace our broker with an imemory broker.
78+
The main point here is to not have an actual connection during testing. It's useful because inmemory broker has
79+
the same interface as a real broker, but it doesn't send tasks acutally.
80+
81+
## Testing tasks
82+
83+
Let's define a task.
84+
85+
```python
86+
from your_project.taskiq import broker
87+
88+
@broker.task
89+
async def parse_int(val: str) -> int:
90+
return int(val)
91+
```
92+
93+
This simple task may be defined anywhere in your project. If you want to test it,
94+
just import it and call as a normal function.
95+
96+
```python
97+
import pytest
98+
from your_project.tasks import parse_int
99+
100+
@pytest.mark.anyio
101+
async def test_task():
102+
assert await parse_int("11") == 11
103+
```
104+
105+
And that's it. Test should pass.
106+
107+
What if you want to test a function that uses task. Let's define such function.
108+
109+
```python
110+
from your_project.taskiq import broker
111+
112+
@broker.task
113+
async def parse_int(val: str) -> int:
114+
return int(val)
115+
116+
117+
async def parse_and_add_one(val: str) -> int:
118+
task = await parse_int.kiq(val)
119+
result = await task.wait_result()
120+
return result.return_value + 1
121+
```
122+
123+
And since we replaced our broker with `InMemoryBroker`, we can just call it.
124+
It would work as you expect and tests should pass.
125+
126+
```python
127+
@pytest.mark.anyio
128+
async def test_add_one():
129+
assert await parse_and_add_one("11") == 12
130+
```
131+
132+
## Dependency injection
133+
134+
If you use dependencies in your tasks, you may think that this can become a problem. But it's not.
135+
Here's what we came up with. We added a method called `add_dependency_context` to the broker.
136+
It sets base dependencies for dependency resolution. You can use it for tests.
137+
138+
Let's add a task that depends on `Path`. I guess this example is not meant to be used in production code bases, but it's suitable for illustration purposes.
139+
140+
```python
141+
from pathlib import Path
142+
from taskiq import TaskiqDepends
143+
144+
from your_project.taskiq import broker
145+
146+
147+
@broker.task
148+
async def modify_path(some_path: Path = TaskiqDepends()):
149+
return some_path.parent / "taskiq.py"
150+
151+
```
152+
153+
To test the task itself, it's not different to the example without dependencies, but we jsut need to pass all
154+
expected dependencies manually as function's arguments or key-word arguments.
155+
156+
```python
157+
import pytest
158+
from your_project.taskiq import broker
159+
160+
from pathlib import Path
161+
162+
@pytest.mark.anyio
163+
async def test_modify_path():
164+
modified = await modify_path(Path.cwd())
165+
assert str(modified).endswith("taskiq.py")
166+
167+
```
168+
169+
But what if we want to test task execution? Well, you don't need to provide dependencies manually, you
170+
must mutate dependency_context before calling a task. We suggest to do it in fixtures.
171+
172+
```python
173+
import pytest
174+
from your_project.taskiq import broker
175+
from pathlib import Path
176+
177+
178+
# We use autouse, so this fixture
179+
# is called automatically before all tests.
180+
@pytest.fixture(scope="function", autouse=True)
181+
async def init_taskiq_dependencies():
182+
# Here we use Path, but you can use other
183+
# pytest fixtures here. E.G. FastAPI app.
184+
broker.add_dependency_context({Path: Path.cwd()})
185+
186+
yield
187+
188+
# After the test we clear all custom dependencies.
189+
broker.custom_dependency_context = {}
190+
191+
```
192+
193+
This fixture will update dependency context for our broker before
194+
every test. Now tasks with dependencies can be used. Let's try it out.
195+
196+
```python
197+
@pytest.mark.anyio
198+
async def test_modify_path():
199+
task = await modify_path.kiq()
200+
result = await task.wait_result()
201+
assert str(result.return_value).endswith("taskiq.py")
202+
203+
```
204+
205+
This should pass. And that's it for now.

0 commit comments

Comments
 (0)