You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I first learned about [testcontainers](https://testcontainers.com/){:target="\_blank"}, I wanted to know how to integrate it with [`asyncpg`](https://magicstack.github.io/asyncpg/current/){:target="\_blank"}, an asynchronous driver for PostgreSQL. At first look, I came across [this text](https://www.linkedin.com/pulse/utilizando-testcontainers-fastapi-guilherme-de-carvalho-carneiro-9cmlf/){:target="\_blank"} from[Guilherme](https://www.linkedin.com/in/guilhermecarvalho/){:target="\_blank"}, who worked on it instantly. Based on that, I decided to write this simple repository with an example application.
10
+
When I first got to know about [testcontainers](https://testcontainers.com/){:target="\_blank"}, I wanted to learn how to integrate it with [`asyncpg`](https://magicstack.github.io/asyncpg/current/){:target="\_blank"}, an asynchronous driver for PostgreSQL, for testing asynchronous routes in FastAPI. In an initial reference search, I found [this article](https://www.linkedin.com/pulse/utilizando-testcontainers-fastapi-guilherme-de-carvalho-carneiro-9cmlf/){:target="\_blank"} by[Guilherme](https://www.linkedin.com/in/guilhermecarvalho/){:target="\_blank"}, and based on his article, I decided to write this example application.
11
11
12
12
You can check the complete repository [here](https://github.com/lealre/fastapi-testcontainer-asyncpg){:target="\_blank"}.
13
13
14
-
TL;DR: The full `conftest.py` setup is available [here](Add link here).
14
+
TL;DR: The full `conftest.py` setup is available [here](#final-version-of-test-fixtures).
15
15
16
16
## Testcontainers
17
17
18
-
Testcontainers is an open-source library for providing lightweight instances of anything that can run in a Docker container. It was originally implemented for .NET, Go, Java, and Node.js, but it was extended to other programming languages through community projects, including Python: [testcontainer-python docs](https://testcontainers-python.readthedocs.io/en/latest/){:target="\_blank"}.
18
+
Testcontainers is an open-source library for providing lightweight instances of anything that can run in a Docker container. It was originally implemented for .NET, Go, Java, and Node.js but has since been extended to other programming languages through community projects, including Python: [testcontainer-python](https://testcontainers-python.readthedocs.io/en/latest/){:target="\_blank"}.
19
19
20
-
Below is the documentation example of how to use an instance of PostgreSQL, which uses [`psycopg2`](https://github.com/psycopg/psycopg2){:target="\_blank"} as the default driver.
20
+
Below is a documentation example of how to use an instance of PostgreSQL, which uses [`psycopg2`](https://github.com/psycopg/psycopg2){:target="\_blank"} as the default driver.
@@ -36,7 +36,7 @@ Below is the documentation example of how to use an instance of PostgreSQL, whic
36
36
37
37
The objective of this repository is to test asynchronous FastAPI endpoints using PostgreSQL as a database. To achieve that, besides the `testcontainers`, it uses [`pytest`](https://docs.pytest.org/en/stable/){:target="\_blank"} and [`anyio`](https://anyio.readthedocs.io/en/stable/testing.html){:target="\_blank"}, which provides built-in support for testing applications in the form of a pytest plugin. The choice of `anyio` over `pytest-asyncio` is because FastAPI is based on Starlette, which uses AnyIO, so we don't need to install an additional package here.
38
38
39
-
The development of the routes uses [aiosqlite](https://aiosqlite.omnilib.dev/en/stable/){:target="\_blank"}, the async driver for SQLite.
39
+
The development of the API routes uses [aiosqlite](https://aiosqlite.omnilib.dev/en/stable/){:target="\_blank"}, the async driver for SQLite.
40
40
41
41
Below are all the dependencies used to run the example.
42
42
@@ -68,7 +68,7 @@ Below is how the example is structured:
68
68
1. Where the example API is written using FastAPI.
69
69
2. Where API test fixtures are written, from the PostgreSQL instance to the client. You can learn more about the `conftest.py` file in the <ahref="https://docs.pytest.org/en/stable/reference/fixtures.html#conftest-py-sharing-fixtures-across-multiple-files"target="_blank">pytest docs</a>.
70
70
71
-
## API routes example
71
+
## API routes implementation
72
72
73
73
This section will show the endpoints created for later tests. For this example, three simple routes were created to simulate a ticketing sell system:
The `database.py` contains the connection with the database, as also the `get_session()` generator, responsible for creating asynchronous sessions for performing transactions in the database.
97
+
The `database.py` contains the database connection, as well as the `get_session()` generator, responsible for creating asynchronous sessions to perform transactions in the database.
The last file before to create the routes is the `schemas.py`, that will contain all the [Pydantic](https://docs.pydantic.dev/latest/){:target="\_blank"} models.
126
+
The last file before creating the routes is the `schemas.py`, which will contain all the [Pydantic](https://docs.pydantic.dev/latest/){:target="\_blank"} models.
127
127
128
128
```py title="src/schemas.py" linenums="1"
129
129
from pydantic import BaseModel
@@ -152,9 +152,9 @@ class ListTickets(BaseModel):
152
152
tickets: list[TicketResponse]
153
153
```
154
154
155
-
The previous three files are imported in `app.py`, which contains the API routes for this example. As mentioned before, although the objective is to test the endpoints in a PostgreSQL database, the development of the API is done using SQLite to avoid the necessity of having an instance of PostgreSQL running all the time.
155
+
The previous three files are imported in `app.py`, which contains the API routes for this example. As mentioned earlier, although the objective is to test the endpoints with a PostgreSQL database, the development of the API uses SQLite to avoid the need for a PostgreSQL instance running constantly.
156
156
157
-
To keep it simple and avoid using database migrations, the database creation is handled using the [lifespan events](https://fastapi.tiangolo.com/advanced/events/){:target="\_blank"}. Here, it will just guarantee that every time we run the application, a database will be created in case it doesn't exist yet.
157
+
To keep things simple and avoid database migrations, the database creation is handled using [lifespan events](https://fastapi.tiangolo.com/advanced/events/){:target="\_blank"}. This guarantees that every time we run the application, a database will be created if it doesn't already exist.
Below are the routes implementation, as also the `SessionDep` to be used as [dependency injection](https://fastapi.tiangolo.com/tutorial/dependencies/){:target="\_blank"} in each route.
188
+
Below are the route implementations, as well as the `SessionDep` to be used as [dependency injection](https://fastapi.tiangolo.com/tutorial/dependencies/){:target="\_blank"} in each route.
Now, running the command below in the terminal, the application should be available at `http://127.0.0.1:8000`.
265
+
Now, by running the command below in the terminal, the application should be available at `http://127.0.0.1:8000`.
266
266
267
-
```bash
268
-
fastapi dev src/app.py
269
-
```
267
+
=== "pip"
268
+
269
+
```bash
270
+
python -m fastapi dev src/app.py
271
+
```
272
+
273
+
=== "uv"
274
+
275
+
``` bash
276
+
uv run -m fastapi dev src/app.py
277
+
```
270
278
271
279
## Tests workflow
272
280
273
281
To use PostgreSQL in the tests, the testcontainer will be set up in `conftest.py`, along with the database session and the client required to test the endpoints.
274
282
275
-
Below is a simple diagram illustrating how it works for each test, where each block represents a different function.
283
+
Below is a simple diagram illustrating how it works **for each test**, where each block represents a different function.
276
284
277
285
```mermaid
278
286
flowchart LR
@@ -300,7 +308,7 @@ flowchart LR
300
308
class postgres_container,async_session,async_client,test fixtureStyle;
301
309
```
302
310
303
-
The `postgres_container` will be passed to `async_session`, which will be used in both `async_client` and directly in the tests in some cases where we need to transact directly with the database.
311
+
The `postgres_container` will be passed to `async_session`, which will be used in both `async_client` and directly in the tests, in cases where we need to transact directly with the database.
304
312
305
313
## Creating the test fixtures
306
314
@@ -326,9 +334,9 @@ def anyio_backend():
326
334
return'asyncio'
327
335
```
328
336
329
-
Now, in the `postgres_container`, the `anyio_backend` is passed, and all the tests that use the `postgres_container` as a fixture at some level will be marked to run asynchronously.
337
+
Now, in the `postgres_container`, the `anyio_backend` is passed, and all the tests that use the `postgres_container` as a fixture at any level will be marked to run asynchronously.
330
338
331
-
Below is the `postgres_container` function, which will be responsible for creating the `PostgresContainer` instance from `testcontainers`. The `asyncpg` is passed as an argument in the class to specify that this will be the driver used.
339
+
Below is the `postgres_container` function, which will be responsible for creating the `PostgresContainer` instance from `testcontainers`. The `asyncpg`driver is passed as an argument to specify that it will be the driver used.
The `async_session` takes the connection URL from the `PostgresContainer` object returned by the `postgres_container` function and uses it to create the tables inside the database, as well as the session that will establish all interactions with the PostgreSQL instance created. The function will return and persist a session to be used, and then restore the database for the next test by deleting the tables.
348
+
The `async_session` takes the connection URL from the `PostgresContainer` object returned by the `postgres_container` function and uses it to create the tables inside the database, as well as the session that will handle all interactions with the PostgreSQL instance created. The function will return and persist a session to be used, and then restore the database for the next test by deleting the tables.
The last fixture is the `async_client` function, which will create the [`AsyncClient`](https://fastapi.tiangolo.com/advanced/async-tests/), directly imported from [HTTPX](https://www.python-httpx.org/), and provide it to make requests to our endpoints. Here, the session provided by `async_session` will override the session originally used in our app as dependency injection while the client is being used.
373
+
The last fixture is the `async_client` function, which will create the [`AsyncClient`](https://fastapi.tiangolo.com/advanced/async-tests/), directly imported from [HTTPX](https://www.python-httpx.org/), and provide it to make requests to our endpoints. Here, the session provided by `async_session` will override the session originally used in our app as a dependency injection while the client is being used.
In total there are 6 test, and the rest of them has the same logic. Their full implementations can be checked in the repository.
427
435
436
+
Adding the following setting in `pyproject.toml` or `pytest.ini` will inform pytest to add the root directory to the Python search path when running tests.
437
+
438
+
=== "pyproject.toml"
439
+
440
+
``` toml
441
+
[tool.pytest.ini_options]
442
+
pythonpath = '.'
443
+
```
444
+
445
+
=== "pytest.ini"
446
+
447
+
```
448
+
[pytest]
449
+
pythonpath = .
450
+
```
451
+
428
452
Now, if we run the following command in the terminal...
============================================= 6 passed in 12.31s =============================================
445
477
```
446
478
447
-
Although all the tests are very simple, it took an average of more than two seconds for each one of them to execute.
479
+
Although all the tests are very simple, it took an average of more than two seconds for each one of them to execute. This happens because for each test, a new PostgreSQL Docker instance is being created, as shown in [Tests workflow](#tests-workflow).
480
+
481
+
To make the tests faster, one option is to create just one PostgreSQL Docker instance and use it for all the tests by configuring the `@pytest.fixture(scope='')`.
482
+
483
+
## The pytest fixture scope
484
+
485
+
Fixtures requiring network access depend on connectivity and are usually time-expensive to create. By setting the `scope` in `@pytest.fixture`, we can tell pytest how to manage the fixture's creation and reuse.
486
+
487
+
Fixtures are created when first requested by a test and are destroyed based on their `scope`. Some of the scope options that can be set are:
488
+
489
+
-`function`: the default scope, the fixture is destroyed at the end of the test.
490
+
-`class`: the fixture is destroyed during the teardown of the last test in the class.
491
+
-`module`: the fixture is destroyed during the teardown of the last test in the module.
492
+
-`package`: the fixture is destroyed during the teardown of the last test in the package.
493
+
-`session`: the fixture is destroyed at the end of the test session.
494
+
495
+
As we want to create just one Docker instance and reuse it for all the tests, we changed the `@pytest.fixture` in the `conftest.py` file in the following highlighted lines.
with PostgresContainer('postgres:16', driver='asyncpg') as postgres:
506
+
yield postgres
507
+
```
508
+
509
+
Now, every time we run the tests, we will have a similar workflow like the one below, where the `postgres_container` fixture will be created at the beginning of the test session, persist to be used in all the other fixtures, and be destroyed only when all the tests finish.
510
+
511
+
```mermaid
512
+
flowchart LR
513
+
%% Nodes for fixtures
514
+
postgres_container["postgres_container"]
515
+
async_session["async_session"]
516
+
async_client["async_client"]
517
+
test["Test 1"]
518
+
async_session_2["async_session"]
519
+
async_client_2["async_client"]
520
+
test_2["Test 2"]
521
+
async_session_n["async_session"]
522
+
async_client_n["async_client"]
523
+
test_n["Test N"]
524
+
525
+
subgraph All fixtures
526
+
direction LR
527
+
528
+
subgraph Function fixtures
529
+
direction LR
530
+
async_session --> async_client
531
+
async_session_2 --> async_client_2
532
+
async_session_n --> async_client_n
533
+
end
534
+
535
+
subgraph Session Fixture
536
+
direction LR
537
+
postgres_container --> async_session
538
+
postgres_container --> async_session_2
539
+
postgres_container --> async_session_n
540
+
end
541
+
end
542
+
543
+
%% Arrows from async_client to test blocks
544
+
async_client --> test
545
+
async_session --> test
546
+
547
+
async_client_2 --> test_2
548
+
async_session_2 --> test_2
549
+
async_client_n --> test_n
550
+
async_session_n --> test_n
551
+
552
+
553
+
%% Style the nodes with rounded corners
554
+
classDef fixtureStyle rx:10, ry:10;
555
+
556
+
%% Style the nodes
557
+
class postgres_container,async_session,async_client,test fixtureStyle;
558
+
class async_session_2,async_client_2,test_2 fixtureStyle;
559
+
class async_session_n,async_client_n,test_n fixtureStyle;
560
+
```
448
561
449
-
## The pytest scope
562
+
Running the tests again, we should see that the time it took to run all the tests decreases to around 4 seconds, with a median of less than one second per test.
============================================= 6 passed in 3.94s =============================================
460
573
```
461
574
462
-
## Final `conftest.py`
575
+
??? note "Documentation reference links"
576
+
577
+
- [Scope: sharing fixtures across classes, modules, packages or session](https://docs.pytest.org/en/6.2.x/fixture.html#scope-sharing-fixtures-across-classes-modules-packages-or-session)
- [Higher-scoped fixtures are executed first](https://docs.pytest.org/en/stable/reference/fixtures.html#higher-scoped-fixtures-are-executed-first)
463
580
464
-
## Conclusion
581
+
## Final version of test fixtures
465
582
466
-
ERROR tests/test_routes.py::test_get_all_tickets_success - docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
583
+
```py title="tests/conftest.py" linenums="1"
584
+
import pytest
585
+
from httpx import ASGITransport, AsyncClient
586
+
from sqlalchemy.ext.asyncio import (
587
+
AsyncSession,
588
+
async_sessionmaker,
589
+
create_async_engine,
590
+
)
591
+
from testcontainers.postgres import PostgresContainer
592
+
593
+
from src.app import app
594
+
from src.database import get_session
595
+
from src.models import table_register
596
+
597
+
598
+
@pytest.fixture(scope='session')
599
+
defanyio_backend():
600
+
return'asyncio'
601
+
602
+
603
+
@pytest.fixture(scope='session')
604
+
defpostgres_container(anyio_backend):
605
+
with PostgresContainer('postgres:16', driver='asyncpg') as postgres:
0 commit comments