Database transactions with asyncio

[dgilge]

The newly released Channels 2.0 Django package "relies on the asyncio library and native Python async support". They recommend to use py.test tests with the asyncio plugin.

Now, you can use the db and django_db fixtures with asyncio (when you use Channels' database_sync_to_async wrapper):

def create_my_model():
    return MyModel.objects.create(name='foo')

@pytest.fixture
async def add_my_model_to_db(db):
    return await database_sync_to_async(create_my_model)()

@pytest.mark.asyncio
async def test_my_db_modification(add_my_model_to_db):
    assert add_my_model_to_db.name == 'foo'
    # ...

However, transactions don't seem to work. As a result all changes to the test database will be kept.

"Note that you can’t mix this with unittest.TestCase subclasses" (which pytest-django does internally – but I don't know if this is of importance concerning this issue, because another asyncio issue seems to be related).

Is there a way to use transactions in an async setting with the current packages? Or are there plans to support transactions in async tests in the near future? Or should this be done in another package?

[vikashtank]

Did you ever found a way around this? It seems like adding the asyncio library is affecting non-async tests as well.

[bluetech]

Possibly related: https://code.djangoproject.com/ticket/32409

Does this still happen with asgiref>=3.3.0?

[willstott101]

Yes, I'm seeing this with asgiref==3.4.1

[seifertm]

There's a corresponding issue in pytest-asyncio that's been lying around for some time. I'm unsure if I can close it, so this is why I'm reaching out.

My current understanding is that pytest-django is a "pytest wrapper" around Django's unittest test cases. Is this correct?

However, pytest-asyncio does not work together with unittest as of v0.18.3 (see the README, and pytest-dev/pytest-asyncio#77).

Is there anything that needs to be done on the pytest-asyncio side to solve this?

[Jenselme]

I think I encountered this issue today. Marking the tests with @pytest.mark.django_db(transaction=True) instead of @pytest.mark.django_db() solved it for me.

[KlemenS189]

@Jenselme It works for me as well. The problem is that marking it with transaction=True slows down the tests. If you have many tests, this is not negligible.

[Jenselme]

Indeed, I noticed that also. I might be reverting to using the sync HTTP client and wrap async operations in async_to_sync where needed to prevent this.

I had a look at TestCase from Django which isn’t subject to this drops in performance and they wrap async tests into async_to_sync.

I’d like for a cleaner way to do this in pytest, but until then, making all test sync seem like a good alternative.

[egorgam]

Looks like a problem is because django async orm don't support atomic blocks. Pytest-django tests isolation logic bases on wrapping each test (and test fixtures) in @transaction.atomic. But async client opens new event loop inside, with self management logic.

[rtpg]

Wrapping your async test in @async_to_sync is a very quick fix for this issue (I had to make a wrapper decorator for this because of pytest fixture nonsense but...)

Here's a note I have on this issue:

        #   I believe this to be an issue with pytest-django and how it does things
        #   in _django_db_helper, probably skipping over some transactional work
        #   (potentially playing not nicely with pytest-asyncio....)

Some other things I have seen made me convinced that pytest-django doesn't properly close out things in this case, and I think the code sharing it's trying to do with TestCase is too clever by a half, causing this stuff to half-work, half-not work.

Thu Aug 29 04:42:47 2024:       /workspace:0: PytestWarning: Error when trying to teardown test databases: OperationalError('database "test_testdb_dbname" is being accessed by other users\nDETAIL:  There is 1 other session using the database.')

^ if you see stuff like this in your logs this is also very likely to be this same issue.

[lode-braced]

I also encountered these cases, #1223 has some explanation and a potential fix to async tests: if you start the transaction in the asgiref thread executor that will be used during the async tests, you do get the desired rollback of test changes.

For those watching this thread: feel free to try out the branch and let me know if that fixes using the db (not transactional_db) fixture, or the mark with transaction=False in your async tests.