Manual rollback of committed data in tests

[mesemus]

Description

The current db fixture in pytest-invenio uses nested transactions (session.begin_nested()) to isolate test data. However, this approach fails when tests or invenio code explicitly call db.session.commit(), as this causes the data to be committed into the outermost transaction and not the nested one (change from sqlalchemy 1.x where the same operation committed the nested transaction). This causes:

1. Data leakage between tests: Committed data from one test remains in the database and affects subsequent tests
2. Test isolation failures: Tests that expect a clean database state encounter unexpected data
3. Database inconsistency: Module-level fixtures may have their data removed by tests, breaking subsequent tests

This PR introduces a manual database cleanup mechanism that:

1. Captures initial database state: Before each test, stores all primary key values from all tables using store_database_values() (this preserves module- scoped db rows from cleanup)
2. Performs explicit cleanup: After each test, identifies and deletes any rows not present in the initial snapshot using purge_database_values()
3. Handles foreign key constraints: Implements a retry mechanism to delete rows in the correct order when foreign key constraints exist
4. Detects database inconsistency: Raises InconsistentDatabaseError if tests remove rows from module-level fixtures, making explicit what would otherwise be silent test pollution
5. Detects connection leaks: Detects if there are any unclosed db connections after the tests

The package was tested with all invenio packages . All tests were successful with the exception of:

• invenio-communities - the tests incorrectly modify module-level db state, PR prepared.
• invenio-files-rest - the test kept connections to db open, PR prepared
• invenio-rdm-records - failing on different error, PR prepared
• invenio-record-files - PR in progress by @fenekku

With those patches, all tests pass.

Alternative approaches tested and rejected

An alternative approach would be to override commit and rollback so that they commit/rollback the nested transaction instead of the outer one and start a new nested transaction automatically. This approach seems not to be working:

• some default values are handled differently by sqlalchemy/postgresql in nested and outer transactions. With this approach, I get (on the same sources) errors like:  

FAILED tests/members/test_members_resource.py::test_invite - 
ValueError: Not naive datetime (tzinfo is already set)

Checklist

Ticks in all boxes and 🟢 on all GitHub actions status checks are required to merge:

• I'm aware of the code of conduct.
• I've created logical separate commits and followed the commit message format.
• I've added relevant test cases.
• I've added relevant documentation.
• I've marked translation strings.
• I've identified the copyright holder(s) and updated copyright headers for touched files (>15 lines contributions).
• I've NOT included third-party code (copy/pasted source code or new dependencies).
  • If you have added third-party code (copy/pasted or new dependencies), please reach out to an architect on Discord.

Frontend

• I've followed the CSS/JS and React guidelines.
• I've followed the web accessibility guidelines.
• I've followed the user interface guidelines.

Reminder

By using GitHub, you have already agreed to the GitHub’s Terms of Service including that:

1. You license your contribution under the same terms as the current repository’s license.
2. You agree that you have the right to license your contribution under the current repository’s license.

[fenekku]

Can you provide the motivating event for this?

Before the holiday break (for me December 6 or so) I didn't encounter any data leakage or too many clients problem with:

    Python 3.14.0, pytest-8.4.2
      psycopg2-binary==2.9.11
      SQLAlchemy==2.0.44
      SQLAlchemy-Continuum==1.5.2
      SQLAlchemy-Utils==0.41.2

e.g.: https://github.com/inveniosoftware/invenio-files-rest/actions/runs/19275974647/job/55115752090 is all good

Coming back from break, I can see that those example CI jobs are now broken: https://github.com/inveniosoftware/invenio-files-rest/actions/runs/20754389283/job/59592233141?pr=340
with

SQLAlchemy==2.0.45

being the change. And locally I've unearthed the reason being (after fixing the sphinx stuff):

ERROR tests/test_views_objectversion.py::test_put_header_invalid_tags - sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "localhost" (127 .0.0.1), port 5432 failed: FATAL: sorry, too many clients already

which would be solved by the PR you've made in so far as disposing the engine. I have not encountered the data leakage issue (yet).

For a patch level change to cause so much problem is very strange. Nothing I've read today really explained this.

If you can provide more motivation and root cause explanation that would be helpful in determining the best course of action. Thanks!