Skip to content

Conversation

@cdecker
Copy link
Member

@cdecker cdecker commented Dec 3, 2025

This has been nagging me for a while, even if the impact is likely only minor, and only on a single configuration.

pytest allows fixtures to be reused across tests, by providing them with a scope. The scope=session would spin up the fixture on the first use, and shut it down only once the last test using it has completed. The problem is that with pytest-xdist the main process just enumerates tests and spawns workers, each in their own process. This also means that each fixture with scope session is also multiplied by the number of workers, because the workers cannot share fixtures among each other, living in different processes.

This plugin implements a global fixture, even when spawning and executing tests in workers. It does so by making the fixture negotiation an explicit step, and the coordinator (main process) is in charge of actually starting / managing / stopping fixtures.

We spawn a small RPC in the coordinator, and inherit the details to contact the RPC to the workers. The workers can then talk to the coordinator, ask for a fixture, and they get a configuration back from the shared fixture (usually just a DSN or other connection string to interact with the fixture).

This is used in our case, for postgres which we used to spawn a new instance of for each test, but since it has a way to serve multiple tenants, via separate schemas and databases, the global fixture just adds dbs and returns connection strings, and on free, the coordinator drops the fixture again.

It may only be sporadically used in CLN, but I found this incredibly useful in more advanced setups, where DB servers and Message Queues, etc must be added to the tests.

@cdecker cdecker force-pushed the 202511-global-fixtures branch from dc6c540 to 042cbf8 Compare December 3, 2025 12:01
cdecker and others added 5 commits December 4, 2025 13:17
Documents the shared infrastructure pattern for pytest tests, explaining
how the plugin enables globally-shared resources (like Docker containers)
across pytest-xdist workers while maintaining per-test isolation through
tenant namespaces.

The README includes:
- Problem statement and solution overview
- Architecture details (InfrastructureService, ServiceManager, Plugin)
- Complete implementation guide with code examples
- Performance analysis showing ~35x speedup
- Technical details on XML-RPC, thread safety, and dynamic loading
- Example services for PostgreSQL, Redis, and RabbitMQ
- Guidance on when to use this pattern

Co-Authored-By: Claude <[email protected]>
Changelog-None
Nobody is fixing it, so remove it. And if it should come back, make it
a dedicated workflow running on a nightly timer, not on the critical
path for PRs.
@cdecker cdecker force-pushed the 202511-global-fixtures branch from 459d8ef to 481636f Compare December 4, 2025 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants