Parametrization with Variables Unexpectedly Changes Fixture Scope from Class-Level to Function-Level

[yogendrasinghx]

I have a setup fixture that I've parameterized with class-level scope. When I directly specify the argument values as [("chrome", "windows", "latest")], it works fine, maintaining the class-level scope. However, when I try to use variables to set these argument values, I've noticed that the scope unexpectedly shifts to function-level.

tests/test_app.py

import pytest


@pytest.mark.usefixtures("setup")
class TestSmoke:
    def test_1(self):
        assert True

    def test_2(self):
        assert True

Scenario 1 with hardcoded values

tests/conftest.py

import pytest

def pytest_generate_tests(metafunc):
    if 'setup' in metafunc.fixturenames:
        metafunc.parametrize("setup",argvalues=[("chrome", "windows", "latest")], indirect=True, scope='class')


@pytest.fixture
def setup(request):
    print(f"\nSetting up")
    print(f"Param {request.param}")
    print(f"fixture (scope={request.scope})")

    yield
    print(f"\nTearing down")

Output

python -m pytest -s -v -k "TestSmoke" tests
======================================================================================================= test session starts =======================================================================================================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0 -- /Volumes/T7/mywork/pythonProject/dynamic_parameter/venv/bin/python
cachedir: .pytest_cache
rootdir: /Volumes/T7/mywork/pythonProject/dynamic_parameter
collected 2 items                                                                                                                                                                                                                 

tests/test_app.py::TestSmoke::test_1[setup0] 
Setting up
Param ('chrome', 'windows', 'latest')
fixture (scope=class)
PASSED
tests/test_app.py::TestSmoke::test_2[setup0] PASSED
Tearing down

Scenario 2 with variable values

tests/conftest.py

import pytest


def pytest_generate_tests(metafunc):
    browser_name = "chrome"
    os_name = "windows"
    browser_version = "latest"

    if 'setup' in metafunc.fixturenames:
        metafunc.parametrize("setup", argvalues=[(os_name, browser_name, browser_version)], indirect=True, scope='class')


@pytest.fixture
def setup(request):
    print(f"\nSetting up")
    print(f"Param {request.param}")
    yield
    print(f"\nTearing down")

Output

python -m pytest -s -v -k "TestSmoke" tests
======================================================================================================= test session starts =======================================================================================================
platform darwin -- Python 3.12.2, pytest-7.4.4, pluggy-1.4.0 -- /Volumes/T7/mywork/pythonProject/dynamic_parameter/venv/bin/python
cachedir: .pytest_cache
rootdir: /Volumes/T7/mywork/pythonProject/dynamic_parameter
collected 2 items                                                                                                                                                                                                                 

tests/test_app.py::TestSmoke::test_1[setup0] 
Setting up
Param ('windows', 'chrome', 'latest')
PASSED
tests/test_app.py::TestSmoke::test_2[setup0] 
Tearing down

Setting up
Param ('windows', 'chrome', 'latest')
PASSED
Tearing down

(venv) yogen@Mac-mini dynamic_parameter % pip list
Package        Version
-------------- -------
attrs          23.2.0
iniconfig      2.0.0
more-itertools 10.2.0
packaging      24.0
pip            24.0
pluggy         1.4.0
py             1.11.0
pytest         8.1.1
toml           0.10.2
wcwidth        0.2.13

[The-Compiler]

Slightly simpler example:

import pytest


@pytest.mark.usefixtures("fixt")
class TestSmoke:
    def test_1(self):
        assert True

    def test_2(self):
        assert True

import pytest


def pytest_generate_tests(metafunc):
    if "fixt" in metafunc.fixturenames:
        arg = "latest"
        metafunc.parametrize(
            "fixt",
            argvalues=[
                (
                    arg,
                    # "latest",
                )
            ],
            indirect=True,
            scope="class",
        )


@pytest.fixture
def fixt(request):
    yield

with --setup-show:

tests/test_app.py 
        SETUP    F fixt[('latest',)]
        tests/test_app.py::TestSmoke::test_1[fixt0] (fixtures used: fixt, request).
        TEARDOWN F fixt[('latest',)]
        SETUP    F fixt[('latest',)]
        tests/test_app.py::TestSmoke::test_2[fixt0] (fixtures used: fixt, request).
        TEARDOWN F fixt[('latest',)]

but when commenting out arg, and using "latest", instead:

tests/test_app.py 
        SETUP    F fixt[('latest',)]
        tests/test_app.py::TestSmoke::test_1[fixt0] (fixtures used: fixt, request).
        tests/test_app.py::TestSmoke::test_2[fixt0] (fixtures used: fixt, request).
        TEARDOWN F fixt[('latest',)]

I'm... completely flabbergasted. How is this even possible‽

[RonnyPfannschmidt]

@The-Compiler i suspect/fear we somewhere match object identities

when the string is passed as a constant, its likely the python interpreter render it as single tuple value instead of creating it

if that suspicion holds true, then can expect the behaviour as you observe

[RonnyPfannschmidt]

i jsut made a small test in ipython on python 3.12

In [1]: import dis

In [2]: def fun():
   ...:     a = ""
   ...:     return (a,)
   ...: 

In [3]: print(dis.dis(fun))
  1           0 RESUME                   0

  2           2 LOAD_CONST               1 ('')
              4 STORE_FAST               0 (a)

  3           6 LOAD_FAST                0 (a)
              8 BUILD_TUPLE              1
             10 RETURN_VALUE
None

In [4]: def fun2():
   ...:     return ("",)
   ...: 

In [5]: print(dis.dis(fun2))
  1           0 RESUME                   0

  2           2 RETURN_CONST             1 (('',))
None

[RonnyPfannschmidt]

its possible #11257 fixes this

[yogendrasinghx]

Any update on this?

[sadra-barikbin]

When test2 requests the value of setup fixture , setup checks if its cached param key (which is that of test1) is the same as the one coming from test2. It does this check using is operator which tests exact object equality.

In the ("chrome", "windows", "latest") scenario, as this value is a literal or const object (as showed above by @RonnyPfannschmidt ), Python reuses the same object created in pytest_generate_tests(test1) for test2 so is test passes and the cached value is returned.

But in (os_name, browser_name, browser_version) scenario, two separate objects are created for test1 and test2 so the check doesn't pass. As a result, the fixture tears down and re-execute itself.

The check takes place here:

pytest/src/_pytest/fixtures.py

Lines 1056 to 1067 in 400b22d

	my_cache_key = self.cache_key(request)
	if self.cached_result is not None:
	cache_key = self.cached_result[1]
	# note: comparison with `==` can fail (or be expensive) for e.g.
	# numpy arrays (#6497).
	if my_cache_key is cache_key:
	if self.cached_result[2] is not None:
	exc, exc_tb = self.cached_result[2]
	raise exc.with_traceback(exc_tb)
	else:
	result = self.cached_result[0]
	return result