Pytest does not show correct tests without `pytest.ini` file

[lexi-k]

Hello, I would like to start using vscode for debuging Python tests. We are using the pytest framework. I read online some guidelines on how to set up the tests UI in VSCode.

I added the following in my global settings.json:

"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"python.testing.autoTestDiscoverOnSaveEnabled": true,

And we are using this configuration in pyproject.toml file:

[tool.pytest.ini_options]
addopts = """
    --verbose
    --ruff
    --mypy
    """
asyncio_mode = "auto"

With this I started to see some tests in the UI:

but that is just incorrect..

Then I tried adding "python.testing.pytestArgs": ["tests"], into the global settings.json and it starts to show the test folder correctly and even picks up all the tests according to the logs, but it only shows this in the menu and no "play arrows" are "inlined" into the code near the test function definitions:

Finally when I add just empty pytest.ini file to the root dir, and I can then remove this "python.testing.pytestArgs": ["tests"], from the global settings.json file and it starts to work properly, or rather it shows all the tests now properly with inlined "play" buttons to run them one by one from the code.
However it still does not pick up the configuration from the pyproject.toml file so to get it working properly I need to add this to the file and now the tests work properly and I am even able to start debugger over them:

[pytest]
asyncio_mode = auto

My question is why does this happen, like why does not the extension pick up the pytest configuration from the pyproject.toml file please? I tried multiple times fiddling with the arguments for the pytest etc. but I was not able to get it working without the pytest.ini file.

I think that my options are either to insert this file to every our repo (we have them around 200) and force this change to everyone else using other IDEs that does not pose this issue or I can add a script that is run every time I create my poetry venv for each project that inserts the pytest.ini file. Either option works but is kind of cumbersome.

So I started looking into the source code of the extension and found this:

vscode-python/src/client/testing/configuration/pytest/testConfigurationManager.ts

Line 53 in 9f6735e

const promises = ['pytest.ini', 'tox.ini', 'setup.cfg'].map(async (cfg) =>

Should not also be there the pyproject.toml? At every other place there is pytest.ini there is the pyproject.toml but not here. I wanted to verify this change but I am failing to run the recompiled extension successfully (I managed to get it installed just now, but it still refuess to start up properly).

I found some similar issues but without any resolution that would apply to my case for example these:
#10722
#22643

My vscode version is:

❯_ code --version
1.93.1
x64

And the extension version is:
v2024.14.1

Appriciate any kind of help with this. Thanks!

[eleanorjboyd]

Hi @lexi-k, thank you for the comprehensive bug report! I think at this point in time we do not use pytest args in the pyproject.toml, @karthiknadig can you confirm this is true and we do not adjust anywhere in the environment as it is being setup? I think the code you found with the testConfigurationManager is old code that is not being used right now which is why it may not be adjusting.

@lexi-k, do you know what version of pytest you are using? What we do from the extension stand-point is just run pytest with the args provided in python.testing.pytestArgs just as you might run it in the terminal. The pytest.ini file being picked up is done by pytest as part of its behavior and not under our control. This being said pytest has begun support for pyproject.toml in version 6.0, you can read about it here: https://docs.pytest.org/en/7.1.x/reference/customize.html#pyproject-toml. I think this is what you are looking for and would handle those settings so try pytest 6 and see if that fixes the problem and let me know. Thanks!

[karthiknadig]

@lexi-k Can you also share the full logs from Output > Python?

@eleanorjboyd I think the test adapter experiment might not be on. So, it might be running the old detection?

[eleanorjboyd]

Good point, @lexi-k can you check in your logs to see by searching for Experiment 'pythonTestAdapter' is active in your python logs. If not can you please turn it on and let me know if the behavior changes? You can do so adding this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. If you are unable to see the log for the experiment chance your log level to info or trace and it should show up (do so by going to "Developer: change log level" in the command palette. Thanks!

[lexi-k]

Hello, thanks for prompt responses!

Let me go through it in order:

• I am not putting anything into the args as I would expect the settings be automatically loaded from the pyproject.toml file as they tend to differ between projects. I tried putting there -c pyproject.toml but it did not help.

• We are using latest pytest version so this should not be the version problem issue:

❯_ pytest --version
pytest 8.3.3

• (I will provide all the debug logs below)

• I just tried with the experiment enabled, but it still does not work :/ (logs and screenshots below)

Firstly we have this in pyproject.toml:

[tool.pytest.ini_options]
addopts = """
    --verbose
    --ruff
    --mypy
    --cov=cto.grafana_templates --cov-report html
    """
asyncio_mode = "auto"

Running just pytest from terminal correctly loads this config and all works as expected.

In the 1.log I am attaching logs from current configuration - pytestArgs empty, Expreiment pytestTestAdapter enabled.
In the 2.log I am attaching logs from the same configuration, but when I create pytest.ini file with asyncio_mode = auto.

When running the first configuration, by clicking on the conftest.py I get (I just realized those tests does not require asyncio_mode to be enabled, but I believe that when I tried it with other tests that require it enabled, it was not actually enabled without the pytest.ini file, but I can double check later and get back to you. Output of running the tests is in 3.log by running the top level tests cto-grafana-templates.

When running the the second configuration by clicking on the cto-grafana-templates run button, everything is "more" fine as can be seen in the 4.log file.

1.log
2.log
3.log
4.log

Is there something more I can provide? Thanks!

[lexi-k]

Sorry, I forget to attach the screenshots, but basically it only shows cto-grafana-templates -> tests -> conftest whereas when adding the pytest.ini file it correctly shows cto-grafana-templates -> tests -> _all the test correctly nested under proper filenames and test names_ (even if I add it empty, but then the asyncio mode is not being used as it is still not being picked up from the pyproject.toml configuration which is being IMO ignored whatsoever which is the problem all along 😅)

[eleanorjboyd]

Hi! Sorry for the delay- is there any way you could send me a minimal repro so I can try it myself? Hard to understand all the files and arrangements without being able to try it myself and I think that would give me a better sense of what's happening

[lexi-k]

Hello, np. Here is a minimal example. It uses poetry as a package manager to create virtual env with poetry shell, then do poetry lock and poetry install. Then you can run pytest from terminal and all the tests will be executed. Then open the "Testing" menu in vscode and click through the tests, all of the two test functions are correctly there and they are runable. Then please remove the pytest.ini file and check how the results in the "Testing" menu in vscode changed - it is now missing the tests and if you press run all, it will run it but without the asyncio_mode = "auto", but running pytest from terminal still works correctly as it picks up the pytest config from the pyproject.toml file. Hope this helps 🙏

minimal-pytest.zip