Skip to content

Stop pytest from always stopping at first error. #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Stop pytest from always stopping at first error. #322

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pytest.ini
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests in xcp-ng-tests are integration tests, running on a complex product. Whenever a test fails, we don't know what state this left the host in. Sure, we attempt to add robust teardown, but when you don't know what state the system is in, and teardown itself relies on that system being not more broken that you can handle (Unless teardown would completely reinstall and setup the hosts, or restore some kind of snapshot... whether it's due to a recoverable issue or not.)

So our approach is to play it safe, fail early and fix.

pytest has options to restart from last failure, couldn't we use that (there's -lf and -ff) to avoid re-running everything?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest has options to restart from last failure, couldn't we use that (there's -lf and -ff) to avoid re-running everything?

We can check if that works. But it does not help in the case of "let's launch the tests and use that time for a break", just to discover that the whole thing stopped a few minutes in - basically that's what happens to me all the time, and why I'm pushing for this.

The tests in xcp-ng-tests are integration tests, running on a complex product. Whenever a test fails, we don't know what state this left the host in. Sure, we attempt to add robust teardown, but when you don't know what state the system is in, and teardown itself relies on that system being not more broken that you can handle (Unless teardown would completely reinstall and setup the hosts, or restore some kind of snapshot... whether it's due to a recoverable issue or not.)

I do understand it is the case for some tests. Very notably, tests done on nested hosts protected with snapshots (eg. tests/install/* do not need that extra safety (and I hope we'll add feature this someday to all tests run in nested hosts). But since this is a global flag, even when running safe tests we have to take explicit action to be efficient, and that just adds a mix of mental burden and time loss.

It would be much better if we could flag those tests that make big-enough changes to the host, and based on that (and on other info, like "is there a snapshot to save us if needed") decide to stop or not. That could need to be a pytest plugin.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the behaviour we want at the moment for most test jobs, outside the ones that spin up their own throw-away test pools. Maybe there's a way to enable it per top level directory? Or to have a "stop of first failure" default as we have currently and override it for whole test directories.

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[pytest]
addopts = -ra --maxfail=1
addopts = -ra
markers =
# *** Markers that change test behaviour ***
default_vm: mark a test with a default VM in case no --vm parameter was given.
Expand Down
Loading