Flaky Asset Test breaking CI sporadically: no_error_on_unknown_type_deferred_load_of_self_path

[kfc35]

Bevy version and features

Main Branch, in bevy_asset

What went wrong

In bevy_asset, there is another flaky test that is sporadically breaking CI builds, specifically no_error_on_unknown_type_deferred_load_of_self_path. It seems possible there is another deadlock of some kind since it “runs out of loops."

test tests::no_error_on_unknown_type_deferred_load_of_self_path ... FAILED

failures:

---- tests::no_error_on_unknown_type_deferred_load_of_self_path stdout ----

thread 'tests::no_error_on_unknown_type_deferred_load_of_self_path' (24129) panicked at crates/bevy_asset/src/lib.rs:947:9:
Ran out of loops to return `Some` from `predicate`


failures:
    tests::no_error_on_unknown_type_deferred_load_of_self_path

test result: FAILED. 87 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.83s

error: test failed, to rerun pass `-p bevy_asset --lib`

thread 'main' (2799) panicked at tools/ci/src/ci.rs:62:13:
One or more CI commands failed:
test: Please fix failing tests in output above.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Example builds:
https://github.com/bevyengine/bevy/actions/runs/20246287836/job/58127085637
https://github.com/bevyengine/bevy/actions/runs/20223702618/job/58050465291

[andriyDev]

This test was introduced in #21619 yesterday. What's interesting is that it also seems to be inducing other tests to deadlock somehow - it's not a consistent set, an arbitrary test seems to fail and I don't see a pattern amongst those tests. I wonder if there's some sort of race-condition that cases it to recursively create a bunch of tasks that never complete or something, resulting in other tests getting stuck from a lack of threads?

Switching to just before this PR, I haven't gotten a deadlock in a while. Just after this PR though, I get the deadlocks quite quickly! Note, I haven't actually been able to reproduce the "ran out of loops" error, but these must be related. We should consider a revert for the sake of the RC... I'll still investigate to see if we can salvage this PR.

[andriyDev]

Oh dang ok, I left the tests running on a loop and found the deadlock without #21619, but it takes much much longer. I still think my theory is correct - we have tasks that are holding up threads and if we get unlucky, enough of those tasks get scheduled at the same time to lock up all the threads, and then #21619 is just creating several of those kinds of tasks. It is really unfortunate that all our tests share a thread pool...

I'll see if I can run this through a debugger and see what it's doing.

[andriyDev]

Hmmm I can't figure out what's going on with the deadlocks. Here's the stack trace I was able to get from the deadlock:

ntdll.dll!00007ffd836e1da4() (Unknown Source:0)
KernelBase.dll!00007ffd80c01ef9() (Unknown Source:0)
bevy_asset-d10bccbb4c61139f.exe!std::os::windows::io::handle::impl$25::drop(std::os::windows::io::handle::OwnedHandle * self) Line 390 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\std\src\os\windows\io\handle.rs:390)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<std::os::windows::io::handle::OwnedHandle>(std::os::windows::io::handle::OwnedHandle *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<std::sys::pal::windows::handle::Handle>(std::sys::pal::windows::handle::Handle *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<std::sys::fs::windows::File>(std::sys::fs::windows::File *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<std::fs::File>(std::fs::File *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<winapi_util::win::Handle>(winapi_util::win::Handle *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<enum2$<same_file::win::HandleKind>>(enum2$<same_file::win::HandleKind> *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<same_file::win::Handle>(same_file::win::Handle *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<same_file::Handle>(same_file::Handle *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!core::ptr::drop_in_place<walkdir::Ancestor>(walkdir::Ancestor *) Line 805 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\mod.rs:805)
bevy_asset-d10bccbb4c61139f.exe!walkdir::IntoIter::pop() Line 953 (c:\Users\adzik\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\walkdir-2.5.0\src\lib.rs:953)
bevy_asset-d10bccbb4c61139f.exe!walkdir::impl$4::next(walkdir::IntoIter * self) Line 726 (c:\Users\adzik\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\walkdir-2.5.0\src\lib.rs:726)
bevy_asset-d10bccbb4c61139f.exe!core::iter::traits::iterator::Iterator::try_fold<walkdir::IntoIter,tuple$<>,core::iter::traits::iterator::Iterator::find_map::check::closure_env$0<enum2$<core::result::Result<walkdir::dent::DirEntry,walkdir::error::Error>>,tuple$<std::path::PathBuf,enum2$<file_id::FileId>>,ref_mut$<notify_debouncer_full::cache::impl$1::add_path::closure_env$0>>,enum2$<core::ops::control_flow::ControlFlow<tuple$<std::path::PathBuf,enum2$<file_id::FileId>>,tuple$<>>>>(walkdir::IntoIter * self, tuple$<> f, core::iter::traits::iterator::Iterator::find_map::check::closure_env$0<enum2$<core::result::Result<walkdir::dent::DirEntry,walkdir::error::Error>>,tuple$<std::path::PathBuf,enum2$<file_id::FileId>>,ref_mut$<notify_debouncer_full::cache::impl$1::add_path::closure_env$0>>) Line 2426 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\iter\traits\iterator.rs:2426)
bevy_asset-d10bccbb4c61139f.exe!core::iter::traits::iterator::Iterator::find_map<walkdir::IntoIter,tuple$<std::path::PathBuf,enum2$<file_id::FileId>>,ref_mut$<notify_debouncer_full::cache::impl$1::add_path::closure_env$0>>(walkdir::IntoIter * self, notify_debouncer_full::cache::impl$1::add_path::closure_env$0 * f) Line 2921 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\iter\traits\iterator.rs:2921)
bevy_asset-d10bccbb4c61139f.exe!core::iter::adapters::filter_map::impl$2::next<tuple$<std::path::PathBuf,enum2$<file_id::FileId>>,walkdir::IntoIter,notify_debouncer_full::cache::impl$1::add_path::closure_env$0>(core::iter::adapters::filter_map::FilterMap<walkdir::IntoIter,notify_debouncer_full::cache::impl$1::add_path::closure_env$0> * self) Line 64 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\iter\adapters\filter_map.rs:64)
bevy_asset-d10bccbb4c61139f.exe!notify_debouncer_full::cache::impl$1::add_path(notify_debouncer_full::cache::FileIdMap * self, ref$<std::path::Path> recursive_mode, notify::config::RecursiveMode) Line 74 (c:\Users\adzik\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\notify-debouncer-full-0.6.0\src\cache.rs:74)
bevy_asset-d10bccbb4c61139f.exe!notify_debouncer_full::Debouncer<notify::windows::ReadDirectoryChangesWatcher,notify_debouncer_full::cache::FileIdMap>::add_root<notify::windows::ReadDirectoryChangesWatcher,notify_debouncer_full::cache::FileIdMap,ref$<std::path::Path>>(ref$<std::path::Path> self, notify::config::RecursiveMode recursive_mode) Line 583 (c:\Users\adzik\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\notify-debouncer-full-0.6.0\src\lib.rs:583)
bevy_asset-d10bccbb4c61139f.exe!notify_debouncer_full::Debouncer<notify::windows::ReadDirectoryChangesWatcher,notify_debouncer_full::cache::FileIdMap>::watch<notify::windows::ReadDirectoryChangesWatcher,notify_debouncer_full::cache::FileIdMap,ref$<std::path::PathBuf>>(std::path::PathBuf * self, notify::config::RecursiveMode path) Line 600 (c:\Users\adzik\.cargo\registry\src\index.crates.io-1949cf8c6b5b557f\notify-debouncer-full-0.6.0\src\lib.rs:600)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::io::file::file_watcher::new_asset_event_debouncer<bevy_asset::io::embedded::embedded_watcher::EmbeddedEventHandler>(std::path::PathBuf root, core::time::Duration handler, bevy_asset::io::embedded::embedded_watcher::EmbeddedEventHandler) Line 248 (c:\Content\bevy\crates\bevy_asset\src\io\file\file_watcher.rs:248)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::io::embedded::embedded_watcher::EmbeddedWatcher::new(bevy_asset::io::memory::Dir dir, alloc::sync::Arc<std::sync::poison::rwlock::RwLock<bevy_platform::collections::hash_map::HashMap<alloc::boxed::Box<std::path::Path,alloc::alloc::Global>,std::path::PathBuf,bevy_platform::hash::FixedHasher>>,alloc::alloc::Global> root_paths, async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>> sender, core::time::Duration) Line 42 (c:\Content\bevy\crates\bevy_asset\src\io\embedded\embedded_watcher.rs:42)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::io::embedded::impl$0::register_source::closure$2(bevy_asset::io::embedded::impl$0::register_source::closure_env$2 * sender, async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>) Line 119 (c:\Content\bevy\crates\bevy_asset\src\io\embedded\mod.rs:119)
bevy_asset-d10bccbb4c61139f.exe!alloc::boxed::impl$30::call_mut<tuple$<async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>>,dyn$<core::ops::function::FnMut<tuple$<async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>>,assoc$<Output,enum2$<core::option::Option<alloc::boxed::Box<dyn$<bevy_asset::io::AssetWatcher>,alloc::alloc::Global>>>>>,core::marker::Send,core::marker::Sync>,alloc::alloc::Global>(alloc::boxed::Box<dyn$<core::ops::function::FnMut<tuple$<async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>>,assoc$<Output,enum2$<core::option::Option<alloc::boxed::Box<dyn$<bevy_asset::io::AssetWatcher>,alloc::alloc::Global>>>>>,core::marker::Send,core::marker::Sync>,alloc::alloc::Global> * self, async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>> args) Line 2013 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\boxed.rs:2013)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::io::source::impl$8::build::closure$3(bevy_asset::io::source::impl$8::build::closure_env$3 w, alloc::boxed::Box<dyn$<core::ops::function::FnMut<tuple$<async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>>,assoc$<Output,enum2$<core::option::Option<alloc::boxed::Box<dyn$<bevy_asset::io::AssetWatcher>,alloc::alloc::Global>>>>>,core::marker::Send,core::marker::Sync>,alloc::alloc::Global> *) Line 198 (c:\Content\bevy\crates\bevy_asset\src\io\source.rs:198)
bevy_asset-d10bccbb4c61139f.exe!enum2$<core::option::Option<ref_mut$<alloc::boxed::Box<dyn$<core::ops::function::FnMut<tuple$<async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>>,assoc$<Output,enum2$<core::option::Option<alloc::boxed::Box<dyn$<bevy_asset::io::AssetWatcher>,alloc::alloc::Global>>>>>,core::marker::Send,core::marker::Sync>,alloc::alloc::Global>>>>::and_then<ref_mut$<alloc::boxed::Box<dyn$<core::ops::function::FnMut<tuple$<async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>>,assoc$<Output,enum2$<core::option::Option<alloc::boxed::Box<dyn$<bevy_asset::io::AssetWatcher>,alloc::alloc::Global>>>>>,core::marker::Send,core::marker::Sync>,alloc::alloc::Global>>,alloc::boxed::Box<dyn$<bevy_asset::io::AssetWatcher>,alloc::alloc::Global>,bevy_asset::io::source::impl$8::build::closure_env$3>(enum2$<core::option::Option<ref_mut$<alloc::boxed::Box<dyn$<core::ops::function::FnMut<tuple$<async_channel::Sender<enum2$<bevy_asset::io::AssetSourceEvent>>>,assoc$<Output,enum2$<core::option::Option<alloc::boxed::Box<dyn$<bevy_asset::io::AssetWatcher>,alloc::alloc::Global>>>>>,core::marker::Send,core::marker::Sync>,alloc::alloc::Global>>>> self, bevy_asset::io::source::impl$8::build::closure_env$3 f) Line 1545 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\option.rs:1545)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::io::source::AssetSourceBuilder::build(enum2$<bevy_asset::io::source::AssetSourceId> self, bool id, bool watch) Line 198 (c:\Content\bevy\crates\bevy_asset\src\io\source.rs:198)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::io::source::AssetSourceBuilders::build_sources(bool self, bool watch) Line 363 (c:\Content\bevy\crates\bevy_asset\src\io\source.rs:363)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::impl$2::build(bevy_asset::AssetPlugin * self, bevy_app::app::App * app) Line 375 (c:\Content\bevy\crates\bevy_asset\src\lib.rs:375)
bevy_asset-d10bccbb4c61139f.exe!bevy_app::app::impl$2::add_boxed_plugin::closure$0(bevy_app::app::impl$2::add_boxed_plugin::closure_env$0 *) Line 531 (c:\Content\bevy\crates\bevy_app\src\app.rs:531)
bevy_asset-d10bccbb4c61139f.exe!core::ops::function::FnOnce::call_once<bevy_app::app::impl$2::add_boxed_plugin::closure_env$0,tuple$<>>(bevy_app::app::impl$2::add_boxed_plugin::closure_env$0) Line 250 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:250)
bevy_asset-d10bccbb4c61139f.exe!core::panic::unwind_safe::impl$25::call_once<tuple$<>,bevy_app::app::impl$2::add_boxed_plugin::closure_env$0>(core::panic::unwind_safe::AssertUnwindSafe<bevy_app::app::impl$2::add_boxed_plugin::closure_env$0> self) Line 275 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\panic\unwind_safe.rs:275)
bevy_asset-d10bccbb4c61139f.exe!bevy_app::app::App::add_boxed_plugin(alloc::boxed::Box<dyn$<bevy_app::plugin::Plugin>,alloc::alloc::Global> self) Line 537 (c:\Content\bevy\crates\bevy_app\src\app.rs:537)
bevy_asset-d10bccbb4c61139f.exe!bevy_app::plugin::sealed::impl$0::add_to_app<bevy_asset::AssetPlugin>(bevy_asset::AssetPlugin self, bevy_app::app::App * app, core::panic::location::Location *) Line 148 (c:\Content\bevy\crates\bevy_app\src\plugin.rs:148)
bevy_asset-d10bccbb4c61139f.exe!bevy_app::plugin::sealed::impl$4::add_to_app<bevy_app::plugin::sealed::PluginMarker,bevy_app::task_pool_plugin::TaskPoolPlugin,bevy_app::plugin::sealed::PluginMarker,bevy_asset::AssetPlugin>(tuple$<bevy_app::task_pool_plugin::TaskPoolPlugin,bevy_asset::AssetPlugin> self, bevy_app::app::App * app, core::panic::location::Location *) Line 180 (c:\Content\bevy\crates\bevy_app\src\plugin.rs:180)
bevy_asset-d10bccbb4c61139f.exe!bevy_app::app::App::add_plugins<tuple$<bevy_app::plugin::sealed::PluginsTupleMarker,bevy_app::plugin::sealed::PluginMarker,bevy_app::plugin::sealed::PluginMarker>,tuple$<bevy_app::task_pool_plugin::TaskPoolPlugin,bevy_asset::AssetPlugin>>(tuple$<bevy_app::task_pool_plugin::TaskPoolPlugin,bevy_asset::AssetPlugin> self, core::panic::location::Location * plugins) Line 635 (c:\Content\bevy\crates\bevy_app\src\app.rs:635)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::tests::unapproved_path_setup(bevy_asset::UnapprovedPathMode mode) Line 2440 (c:\Content\bevy\crates\bevy_asset\src\lib.rs:2440)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::tests::unapproved_path_deny_should_panic() Line 2480 (c:\Content\bevy\crates\bevy_asset\src\lib.rs:2480)
bevy_asset-d10bccbb4c61139f.exe!bevy_asset::tests::unapproved_path_deny_should_panic::closure$0(bevy_asset::tests::unapproved_path_deny_should_panic::closure_env$0 *) Line 2479 (c:\Content\bevy\crates\bevy_asset\src\lib.rs:2479)
bevy_asset-d10bccbb4c61139f.exe!core::ops::function::FnOnce::call_once<bevy_asset::tests::unapproved_path_deny_should_panic::closure_env$0,tuple$<>>(bevy_asset::tests::unapproved_path_deny_should_panic::closure_env$0) Line 250 (c:\Users\adzik\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:250)
[Inline Frame] bevy_asset-d10bccbb4c61139f.exe!core::ops::function::FnOnce::call_once() Line 250 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\core\src\ops\function.rs:250)
bevy_asset-d10bccbb4c61139f.exe!test::__rust_begin_short_backtrace<enum2$<core::result::Result<tuple$<>,alloc::string::String>>,enum2$<core::result::Result<tuple$<>,alloc::string::String>> (*)()>() Line 667 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\test\src\lib.rs:667)
[Inline Frame] bevy_asset-d10bccbb4c61139f.exe!test::run_test_in_process() Line 686 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\test\src\lib.rs:686)
bevy_asset-d10bccbb4c61139f.exe!test::run_test::closure$0() Line 607 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\test\src\lib.rs:607)
[Inline Frame] bevy_asset-d10bccbb4c61139f.exe!test::run_test::closure$1() Line 637 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\test\src\lib.rs:637)
bevy_asset-d10bccbb4c61139f.exe!std::sys::backtrace::__rust_begin_short_backtrace<test::run_test::closure_env$1,tuple$<>>() Line 158 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\std\src\sys\backtrace.rs:158)
bevy_asset-d10bccbb4c61139f.exe!core::ops::function::FnOnce::call_once<std::thread::impl$0::spawn_unchecked_::closure_env$1<test::run_test::closure_env$1,tuple$<>>,tuple$<>>() Line 250 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\core\src\ops\function.rs:250)
[Inline Frame] bevy_asset-d10bccbb4c61139f.exe!alloc::boxed::impl$29::call_once() Line 2005 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\alloc\src\boxed.rs:2005)
[Inline Frame] bevy_asset-d10bccbb4c61139f.exe!alloc::boxed::impl$29::call_once() Line 2005 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\alloc\src\boxed.rs:2005)
bevy_asset-d10bccbb4c61139f.exe!std::sys::thread::windows::impl$0::new::thread_start() Line 60 (c:\rustc\ded5c06cf21d2b93bffd5d884aa6e96934ee4234\library\std\src\sys\thread\windows.rs:60)
kernel32.dll!00007ffd8289e8d7() (Unknown Source:0)
ntdll.dll!00007ffd8360c53c() (Unknown Source:0)

First let me note this happens when running cargo t -p bevy_asset --all-features --tests in a loop for long enough. Based on the stack trace, it's clear that the problem relates to the embedded_watcher. Setting up the embedded file watcher means it starts the debouncer which internally iterates through the directories of src. For some reason, it seems to get stuck at dropping the directories as its iterating them. But based on those stack frames, it sure looks like it's blocked on dropping the file handle (note that running for a bit and then pausing again leads to the exact same spot, and I checked which paths the walkdir iterator is on and it's the same - so it really seems blocked).

So the only thing I can conclude is dropping files on Windows is not multi-thread safe??? That seems wild if true.

One fix for the deadlocks is to disabled the embedded_watcher - in fact our CI tests won't even hit this, it's just a problem for all-features. We could maybe fix this by adding an embedded_watcher_override flag to our AssetPlugin, and then set it to false in all our tests, but that's a little cursed... Note: we can't just piggy-back off of the existing watch_override flag, since we have tests that explicitly set that to true.

[kfc35]

I know you mentioned that isolation may not be the answer (on Discord), but I think the underlying assumption of that statement is that this deadlock is something that can conceivably happen to users during normal execution of the bevy engine. Would you say that the deadlock you have seen so far fits that? The impression I am getting is that this locking is sporadically happening because some tests may not be cleaning up properly enough between test runs, and the test suites run in rapid succession (or in parallel maybe?) which causes this effect. Is this something where adding some type of delay between tests (if they are run sequentially and not in parallel) fixes the deadlock issue because then tests properly clean up after themselves?

[andriyDev]

@kfc35, what I meant by "isolation is not the solution" is that we can't exactly force users to use cargo nextest haha. While in theory it might be possible for a user to hit this, I agree the problem relates to the speed of starting apps. But adding a delay to every test sounds rough...