Deadlock in bevy_rapier2d 0.30 #670
Description
tldr;> I'm building a game that works fine in bevy 0.15 with bevy_rapier2d 0.29, but when upgrading to bevy 0.16 with bevy_rapier2d 0.30, it deadlocks and stops running any systems. Probably one of the systems in bevy_rapier2d isn't completing and that stops bevy from running further frames. I've kept it running for a few minutes, it remains stuck. CPU utilization is high, so it seems to be doing something.
I have then disabled most of the bevy plugins and systems of the game, and the minimal set of systems reproducing the issue is
- A module generating a large number of entities in a
Updateschedule, each entity has abevy_rapier2dCollidercomponent - The
RapierPhysicsPlugin::<NoUserData>::pixels_per_meter(1.0)plugin
In this configuration, i.e. not running any manual intersection tests and the only code using rapier2d being those Collider components being added and the RapierPhysicsPlugin being enabled, the deadlock still reproduces. And this subset of the running plugins/systems did not require any code changes with the bevy 0.15 to 0.16 upgrade, so it's the very same code running fine in bevy 0.15 with bevy_rapier2d 0.29 and deadlocking in bevy 0.16 with bevy_rapier 0.30.
In rust-lldb, when the game is deadlocked, I see that there is one thread mentioning rapier. This is the backtrace. Interestingly, continuing to run the process for a while and then interrupting it again yields the same backtrace at the same line. So it doesn't seem to have made any progress. Unfortunately, the backtrace doesn't tell me much.
* frame #0: 0x0000589dc181bfef my-game`rapier2d::data::graph::Graph$LT$N$C$E$GT$::find_edge_undirected::h034ff808fa362e8c [inlined] rapier2d::data::graph::Graph$LT$N$C$E$GT$::find_edge_undirected_from_node::hb61eb1d38e55ad28(self=<unavailable>, node=0x0000789738043170, b=NodeIndex @ scalar) at graph.rs:473:24
frame #1: 0x0000589dc181bfb5 my-game`rapier2d::data::graph::Graph$LT$N$C$E$GT$::find_edge_undirected::h034ff808fa362e8c(self=<unavailable>, a=<unavailable>, b=NodeIndex @ scalar) at graph.rs:457:32
frame #2: 0x0000589dc184f67a my-game`rapier2d::geometry::narrow_phase::NarrowPhase::add_pair::hf8493219871d3172 [inlined] rapier2d::data::graph::Graph$LT$N$C$E$GT$::find_edge::h9472c6623818ea8e(self=0x0000589dcf984650, a=NodeIndex @ scalar, b=NodeIndex @ scalar) at graph.rs:440:14
frame #3: 0x0000589dc184f66e my-game`rapier2d::geometry::narrow_phase::NarrowPhase::add_pair::hf8493219871d3172(self=0x0000589dcf984650, colliders=<unavailable>, pair=<unavailable>) at narrow_phase.rs:654:22
frame #4: 0x0000589dc18749ee my-game`rapier2d::pipeline::physics_pipeline::PhysicsPipeline::detect_collisions::h8c2defee22474b14 [inlined] rapier2d::geometry::narrow_phase::NarrowPhase::register_pairs::h6050dd8eb2a23b24(self=0x0000589dcf984650, islands=Option<&mut rapier2d::dynamics::island_manager::IslandManager> @ scalar, colliders=0x0000589dcf971a00, bodies=<unavailable>, broad_phase_events=<unavailable>, events=<unavailable>) at narrow_phase.rs:679:26
frame #5: 0x0000589dc18749bb my-game`rapier2d::pipeline::physics_pipeline::PhysicsPipeline::detect_collisions::h8c2defee22474b14(self=<unavailable>, integration_parameters=<unavailable>, islands=0x0000589dcf984510, broad_phase=<unavailable>, narrow_phase=0x0000589dcf984650, bodies=<unavailable>, colliders=<unavailable>, impulse_joints=<unavailable>, multibody_joints=<unavailable>, modified_colliders=<unavailable>, removed_colliders=<unavailable>, hooks=<unavailable>, events=<unavailable>, handle_user_changes=<unavailable>) at physics_pipeline.rs:143:22
frame #6: 0x0000589dc1875514 my-game`rapier2d::pipeline::physics_pipeline::PhysicsPipeline::step::hcdd2bee2124c43c6(self=<unavailable>, gravity=<unavailable>, integration_parameters=<unavailable>, islands=<unavailable>, broad_phase=<unavailable>, narrow_phase=<unavailable>, bodies=<unavailable>, colliders=<unavailable>, impulse_joints=<unavailable>, multibody_joints=<unavailable>, ccd_solver=<unavailable>, query_pipeline=<unavailable>, hooks=<unavailable>, events=<unavailable>) at physics_pipeline.rs:489:14
frame #7: 0x0000589dc1796043 my-game`bevy_rapier2d::plugin::context::RapierContextSimulation::step_simulation::h19e7628718f94f57(self=<unavailable>, colliders=<unavailable>, joints=<unavailable>, rigidbody_set=<unavailable>, gravity=<unavailable>, timestep_mode=<unavailable>, events=<unavailable>, hooks=<unavailable>, time=<unavailable>, sim_to_render_time=<unavailable>, interpolation_query=<unavailable>) at mod.rs:897:35
frame #8: 0x0000589dc1674c90 my-game`core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnMut$LT$A$GT$$u20$for$u20$$RF$mut$u20$F$GT$::call_mut::hc463b2ca8f3ce5d6 [inlined] bevy_rapier2d::plugin::systems::step_simulation::h757ef81606f85fd4(context=Query<(&mut bevy_rapier2d::plugin::context::RapierContextSimulation, &mut bevy_rapier2d::plugin::context::RapierContextColliders, &mut bevy_rapier2d::plugin::context::RapierQueryPipeline, &mut bevy_rapier2d::plugin::context::RapierContextJoints, &mut bevy_rapier2d::plugin::context::RapierRigidBodySet, &bevy_rapier2d::plugin::configuration::RapierConfiguration, &mut bevy_rapier2d::plugin::context::SimulationToRenderTime), ()> @ 0x00007896ed7e7020, timestep_mode=Res<bevy_rapier2d::plugin::configuration::TimestepMode> @ 0x00007896ed7e7040, hooks=<unavailable>, time=Res<bevy_time::time::Time<()>> @ 0x00007896ed7e7060, collision_events=EventWriter<bevy_rapier2d::pipeline::events::CollisionEvent> @ 0x00007896ed7e7080, contact_force_events=EventWriter<bevy_rapier2d::pipeline::events::ContactForceEvent> @ 0x00007896ed7e70a0, interpolation_query=Query<(&bevy_rapier2d::dynamics::rigid_body::RapierRigidBodyHandle, &mut bevy_rapier2d::dynamics::rigid_body::TransformInterpolation), ()> @ 0x00007896ed7e70c0) at mod.rs:70:21
frame #9: 0x0000589dc16740f1 my-game`core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnMut$LT$A$GT$$u20$for$u20$$RF$mut$u20$F$GT$::call_mut::hc463b2ca8f3ce5d6 [inlined] core::ops::function::FnMut::call_mut::hdc6352e3f7e9f919((null)=<unavailable>, (null)=(bevy_ecs::system::query::Query<(&mut bevy_rapier2d::plugin::context::RapierContextSimulation, &mut bevy_rapier2d::plugin::context::RapierContextColliders, &mut bevy_rapier2d::plugin::context::RapierQueryPipeline, &mut bevy_rapier2d::plugin::context::RapierContextJoints, &mut bevy_rapier2d::plugin::context::RapierRigidBodySet, &bevy_rapier2d::plugin::configuration::RapierConfiguration, &mut bevy_rapier2d::plugin::context::SimulationToRenderTime), ()>, bevy_ecs::change_detection::Res<bevy_rapier2d::plugin::configuration::TimestepMode>, bevy_ecs::system::system_param::StaticSystemParam<()>, bevy_ecs::change_detection::Res<bevy_time::time::Time<()>>, bevy_ecs::event::writer::EventWriter<bevy_rapier2d::pipeline::events::CollisionEvent>, bevy_ecs::event::writer::EventWriter<bevy_rapier2d::pipeline::events::ContactForceEvent>, bevy_ecs::system::query::Query<(&bevy_rapier2d::dynamics::rigid_body::RapierRigidBodyHandle, &mut bevy_rapier2d::dynamics::rigid_body::TransformInterpolation), ()>) @ 0x00007896ed7e7020) at function.rs:168:5
frame #10: 0x0000589dc16740f1 my-game`core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnMut$LT$A$GT$$u20$for$u20$$RF$mut$u20$F$GT$::call_mut::hc463b2ca8f3ce5d6(self=<unavailable>, args=<unavailable>) at function.rs:301:21
frame #11: 0x0000589dc16737bd my-game`_$LT$bevy_ecs..system..schedule_system..InfallibleSystemWrapper$LT$S$GT$$u20$as$u20$bevy_ecs..system..system..System$GT$::run_unsafe::hfb26a9f300a3f958 [inlined] _$LT$Func$u20$as$u20$bevy_ecs..system..function_system..SystemParamFunction$LT$fn$LP$F0$C$F1$C$F2$C$F3$C$F4$C$F5$C$F6$RP$$u20$.$GT$$u20$Out$GT$$GT$::run::call_inner::h0c9a0b1b9482490e(f=0x0000589dcf30b9ac, F0=Query<(&mut bevy_rapier2d::plugin::context::RapierContextSimulation, &mut bevy_rapier2d::plugin::context::RapierContextColliders, &mut bevy_rapier2d::plugin::context::RapierQueryPipeline, &mut bevy_rapier2d::plugin::context::RapierContextJoints, &mut bevy_rapier2d::plugin::context::RapierRigidBodySet, &bevy_rapier2d::plugin::configuration::RapierConfiguration, &mut bevy_rapier2d::plugin::context::SimulationToRenderTime), ()> @ 0x00005ef04a6b8e00, F1=Res<bevy_rapier2d::plugin::configuration::TimestepMode> @ 0x00005ef04b02c1c0, F2=<unavailable>, F3=Res<bevy_time::time::Time<()>> @ 0x00005ef04b02f910, F4=EventWriter<bevy_rapier2d::pipeline::events::CollisionEvent> @ 0x00005ef04b032f10, F5=EventWriter<bevy_rapier2d::pipeline::events::ContactForceEvent> @ 0x00005ef04b02f8b0, F6=Query<(&bevy_rapier2d::dynamics::rigid_body::RapierRigidBodyHandle, &mut bevy_rapier2d::dynamics::rigid_body::TransformInterpolation), ()> @ 0x00005ef04b0327f0) at function_system.rs:945:21
frame #12: 0x0000589dc1673747 my-game`_$LT$bevy_ecs..system..schedule_system..InfallibleSystemWrapper$LT$S$GT$$u20$as$u20$bevy_ecs..system..system..System$GT$::run_unsafe::hfb26a9f300a3f958 [inlined] _$LT$Func$u20$as$u20$bevy_ecs..system..function_system..SystemParamFunction$LT$fn$LP$F0$C$F1$C$F2$C$F3$C$F4$C$F5$C$F6$RP$$u20$.$GT$$u20$Out$GT$$GT$::run::h5aeb12dbaa4021e3(self=0x0000589dcf30b9ac, (null)=<unavailable>, param_value=(bevy_ecs::system::query::Query<(&mut bevy_rapier2d::plugin::context::RapierContextSimulation, &mut bevy_rapier2d::plugin::context::RapierContextColliders, &mut bevy_rapier2d::plugin::context::RapierQueryPipeline, &mut bevy_rapier2d::plugin::context::RapierContextJoints, &mut bevy_rapier2d::plugin::context::RapierRigidBodySet, &bevy_rapier2d::plugin::configuration::RapierConfiguration, &mut bevy_rapier2d::plugin::context::SimulationToRenderTime), ()>, bevy_ecs::change_detection::Res<bevy_rapier2d::plugin::configuration::TimestepMode>, bevy_ecs::system::system_param::StaticSystemParam<()>, bevy_ecs::change_detection::Res<bevy_time::time::Time<()>>, bevy_ecs::event::writer::EventWriter<bevy_rapier2d::pipeline::events::CollisionEvent>, bevy_ecs::event::writer::EventWriter<bevy_rapier2d::pipeline::events::ContactForceEvent>, bevy_ecs::system::query::Query<(&bevy_rapier2d::dynamics::rigid_body::RapierRigidBodyHandle, &mut bevy_rapier2d::dynamics::rigid_body::TransformInterpolation), ()>) @ 0x00005ef04a94ca40) at function_system.rs:948:17
frame #13: 0x0000589dc1673747 my-game`_$LT$bevy_ecs..system..schedule_system..InfallibleSystemWrapper$LT$S$GT$$u20$as$u20$bevy_ecs..system..system..System$GT$::run_unsafe::hfb26a9f300a3f958 [inlined] _$LT$bevy_ecs..system..function_system..FunctionSystem$LT$Marker$C$F$GT$$u20$as$u20$bevy_ecs..system..system..System$GT$::run_unsafe::hbcf6ba650f17d537(self=<unavailable>, input=<unavailable>, world=UnsafeWorldCell @ 0x00005ef04b03bd10) at function_system.rs:735:29
frame #14: 0x0000589dc16733c4 my-game`_$LT$bevy_ecs..system..schedule_system..InfallibleSystemWrapper$LT$S$GT$$u20$as$u20$bevy_ecs..system..system..System$GT$::run_unsafe::hfb26a9f300a3f958(self=0x0000589dcf30b5e0, (null)=<unavailable>, world=UnsafeWorldCell @ 0x00005ef04b03b260) at schedule_system.rs:68:16
frame #15: 0x0000589dc423c1ae my-game`bevy_ecs::schedule::executor::__rust_begin_short_backtrace::run_unsafe::h617813333f0a9285(system=<unavailable>, world=<unavailable>) at mod.rs:280:29
frame #16: 0x0000589dc41a5354 my-game`std::panic::catch_unwind::h77a5f10e33e04484 [inlined] bevy_ecs::schedule::executor::multi_threaded::ExecutorState::spawn_system_task::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h99c99494c87f4d87 at multi_threaded.rs:634:39
frame #17: 0x0000589dc41a5344 my-game`std::panic::catch_unwind::h77a5f10e33e04484 [inlined] core::ops::function::FnOnce::call_once::h82ce4a6c888a5c53((null)=<unavailable>, (null)=<unavailable>) at function.rs:253:5
frame #18: 0x0000589dc41a5344 my-game`std::panic::catch_unwind::h77a5f10e33e04484 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hb68a30ab24f3835a(self=<unavailable>, (null)=<unavailable>) at unwind_safe.rs:272:9
frame #19: 0x0000589dc41a5344 my-game`std::panic::catch_unwind::h77a5f10e33e04484 [inlined] std::panicking::catch_unwind::do_call::h0af680538c461f34(data=<unavailable>) at panicking.rs:589:40
frame #20: 0x0000589dc41a5344 my-game`std::panic::catch_unwind::h77a5f10e33e04484 [inlined] std::panicking::catch_unwind::h2cf24675cc9f1706(f=<unavailable>) at panicking.rs:552:19
frame #21: 0x0000589dc41a5330 my-game`std::panic::catch_unwind::h77a5f10e33e04484(f=<unavailable>) at panic.rs:359:14
frame #22: 0x0000589dc417c35e my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424 [inlined] bevy_ecs::schedule::executor::multi_threaded::ExecutorState::spawn_system_task::_$u7b$$u7b$closure$u7d$$u7d$::h62e81a979454a918 at multi_threaded.rs:627:23
frame #23: 0x0000589dc417c33a my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::ha768e78a8952d322(self=<unavailable>, cx=<unavailable>) at unwind_safe.rs:297:9
frame #24: 0x0000589dc417c33a my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424 [inlined] _$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h4422985b8c4669c4 at future.rs:656:53
frame #25: 0x0000589dc417c33a my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h2630a04afd739dc5(self=<unavailable>, (null)=<unavailable>) at unwind_safe.rs:272:9
frame #26: 0x0000589dc417c33a my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424 [inlined] std::panicking::catch_unwind::do_call::h3e324f83e80ddbd8(data=<unavailable>) at panicking.rs:589:40
frame #27: 0x0000589dc417c33a my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424 [inlined] std::panicking::catch_unwind::h8ff92d24de3afc6c(f=<unavailable>) at panicking.rs:552:19
frame #28: 0x0000589dc417c33a my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424 [inlined] std::panic::catch_unwind::he8b4146f3d2b562f(f=<unavailable>) at panic.rs:359:14
frame #29: 0x0000589dc417c33a my-game`_$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h22989058d3272424(self=<unavailable>, cx=<unavailable>) at future.rs:656:9
frame #30: 0x0000589dc420d10d my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029 [inlined] _$LT$async_executor..AsyncCallOnDrop$LT$Fut$C$Cleanup$GT$$u20$as$u20$core..future..future..Future$GT$::poll::ha00aeccd836dde03(self=Pin<&mut async_executor::AsyncCallOnDrop<futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>, async_executor::{impl#5}::spawn_inner::{closure_env#0}<core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>>>> @ scalar, cx=0x00007896ed7e7588) at lib.rs:1179:31
frame #31: 0x0000589dc420d104 my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029 [inlined] async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::_$u7b$$u7b$closure$u7d$$u7d$::hd7134d772cc528c3 at raw.rs:550:21
frame #32: 0x0000589dc420d104 my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029 [inlined] core::ops::function::FnOnce::call_once::h316385d1844a6819((null)={closure_env#1}<async_executor::AsyncCallOnDrop<futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>, async_executor::{impl#5}::spawn_inner::{closure_env#0}<core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>>>, core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, async_executor::{impl#5}::schedule::{closure_env#0}, ()> @ 0x00005ef04b0ddc40, (null)=<unavailable>) at function.rs:253:5
frame #33: 0x0000589dc420d104 my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha0ec3db2cc2b4c40(self=AssertUnwindSafe<async_task::raw::{impl#3}::run::{closure_env#1}<async_executor::AsyncCallOnDrop<futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>, async_executor::{impl#5}::spawn_inner::{closure_env#0}<core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>>>, core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, async_executor::{impl#5}::schedule::{closure_env#0}, ()>> @ 0x00005ef04b0dde40, (null)=<unavailable>) at unwind_safe.rs:272:9
frame #34: 0x0000589dc420d104 my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029 [inlined] std::panicking::catch_unwind::do_call::h9a66c8d6d198f9a7(data=<unavailable>) at panicking.rs:589:40
frame #35: 0x0000589dc420d104 my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029 [inlined] std::panicking::catch_unwind::hb35d1273474ff4a9(f=AssertUnwindSafe<async_task::raw::{impl#3}::run::{closure_env#1}<async_executor::AsyncCallOnDrop<futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>, async_executor::{impl#5}::spawn_inner::{closure_env#0}<core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>>>, core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, async_executor::{impl#5}::schedule::{closure_env#0}, ()>> @ 0x00005ef04b0de9b0) at panicking.rs:552:19
frame #36: 0x0000589dc420d104 my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029 [inlined] std::panic::catch_unwind::hee6c73f24715c962(f=AssertUnwindSafe<async_task::raw::{impl#3}::run::{closure_env#1}<async_executor::AsyncCallOnDrop<futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>, async_executor::{impl#5}::spawn_inner::{closure_env#0}<core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<bevy_ecs::schedule::executor::multi_threaded::{impl#5}::spawn_system_task::{async_block_env#0}>>>>, core::result::Result<(), alloc::boxed::Box<(dyn core::any::Any + core::marker::Send), alloc::alloc::Global>>, async_executor::{impl#5}::schedule::{closure_env#0}, ()>> @ 0x00005ef04b082670) at panic.rs:359:14
frame #37: 0x0000589dc420d104 my-game`async_task::raw::RawTask$LT$F$C$T$C$S$C$M$GT$::run::ha0ae0416c5220029(ptr=0x00007896589d0c70) at raw.rs:549:23
frame #38: 0x0000589dc30a794e my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h00e41ff38344574f [inlined] async_executor::State::run::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hb19ec62cc1c7e818((null)=0x00007896ed7e7d30) at lib.rs:745:30
frame #39: 0x0000589dc30a78e9 my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h00e41ff38344574f(self=Pin<&mut futures_lite::future::Or<bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block#0}::{async_block_env#0}<(usize, wgpu::api::command_buffer::CommandBuffer), bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::renderer::{impl#1}::finish::{closure_env#0}, (usize, wgpu::api::command_buffer::CommandBuffer)>>, async_executor::{impl#13}::run::{async_fn#0}::{async_block_env#0}<!, bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block#0}::{async_block_env#0}<(usize, wgpu::api::command_buffer::CommandBuffer), bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::renderer::{impl#1}::finish::{closure_env#0}, (usize, wgpu::api::command_buffer::CommandBuffer)>>>>> @ scalar, cx=0x00007896ed7e7d30) at future.rs:457:46
frame #40: 0x0000589dc308e59e my-game`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h264c287bbb17249b [inlined] async_executor::State::run::_$u7b$$u7b$closure$u7d$$u7d$::hfa699d523528a468((null)=<unavailable>) at lib.rs:752:32
frame #41: 0x0000589dc308e4cb my-game`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h264c287bbb17249b [inlined] async_executor::Executor::run::_$u7b$$u7b$closure$u7d$$u7d$::h2657bf955139af5f((null)=<unavailable>) at lib.rs:343:34
frame #42: 0x0000589dc308e413 my-game`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h264c287bbb17249b(self=Pin<&mut core::panic::unwind_safe::AssertUnwindSafe<async_executor::{impl#5}::run::{async_fn_env#0}<!, bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block#0}::{async_block_env#0}<(usize, wgpu::api::command_buffer::CommandBuffer), bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::renderer::{impl#1}::finish::{closure_env#0}, (usize, wgpu::api::command_buffer::CommandBuffer)>>>>> @ scalar, cx=<unavailable>) at unwind_safe.rs:297:9
frame #43: 0x0000589dc30a917b my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a [inlined] _$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h0f8e925325dfe14f at future.rs:656:53
frame #44: 0x0000589dc30a916f my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h59d9c9da508f5c93(self=AssertUnwindSafe<futures_lite::future::{impl#11}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<async_executor::{impl#5}::run::{async_fn_env#0}<!, bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block#0}::{async_block_env#0}<core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>, bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>>>>>>> @ 0x00005ef04a98c830, (null)=<unavailable>) at unwind_safe.rs:272:9
frame #45: 0x0000589dc30a916f my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a [inlined] std::panicking::catch_unwind::do_call::h90d6d7bd4fe4b831(data=<unavailable>) at panicking.rs:589:40
frame #46: 0x0000589dc30a916f my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a [inlined] std::panicking::catch_unwind::habdc6ffd00d71085(f=AssertUnwindSafe<futures_lite::future::{impl#11}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<async_executor::{impl#5}::run::{async_fn_env#0}<!, bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block#0}::{async_block_env#0}<core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>, bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>>>>>>> @ 0x00005ef04a98dab0) at panicking.rs:552:19
frame #47: 0x0000589dc30a916f my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a [inlined] std::panic::catch_unwind::h4ce9e07b78c1023f(f=AssertUnwindSafe<futures_lite::future::{impl#11}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<async_executor::{impl#5}::run::{async_fn_env#0}<!, bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block#0}::{async_block_env#0}<core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>, bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>>>>>>> @ 0x00005ef04a98da70) at panic.rs:359:14
frame #48: 0x0000589dc30a916f my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a [inlined] _$LT$futures_lite..future..CatchUnwind$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h2b8982c4aeb082b8(self=Pin<&mut futures_lite::future::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<async_executor::{impl#5}::run::{async_fn_env#0}<!, bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block#0}::{async_block_env#0}<core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>, bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>>>>>>> @ scalar, cx=0x00007896ed7e7d30) at future.rs:656:9
frame #49: 0x0000589dc30a916f my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a [inlined] bevy_tasks::task_pool::TaskPool::execute_global_scope::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hf3e33972b078ff5b((null)=0x00007896ed7e7d30) at task_pool.rs:526:22
frame #50: 0x0000589dc30a915b my-game`_$LT$futures_lite..future..Or$LT$F1$C$F2$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hec7a3ea6353f539a(self=Pin<&mut futures_lite::future::Or<bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>>, bevy_tasks::task_pool::{impl#2}::execute_global_scope::{async_fn#0}::{async_block_env#0}<core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>, bevy_tasks::task_pool::{impl#2}::scope_with_executor_inner::{async_block#0}::{async_block_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>>>>> @ scalar, cx=0x00007896ed7e7d30) at future.rs:457:46
frame #51: 0x0000589dc339e280 my-game`std::thread::local::LocalKey$LT$T$GT$::with::hdd9aab69eb9244d1 [inlined] bevy_tasks::task_pool::TaskPool::execute_global_scope::_$u7b$$u7b$closure$u7d$$u7d$::h50ee3175d081d37b((null)=0x00007896ed7e7d30) at task_pool.rs:530:41
frame #52: 0x0000589dc339e1b9 my-game`std::thread::local::LocalKey$LT$T$GT$::with::hdd9aab69eb9244d1 [inlined] bevy_tasks::task_pool::TaskPool::scope_with_executor_inner::_$u7b$$u7b$closure$u7d$$u7d$::h1d0669689cca3d29((null)=0x00007896ed7e7d30) at task_pool.rs:457:89
frame #53: 0x0000589dc339e1a6 my-game`std::thread::local::LocalKey$LT$T$GT$::with::hdd9aab69eb9244d1 [inlined] futures_lite::future::block_on::_$u7b$$u7b$closure$u7d$$u7d$::h181bbf5158fe9670(cache=0x00007896ed7eb5d0) at future.rs:99:35
frame #54: 0x0000589dc339e17e my-game`std::thread::local::LocalKey$LT$T$GT$::with::hdd9aab69eb9244d1 [inlined] std::thread::local::LocalKey$LT$T$GT$::try_with::h1997fdf791042418(self=<unavailable>, f=<unavailable>) at local.rs:315:12
frame #55: 0x0000589dc339e17e my-game`std::thread::local::LocalKey$LT$T$GT$::with::hdd9aab69eb9244d1(self=<unavailable>, f=<unavailable>) at local.rs:279:20
frame #56: 0x0000589dc338ce30 my-game`bevy_tasks::task_pool::TaskPool::scope_with_executor_inner::hd08455c35f4b925b [inlined] futures_lite::future::block_on::ha74ca1c172477345(future={async_block_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>> @ 0x00005ef04b083d00) at future.rs:78:11
frame #57: 0x0000589dc338cdd3 my-game`bevy_tasks::task_pool::TaskPool::scope_with_executor_inner::hd08455c35f4b925b(self=0x0000589dc4867948, tick_task_pool_executor=true, external_executor=0x0000789658000c50, scope_executor=0x0000789658000c50, f=<unavailable>) at task_pool.rs:413:13
frame #58: 0x0000589dc309841f my-game`std::sys::backtrace::__rust_begin_short_backtrace::h0886a27e65f47aa5 [inlined] bevy_tasks::task_pool::TaskPool::scope::_$u7b$$u7b$closure$u7d$$u7d$::hfe9d846c0910cf85(scope_executor=<unavailable>) at task_pool.rs:316:18
frame #59: 0x0000589dc3098402 my-game`std::sys::backtrace::__rust_begin_short_backtrace::h0886a27e65f47aa5 [inlined] std::thread::local::LocalKey$LT$T$GT$::try_with::ha74abbb3fe99c5bf(self=<unavailable>, f={closure_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>> @ 0x00005ef04a9f4130) at local.rs:315:12
frame #60: 0x0000589dc30983f5 my-game`std::sys::backtrace::__rust_begin_short_backtrace::h0886a27e65f47aa5 [inlined] std::thread::local::LocalKey$LT$T$GT$::with::h6651cb7bef3ab2f4(self=<unavailable>, f={closure_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure#0}::{closure_env#0}, core::result::Result<bevy_app::sub_app::SubApp, async_channel::RecvError>> @ 0x00005ef04b834ef0) at local.rs:279:20
frame #61: 0x0000589dc30983f5 my-game`std::sys::backtrace::__rust_begin_short_backtrace::h0886a27e65f47aa5 [inlined] bevy_tasks::task_pool::TaskPool::scope::h66b0800234eda04c(self=0x0000589dc4867948, f={closure_env#0} @ 0x00007896ed7e9188) at task_pool.rs:315:31
frame #62: 0x0000589dc30983f5 my-game`std::sys::backtrace::__rust_begin_short_backtrace::h0886a27e65f47aa5 [inlined] _$LT$bevy_render..pipelined_rendering..PipelinedRenderingPlugin$u20$as$u20$bevy_app..plugin..Plugin$GT$::cleanup::_$u7b$$u7b$closure$u7d$$u7d$::hbd692502122082fa at pipelined_rendering.rs:157:22
frame #63: 0x0000589dc3098393 my-game`std::sys::backtrace::__rust_begin_short_backtrace::h0886a27e65f47aa5(f={closure_env#0} @ 0x00007896ed7eaaa0) at backtrace.rs:158:18
frame #64: 0x0000589dc31d3cc9 my-game`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc0d57da626f909e0 [inlined] std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h928f8eec91562df3 at mod.rs:559:17
frame #65: 0x0000589dc31d3cb0 my-game`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc0d57da626f909e0 [inlined] _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::he493bd6877794ca1(self=AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure_env#0}, ()>> @ 0x00005ef04b845ec0, (null)=<unavailable>) at unwind_safe.rs:272:9
frame #66: 0x0000589dc31d3cb0 my-game`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc0d57da626f909e0 [inlined] std::panicking::catch_unwind::do_call::hf48b6a53c50a3c86(data=<unavailable>) at panicking.rs:589:40
frame #67: 0x0000589dc31d3cb0 my-game`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc0d57da626f909e0 [inlined] std::panicking::catch_unwind::h4afcea42fe65f67f(f=AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure_env#0}, ()>> @ 0x00005ef04b9da430) at panicking.rs:552:19
frame #68: 0x0000589dc31d3cb0 my-game`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc0d57da626f909e0 [inlined] std::panic::catch_unwind::hbdb42c21bb5f2c82(f=AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<bevy_render::pipelined_rendering::{impl#2}::cleanup::{closure_env#0}, ()>> @ 0x00005ef04b95e590) at panic.rs:359:14
frame #69: 0x0000589dc31d3cb0 my-game`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc0d57da626f909e0 [inlined] std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::h5a0ad65459351eee at mod.rs:557:30
frame #70: 0x0000589dc31d3c14 my-game`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc0d57da626f909e0((null)=0x0000589dcf95b680, (null)=<unavailable>) at function.rs:253:5
frame #71: 0x0000589dc434e137 my-game`std::sys::pal::unix::thread::Thread::new::thread_start::hd5dce28806973ef9 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hcd81d65010c14a3e at boxed.rs:1971:9
frame #72: 0x0000589dc434e12f my-game`std::sys::pal::unix::thread::Thread::new::thread_start::hd5dce28806973ef9 [inlined] _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h96a52a5b098b326a at boxed.rs:1971:9
frame #73: 0x0000589dc434e128 my-game`std::sys::pal::unix::thread::Thread::new::thread_start::hd5dce28806973ef9 at thread.rs:97:17
frame #74: 0x00007897d269caa4 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:447:8
frame #75: 0x00007897d2729c3c libc.so.6`__clone3 at clone3.S:78