High entity counts cause decreased rendering performance even when rendering nothing.

[StarArawn]

Bevy version

Main branch commit: d8974e7c3d84189313268f475ceeff9393cedffa

Operating system & version

Windows 10 Vulkan and DX12 API.

What you did

The example code is small and easy to understand:

use bevy::{prelude::*, diagnostic::{FrameTimeDiagnosticsPlugin, LogDiagnosticsPlugin}};

#[derive(Component)]
pub struct Nothing;

#[derive(Bundle)]
pub struct NoBundle {
    nothing: Nothing,
}

fn startup(
    mut commands: Commands,
) {
    let mut entities = Vec::new();
    for _ in 0..5000000 {
        entities.push(NoBundle {
            nothing: Nothing,
        });
    }

    commands.spawn_batch(entities);
}

fn main() {
    App::new()
        .insert_resource(WindowDescriptor {
            width: 1270.0,
            height: 720.0,
            title: String::from("Bug"),
            ..Default::default()
        })
        .insert_resource(ClearColor(Color::rgb(0.211, 0.643, 0.949)))
        .add_plugin(FrameTimeDiagnosticsPlugin::default())
        .add_plugin(LogDiagnosticsPlugin::default())
        .add_plugins(DefaultPlugins)
        .add_startup_system(startup)
        .run();
}

What you expected to happen

Creating large amounts of entities shouldn't decrease rendering performance unless they are actually drawing something to the screen.

What actually happened

Performance decreases rapidly with increased entity counts. On higher end machines this is less noticeable, but still becomes an issue.

Additional information

The decrease in performance can be traced to this:

bevy/crates/bevy_render/src/lib.rs

Lines 193 to 202 in 458cb7a

	let meta_len = app_world.entities().meta.len();
	render_app
	.world
	.entities()
	.reserve_entities(meta_len as u32);
	// flushing as "invalid" ensures that app world entities aren't added as "empty archetype" entities by default
	// these entities cannot be accessed without spawning directly onto them
	// this _only_ works as expected because clear_entities() is called at the end of every frame.
	render_app.world.entities_mut().flush_as_invalid();

Reserving entity ID's for every entity seems to be costly when dealing with a large number of entities.

Possible solutions:

• Don't reserve entities that don't render anything. I'm not sure how performant that will be overall?
• Possibly this could be mitigated by using a sub world where you could store extremely large numbers of entities without the rendering world being aware of said entities.

[alice-i-cecile]

Could we cache the reserved entities in some fashion? It seems strange that we're reserving this list from scratch each time.

[StarArawn]

Could we cache the reserved entities in some fashion? It seems strange that we're reserving this list from scratch each time.

Possibly, I'm not sure. We might have to benchmark a few different approaches to see which one is best. 🤔

[zyllian]

I've implemented a possible solution in my fork which reserves the number of entities used in the render world for the previous frame.

This seems to fix performance with large numbers of non-rendering entities but does mean there's a single-frame slowdown if there are a lot of new rendering entities spawned. I think this is generally solvable by having a (configurable, maybe) minimum number of entities reserved.

I tested this with the many_sprites and many_cubes examples (with an additional 50,000,000 entities each) with no apparent slowdown. Still, I'm not familiar with Bevy's internals and there might be something I've missed here.

[alice-i-cecile]

Excellent! @zyllian, are you comfortable making a PR for this? It will be easier to discuss and verify this further there.

[zyllian]

@alice-i-cecile Sure, here it is.

[alice-i-cecile]

So, thinking about this more: can we use the ExactSizeIterator tech in #4244 to get a quick and exact estimate? If so, we should merge that ASAP and fix it like that.

[sambiak]

Hello, what issues were there with Zylian's fix ? I see it has been closed, but there was no explanation given.

[alice-i-cecile]

From the discussion:

Rob: The point of reserving the entities is so that the same entity from the main world can be used to refer to an entity in the render world. If we don’t reserve the entire active main world entity address space, something could be spawned after what is reserved and then something else is supposed to refer to that. I’m not convinced this change is safe to do. I’ll need to think about it some more. Maybe @cart can beat me to it.

Cart: Yup the approach in this PR breaks the intended behavior. This line isn't about reserving space for rendering entities. Its about ensuring that rendering entities don't get spawned in the "current app id space". We should not merge this PR.