How to implement a more complex Command processing scheme

[rohel01]

Hello,

I am using Bevy to experiment with data driven architectures for embedded development.
I am really impressed with Bevy ECS built-in scheduler, how system and their queries are handled in a type-safe manner at compile time.

Embedded systems typically implement increasingly more complex routine, periodic processing, for which Bevy system/query/scheduler design seems... simply ideal.

But such systems are also expected to react to numerous inbound commands, each associated with arbitrarily complex logics. Commands must be processed in the ordered they were received and can arbitrarily mutate the application state.

Bevy's Command concept seems to fit this usecase well. In traditional SW, significant boilerplate code is required to connect an inbound Command type to the desired trigerred behavior. It seems Bevy does not provide mechanisms as powerful as the system/query mechanism there.

If I have understood the documentation correctly, I can register a system with the World and the list of Commands as parameters. I can then implement something similar to the system/query mechanisms by looking at the relevant Bevy source code. I also found this blog post with very helpful explanations.

As a first step, my goal would be to allow registration of command handlers based on the Command type.

Is my approach sane? Since I am new to the ECS approach, I fear I am missing something important.

Kind regards,

[alice-i-cecile]

It sounds like you're likely to want to use Events here. These are lightweight, processed in order, dispatched across systems and strongly typed.

If you instead need something significantly more powerful / flexible, you're likely interested in one-shot systems: see #2192 for discussions and links to draft implementations.

[rohel01]

Thanks for your reply.

I was considering events at first, actually. What is the difference between event processing and event dispatching to systems then?

My initial design consisted in defining one handler systems for each event type. But, it seemed to me the system execution order was first computed from the (optional) dependencies between systems. How is the event reception order taken into account in that case?

I considered the case where two systems (SA and SB) each process one event (EA and EB respectively). Furthermore, let's say I register SB so it is always executed after SA. For a given cycle, I receive EB then EA. In which order my systems are executed?

My key requirement here is to guarantee each event handling system is invoked in the order their associated event is received, which ever this order might be for a given cycle.

Commands seemed the safer option, there.

[alice-i-cecile]

My key requirement here is to guarantee each event handling system is invoked in the order their associated event is received, which ever this order might be for a given cycle.

This is the bit I was missing. In that case yes, Commands are the way to go :)

The fundamental distinction between Commands and Events is that event processing is polling-based (run once each frame in distinct systems in the order received), whereas commands use a first-in-first-out queue to be applied.

If you wanted all EA to be evaluated before all EB, add a system to handle each, and then order those event processing systems. But if you want to interweave EA and EB based on the time they're received, you fundamentally need to tackle them in a sequential loop. Commands work well here, and like you've seen, can be arbitrarily powerful.

[rohel01]

Some of my commands (power off, reset...) will definitely be Commands, then.

For others, I am considering a "mixed" design that should provide more flexibility. Command handlers could be entities with a "command" FIFO component. Marker components could be use to mark handlers with "commands" available for processing. Dedicated systems would then be in charge of emptying the queue when scheduled.

IMO, the main advantage of this "mixed" design are parallelism and persistence. If I can identify command groups which can run in paralel l, then I can leverage the existing scheduler.

What do you think?

[alice-i-cecile]

If I can identify command groups which can run in paralel l, then I can leverage the existing scheduler.

Yep, this is something that I've looked into before. Check out the approach I took in #4090. I'd be curious to see how this works out!

[rohel01]

Thanks for your replies! I mark this question as resolved.
As soon as I can, hopefully in a few months, I hope to share my findings!

[rohel01]

Hello again,

Following up on your really helpful comments and links, I have studied #7999 in more details and come up with a new design.

I want to execute a small (<20) sequence of arbitrary commands each cycle, depending on the network messages received each cycle:

• Each command to be is computed at runtime based on the type of the associated network message
• Commands must be executed in the reception order of the network messages
• A given command should be invoked as many times as the associated message event has been received for the current cycle

Given those requirements, my use case seemed to map naturally to Bevy's commands. But, ergonomics are clearly in favor of systems: extensive user system registration API, type-safe system dependency injection...

Since the release of Bevy ECS schedule v3, one of the key differences commands and exclusive systems (w.r.t. to my use case at least) is now captured in the much improved Schedule abstractions.

Maybe one should bridge the developer experience gap between Commands and Systems by extending the Schedule APIs instead ?

For example, I can implement my command execution scheme using an exclusive system which would dynamically construct a schedule and then run it. A specific resource can provide the mapping between network messages and target system-like commands.

My main concern is I read building schedules dynamically might be expensive. It is true I do not need a full DAG-based parallel execution for this use case. Maybe I can customize a schedule just for this?

What do you think?

[alice-i-cecile]

Use World::resource_scope to split the borrow :) It's a nice abstraction around the remove-operate-reinsert pattern (the hokey-pokey ;) ) that comes in handy for dealing with this sort of problem in general.

The SystemState type can also be used to split the borrow as if you were defining a system, but that's overkill here.

[rohel01]

Thank you, I was able to remove the borrowing issue!

I have set up a resource where my a mapping between a command id and the corresponding system is stored.

// A key to uniquely identify a command to be executed by a specific command system
type CmdSystemKey = (u8, u8);

// Convenient type alias for command systems
pub type BoxedCmdSystem<In = CmdSystemInput, Out = CmdSystemOutput> =
    Box<dyn System<In = In, Out = Out>>;

// The resource to store the key -> system mapping
#[derive(Default, Resource)]
pub struct Cmds(HashMap<CmdSystemKey, BoxedCmdSystem>);

// Trait to be able to register command systems for a specific command key
pub trait CommandSystemRegistry {
    fn register_command<M>(
        &mut self,
        cmd: CmdSystemKey,
        system: impl IntoSystem<CmdSystemInput, CmdSystemOutput, M>,
    );
}

impl CommandSystemRegistry for World {
    fn register_command<M>(
        &mut self,
        cmd_config: CmdSystemKey,
        system: impl IntoSystem<CmdSystemInput, CmdSystemOutput, M>,
    ) {
        let mut cmds = self.resource_mut::<Cmds>();

        cmds.0
            .insert(cmd_config, Box::new(IntoSystem::into_system(system)));
    }
}

// This structure contains the optional payload of the message that triggered the system command
#[derive(Debug, Clone)]
struct CmdSystemInput {
    data: Option<Box<[u8]>>,
}

// This enum is used by command systems to notify the scheduler of execution results. In particular, some commands should stop command execution for the current cycle. 
pub enum CmdSystemOutput {
  Ok,
  Suspend,
  Error(u8)
}

As you might notice, I am using the In/Out generic arguments of the System trait so command system implementers can conveniently access the input message payload and can output an execution result. I have implemented a custom schedule type, which uses a vector instead of a Graph and is able to exploit the command system output for my use case.

I have now an issue with trait resolution very likely because of my In/out system shenanigans. When I attempt to use my command registering functions in a startup system, such as:

// A sample command execution system!!
fn execute_simple_cmd(msg: CmdSystemInput, query: Query<&Active>) -> CmdSystemOutput {
    println!("Yes!!! {:?}", msg);
    CmdSystemOutput::Ok
}

fn startup_cmd_registry(world: &mut World) {
    world.init_resource::<Cmds>();
    world.register_command(
        (1, 45), // Dummy command message 1, 45
        execute_simple_cmd,
    );
}

I get a nasty trait resolution error:

error[E0277]: the trait bound `for<'a, 'b, 'c> fn(fwk::CmdSystemInput, bevy::prelude::Query<'a, 'b, &'c fwk::Active>) -> fwk::CmdSystemOutput {execute_simple_cmd}: IntoSystem<fwk::CmdSystemInput, fwk::CmdSystemOutput, _>` is not satisfied
   --> crates/example/src/fwk/mod.rs:224:13
    |
218 |         world.register_command(
    |               ---------------- required by a bound introduced by this call
...
224 |             execute_simple_cmd,
    |             ^^^^^^^^^^ the trait `IntoSystem<fwk::CmdSystemInput, pus::app::TcResult, _>` is not implemented for fn item `for<'a, 'b, 'c> fn(fwk::CmdSystemInput, bevy::prelude::Query<'a, 'b, &'c fwk::Active>) -> fwk::CmdSystemOutput {execute_simple_cmd}`
    |
note: required by a bound in `fwk::CommandSystemRegistry::register_command`
   --> crates/plsw/src/fwk/mod.rs:30:22
    |
27  |     fn register_command<M>(
    |        ---------------- required by a bound in this associated function
...
30  |         system: impl IntoSystem<CmdSystemInput, CmdSystemOutput, M>,
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `CommandSystemRegistry::register_command`

Can you provide pointers on how how I can investigate that?

[alice-i-cecile]

Hmm. I'm not immediately sure. bevy_system_check can be helpful to validate systems, but I don't think it'll be much good here. Check what I did in my one-shot system PR? It was a very similar flavor, although I struggled to add system chaining.

[rohel01]

I had a look at the system chaining code and found the issue: each of my command system input must be wrapped in a In structure !

// A sample command execution system!!
fn execute_simple_cmd(msg: In<CmdSystemInput>, query: Query<&Active>) -> CmdSystemOutput {
//                         ^ Critical for the system trait implementation to work !!!
    println!("Yes!!! {:?}", msg);
    CmdSystemOutput::Ok
}

I have spent most of the day testing this command scheduler in various configurations. I am now running into B0001 errors, and according to my testing, they are triggered as soon as I initialize a system a second time. I am not sure about the rules I should follow there. In Bevy, I see the schedule graph maintains a list of uninit system nodes and graph initialization occurs when a graph change is detected.

Does it mean I should only initialize each of my system once for the duration of application?

Also, I tried sticking my BoxedCmdSystem into a HashSet to track which systems were initialized, but this type does not implement the necessary traits. I am considering initializing all systems in the registry once and stop worrying about it. What do you think?

[rohel01]

Answering myself here, if it helps someone else.

It seems each system node must be initialized once and only once. You need mutable access to the World to do that, so in my case, the simplest solution was to create a startup system to do just that. Previously, I tried to be smart in my custom schedule run method.

Example code

#[derive(Default, Resource)]
pub struct Cmds(HashMap<CmdSystemKey, BoxedCmdSystem>);

pub fn initialize_cmd_system_register(world: &mut World) {
    world.resource_scope(|world, mut registry: Mut<Cmds>| {
        // First, initialize all systems in the schedule ONCE
        for system in registry.0.values_mut() {
            system.initialize(world);
        }
    });
}