[Brainstorm] Manager-based Multi-agent environment

[bikcrum]

Manager-based Multi-agent environment

I'm implementing a multi-agent RL environment in IsaacLab using the Manager-Based Approach. Currently, I duplicate assets based on the number of agents and replicate corresponding configurations for observations, actions, rewards, and termination.

Current Implementation

For example:

NUM_AGENTS = 4
AGENT_SPACING = 1.5

@configclass
class MySceneCfg(InteractiveSceneCfg):
    """Configuration for the terrain scene with a legged robot."""

    # ground terrain
    terrain = ...

    # robots
    robot: ArticulationCfg = UNITREE_GO2_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
    
    # # sensors
    contact_forces = ContactSensorCfg(prim_path="{ENV_REGEX_NS}/Robot/.*", history_length=3, track_air_time=True)
    
    # lights
    sky_light = ...

    def __post_init__(self):

        if NUM_AGENTS > 1:
            positions = math_utils.generate_grid_positions(n=NUM_AGENTS, spacing=AGENT_SPACING)
            self.robot.init_state.pos = (positions[0][0].item(), positions[0][1].item(), self.robot.init_state.pos[2])
        for i in range(1, NUM_AGENTS):
            setattr(self, f"robot_{i}", self.robot.replace(prim_path="{ENV_REGEX_NS}" + f"/Robot_{i}"))
            getattr(self, f"robot_{i}").init_state.pos = (positions[i][0].item(), positions[i][1].item(), self.robot.init_state.pos[2])       

            setattr(self, f"contact_forces_{i}", self.contact_forces.replace())
            getattr(self, f"contact_forces_{i}").prim_path = "{ENV_REGEX_NS}" + f"/Robot_{i}/.*"

For observations, I currently duplicate observation groups, modifying them per agent:

def _update_term_params(term: ManagerTermBaseCfg, agent_id: int):
    if 'asset_cfg' in term.params:
        term.params["asset_cfg"].name = f"robot_{agent_id}"
    if 'command_name' in term.params:
        term.params["command_name"] = f"base_velocity_{agent_id}"
    if 'sensor_cfg' in term.params:
        term.params["sensor_cfg"].name = f"contact_forces_{agent_id}"
    if 'action_name' in term.params:
        term.params["action_name"] = f"joint_pos_{agent_id}"

@configclass
class ObservationsCfg:
    """Observation specifications for the MDP."""

    @configclass
    class PolicyCfg(ObsGroup):
        ....
      
        def __post_init__(self):
            self.enable_corruption = True
            self.concatenate_terms = True
            
    @configclass
    class CriticCfg(ObsGroup):
       ....

        def __post_init__(self):
            self.enable_corruption = False
            self.concatenate_terms = True

    policy: PolicyCfg = PolicyCfg()
    critic: CriticCfg = CriticCfg()

    def __post_init__(self):
        """Dynamically create attributes for multiple robots under self.policy."""

        for i in range(1, NUM_AGENTS):

            setattr(self, f"policy_{i}", self.policy.replace())
            setattr(self, f"critic_{i}", self.critic.replace())

            for term in vars(getattr(self, f"policy_{i}")).values():
                if isinstance(term, ObsTerm):
                    _update_term_params(term, i)

            for term in vars(getattr(self, f"critic_{i}")).values():
                if isinstance(term, ObsTerm):
                    _update_term_params(term, i)

---

Current Issue

While this setup allows me to obtain observations in a structured dictionary format, handling actions, rewards, and terminations as single vectors is problematic. Ideally, these should also support grouping mechanisms similar to observation groups. Specifically:

1. Rewards and terminations lack grouping:
   • Currently, rewards and terminations are computed as a single tensor for all agents.
   • There should be reward groups and termination groups, similar to observation groups, to facilitate agent-wise computation.
2. Actions must be passed as tensors and manually sliced per agent:
   • Instead of concatenating actions into a single tensor, the environment should natively accept a dictionary of actions, mapping each key to a corresponding group in the action configuration.
3. Decentralized training support:
   • Observations per agent are already handled via observation groups.
   • However, rewards, terminations, and actions need to be processed separately per agent to maintain consistency.

Proposed Solution

A recent pull request (#729) introduces a similar concept for reward grouping. Extending this to terminations and actions would significantly improve the manager-based multi-agent setup.

I propose:

1. Implementing reward groups and termination groups, so each agent’s step function returns structured outputs.
2. Supporting dictionary-based actions, eliminating the need for manual slicing and indexing.

Would love to hear your thoughts and insights!

[RandomOakForest]

Thanks for posting this discussion. You bring up very good points. The team is reviewing how to follow up in the best way possible. We will reply soon. Thank you for your interest in Isaac Lab.