[Question] Is there a way to record the actor coordinates at each step and use them in the terrain curriculum?

[H-Hisamichi]

Hello everyone,

In the initial implementation of the terrain curriculum in Isaac Lab, the distance moved by the actor is computed using the norm between the starting point and the final position at the end of the episode.
However, this approach can be inaccurate when the actor moves in an irregular path.
To address this, I would like to record the actor's position at every learning steps and calculate the total traveled distance by summing the norms of the position differences between each step within an episode.
However, it seems that RecorderManager only allows access to position data after training is complete.
Is there a way to record actor positions at each step and access that data at the end of the episode?

Thanks in advance!

[RandomOakForest]

Thank you for starting this discussion. To accurately compute the total traveled distance for an actor in such way, probably the best is to implement a custom manager that caches positions during the episode and calculates the distance at episode end. Here's a step-by-step solution you may try:

1. Create a Custom Manager Class

Implement a manager to record positions at each step and compute distances at episode end:

import torch
from omni.isaac.lab.managers import ManagerTermBase

class TraveledDistanceRecorder(ManagerTermBase):
    def __init__(self, config, env):
        super().__init__(config, env)
        self._env = env
        self._num_envs = env.num_envs
        self.device = env.device
        
        # Buffers to store positions and distances
        self._position_cache = [[] for _ in range(self._num_envs)]
        self._episode_travel_dist = torch.zeros(self._num_envs, device=self.device)

    def reset(self, env_ids):
        # Reset cache and distance for specified environments
        for idx in env_ids:
            self._position_cache[idx] = []
            self._episode_travel_dist[idx] = 0.0

    def pre_physics_step(self):
        # Record positions at the start of each physics step
        current_pos, _ = self._env.actors.get_world_poses()
        for i in range(self._num_envs):
            self._position_cache[i].append(current_pos[i].clone())

    def post_episode(self):
        # Calculate total distance at episode end
        for env_idx in range(self._num_envs):
            if len(self._position_cache[env_idx]) < 2:
                continue
                
            positions = torch.stack(self._position_cache[env_idx])
            deltas = positions[1:] - positions[:-1]
            distances = torch.norm(deltas, dim=1)
            self._episode_travel_dist[env_idx] = torch.sum(distances)

    def get_episode_travel_dist(self, env_ids=None):
        # Retrieve distances for curriculum manager
        if env_ids is None:
            return self._episode_travel_dist.clone()
        return self._episode_travel_dist[env_ids].clone()

2. Integrate with the Environment

Add the manager to your environment setup:

class MyEnvironment(RLTaskEnv):
    def _setup_managers(self):
        super()._setup_managers()
        self.add_manager(
            "traveled_distance_recorder",
            TraveledDistanceRecorder, 
            cfg={}, 
            env=self
        )

3. Connect to Curriculum Manager

Access the distance data from the curriculum manager:

class TerrainCurriculumManager:
    def update_curriculum(self, env_ids):
        dist_recorder = self._env.managers["traveled_distance_recorder"]
        travel_distances = dist_recorder.get_episode_travel_dist(env_ids)
        
        # Use distances for curriculum updates
        self._adjust_difficulty(travel_distances)

Key Implementation Notes:

1. Position Caching:
   • _position_cache stores positions at each physics step.
   • pre_physics_step() captures positions before physics simulation.
2. Distance Calculation:
   • post_episode() computes the sum of Euclidean distances between consecutive positions.
   • Uses vectorized PyTorch operations for efficiency.
3. Data Flow:
   • Curriculum manager accesses data via get_episode_travel_dist().
   • Reset logic ensures clean state per episode.
4. Performance:
   • Minimal overhead: Only stores positions.
   • GPU-accelerated computations using PyTorch.

Usage Workflow:

1. During Episode:
   • Positions recorded at every physics step.
   • Data stored per-environment.
2. At Episode End:
   • Total distance computed for each environment.
   • Results cached in _episode_travel_dist.
3. Curriculum Update:
   • Terrain manager pulls distances.
   • Adjusts terrain difficulty based on traveled distance.

This solution bypasses RecorderManager limitations by:

• Capturing per-step data in real-time
• Computing distances on-device (GPU)
• Providing immediate access to curriculum parameters
• Maintaining compatibility with Isaac Lab's manager architecture

[H-Hisamichi]

Hello @RandomOakForest,

Thank you for your kind advice!
I will try implementing it in my learning environment right away.

[H-Hisamichi]

Hello @RandomOakForest,

Maybe the managers attribute is not implemented in IsaacLab v2.0.2?

[H-Hisamichi]

Hello @RandomOakForest,

I have a question about implementing the code you suggested.
Taking "Isaac-Velocity-Rough-Anymal-C-v0" as an example, should I:

1. Add the code from Step 1 to "managers" as a new addition,
2. Add the code from Step 2 to "AnymalCRoughEnvCfg" in "rough_env_cfg.py", and
3. Add the code from Step 3 to "curriculum_manager.py"?

[toenails6]

I'm also on this, getting close, will update when done. @H-Hisamichi

[toenails6]

@H-Hisamichi @RandomOakForest I cannot find _setup_managers nor add_manager in the source codes, none of the manager environment classes have it, these steps seem wrong.
I do not see any other way that a custom manager can be explicitly implemented, only custom terms for existing managers.
This still does not solve the problem that the curriculum cannot access step information.

[toenails6]

I do not suggest following RandomOakTree's approach, but there is a way to do this:
You can create your own entry point, such as:

class WheelDog_BlindLocomotionEnv(ManagerBasedRLEnv):
    def __init__(self, cfg, **kwargs):
        super().__init__(cfg, **kwargs)
        # Initialize per-environment cumulative error tensor (shape: num_envs)
        # self._cumulative_vel_error = torch.zeros(self.num_envs, device=self.device)
        self.velocity_error_recorder = VelocityErrorRecorder(
            config={"angular_scale": 1.0},
            env=self
        )
        print("[INFO]: Added velocity_error_recorder manager.")

    def step(self, actions: torch.Tensor) -> VecEnvStepReturn:
        obs, rew, terminated, truncated, info = super().step(actions)
        self.velocity_error_recorder.post_physics_step()
        return obs, rew, terminated, truncated, info
    
    def _reset_idx(self, env_ids: Sequence[int]):
        super()._reset_idx(env_ids)
        self.velocity_error_recorder.reset(env_ids)

then put this in the gym registry, such as this:

gym.register(
    id="Wheeldog-Rl-v0",
    entry_point="wheelDog_RL.tasks.manager_based.wheeldog_rl.envEntry:WheelDog_BlindLocomotionEnv",
    disable_env_checker=True,
    kwargs={
        "env_cfg_entry_point": f"{__name__}.robotCfg:WheelDog_BlindLocomotionEnvCfg",
        "skrl_cfg_entry_point": f"{agents.__name__}:skrl_ppo_cfg.yaml",
    },
)

With this setup you can make your own managers that have their manager methods auto called by the runner (remember to put custom manager methods in the necessary wrappers, my example should be good for reference), and the curriculum manager can use functions that use these custom managers just fine, so long as you get the other basic grammar right.

@H-Hisamichi If you need more details for reference feel free to ask away, I won't write too much more here for sake of clarity.

[toenails6]

The key takeaway is to have gym use your own environment as entry point, after which you can use inheritance super call to do whatever the library modules do first, then add your own stuff.

[H-Hisamichi]

Hello @toenails6 ,

Thank you for the advice! I will definitely refer to it.
I actually came up with a simpler method while working on something else.
I originally wrote this code to calculate sparse rewards, but I believe that by applying it to the existing Recorder MDP, I can access the actor's trajectory over a single episode.

I plan to verify this idea once I finish my current work.
I will update this thread when I have the results.