Issue on robosuite tasks

[XizoB]

I've tried using IQ-learn for tasks on robosuite, using their dataset as well as data I've collected myself, and IQ-Learn doesn't perform as well as BC and other offline RL inside robomimic, can you please explain what kind of reasons are there. Thank you very much.

[mw9385]

The most common issue is a biased reward comming from inproper absorbing state treatment. As the task gets complex, the collected transitions from agents have huge difference with expert demos, and in most of the cases, the rewards are not correctly restored. To solve this problem, you could try Q-function clipping to prevent Q-function divergence resulting from incorrect reward function. Take a look at this paper (https://openreview.net/forum?id=o3Q4m8jg4BR).

[XizoB]

Thank you very much for your answer. You are absolutely right. In IRL theory, there is an inherent issue of reward function ambiguity.
[1] Apprenticeship learning via inverse reinforcement learning (P. Abbeel and A. Y. Ng)
[2] Inverse reinforcement learning with multiple ranked experts

I am very excited to hear about your interest in IRL theory. In my findings, transitions that do not correctly restore rewards are often key-points data. In robotics tasks, these key-points data are signals such as grasping and pressing. Although these key-points data account for a very small proportion of the entire demonstration data, they are extremely important.

From another perspective, IRL and adversarial imitation learning focus on learning the data distribution, while BC learns the mapping relationship between states and actions in the data. For the small amount of critical data, algorithms that learn data distribution may not prioritize it as much.

The final conclusion is that IRL is not suitable for robot tasks involving key-points data. If a general-purpose algorithm is desired, IRL must be improved by combining it with BC. This direction has already been extensively researched.
[3] Coherent soft imitation learning
[4] BC-IRL: Learning generalizable reward functions from demonstrations
[5] Watch and match: Supercharging imitation with regularized optimal transport

[mw9385]

Hi, thank you insightful answer.

I have further question regarding key-points data. Does the key-points data are refering transitions that are critical to performance (i.e., the moment of grasping, pressing)? I think I have experienced the similar problems when training my policy.

My previous research was about vision-based navigation (vision-based collision avoidance task). In most of the cases, a drones navigates towards the goal, but when it comes to the critical moment (right before collide with obstacles), the reward value is not correctly reflected. In order to solve this problem, I explicitly define additional reward for that states.

Like you mentioned above, if IRL method is combined with BC, IRL can learn general purpose (as it learns from data distribution) while BC learns much better in key-point transitions (learns mapping relationship). I didn't fully understand the papers listed above, but as far as I understood, updating IRL policy loss along with BC can mitigate problems to some extend. Could you explain more about above methods?

Many thanks.

[XizoB]

Yes, key-point data refers to transitions that are critical to performance. It can be simplified as the process of crossing a narrow bridge.

Regarding the combination of BC and IRL, the following three highlight articles have the same idea, but different specific solutions. They all use the BC loss for reward function recovery in the IRL process.

[XizoB]

I am once again thrilled by your interest in IRL theory. My understanding of the three highlighted articles above is as follows:

ROT: Based on the BC prior policy, the reward is defined as the difference from the demonstration data, followed by RL optimization of the prior policy. (Basic assumption: the expert is the optimal policy)

BC-IRL: Based on the BC loss, the mapping from the expert state to the action of the policy actor is minimized with respect to the expert action MSE, and the actor's parameters are transformed into a function of the critic's parameters. This allows the critic's parameters to be optimized using the BC loss. The actor's update is performed using the SAC method. (Basic assumption: the expert is the optimal policy)

CSIL: Based on the BC prior policy, the policy dynamically adjusts the behavioral tendencies based on the prior policy according to feedback from the actual task environment using the soft update method of SAC. (Basic assumption: the expert is the initialized policy)

[mw9385]

Again, thank you so much! I will carefully read the papers. If there is a chance, we could have further discussion in the near future :)

[XizoB]

I am very much looking forward to your work ！