[Proposal] Any plans for integration of VLM/LLM module and affordance leanring

[Guanbin-Huang]

Proposal Issue: Enhancing Robotics Learning Through Grounding and Affordance Learning

Motivation

In recent developments within the field of robotics, there is a noticeable shift towards grounding-based learning approaches rather than relying solely on Reinforcement Learning (RL) models. This transition underscores the necessity for robots to understand and interpret the environment in a way that mimics human-like understanding, enabling more sophisticated interaction with their surroundings. Grounding in robotics facilitates a more intuitive connection between perception and action, thereby improving the robot's ability to perform tasks in dynamic and unstructured environments.

The significance of incorporating a grounding module along with affordance learning in robotics can be further understood by examining the insights provided by the following key papers. These papers collectively emphasize the need for and benefits of adopting grounding and affordance learning strategies in robotic systems:

[Link to papers highlighting the importance of grounding module and affordance learning]

Proposal for Task Planning Using Visual Language Models (VLM)

One promising direction for enhancing task planning in robots is the integration of Visual Language Models (VLM). VLMs, which combine visual perception with natural language processing, offer a robust framework for interpreting and navigating complex environments. By leveraging VLMs, robots can achieve a higher level of understanding and reasoning, which is critical for effective task planning.

For more information on the application of VLMs in task planning, the following resources are recommended:

• Robot-VILA Project
• COPA 2024 Initiative

These resources provide comprehensive insights into how VLMs can be utilized to facilitate better planning and execution of tasks by robotic systems.

Emphasis on Affordance Learning

Affordance learning represents another pivotal aspect of advancing robotics learning. It involves teaching robots to recognize and utilize the possibilities an object or environment offers for action. This capability is crucial for robots to interact effectively with their environment and adapt to new or unforeseen situations.

For a deeper exploration of affordance learning and its applications in robotics, the following resource is highly recommended:

• Robo-Affordances Project

Conclusion

By focusing on grounding and affordance learning, and incorporating advanced technologies such as Visual Language Models, we can significantly enhance the cognitive capabilities of robots. This not only improves their efficiency and adaptability but also paves the way for more natural and intuitive human-robot interactions. The proposed direction not only aligns with the latest trends in robotics research but also offers a comprehensive approach for tackling some of the most challenging aspects of robotics learning and task execution.

[Dhoeller19]

Hi @Guanbin-Huang,
We now have an example for LLM integration here https://github.com/isaac-sim/IsaacLabEureka.
For now we do not have the bandwidth to integrate more advances VLMs.