Skip to content

Latest commit

 

History

History
160 lines (97 loc) · 7.64 KB

README.md

File metadata and controls

160 lines (97 loc) · 7.64 KB

BabyBench: A Multimodal Benchmark of Infant Behaviors for Developmental Artificial Intelligence

UPDATE: The submission deadline has been extended until September 8, 2025

BabyBench 2025 Competition Starter Kit

BabyBench is a multimodal benchmark for intrinsic motivations and open-ended learning in developmental artificial ingelligence. The objective is to teach typical behavioral milestones to MIMo, the multimodal infant model. We provide the embodiment, the simulation environments, and the evaluation metrics; all you need is to implement your ideas.

The first BabyBench Competition will take place at the IEEE ICDL 2025 Conference in Prague! The topic of this first edition will be how infants discover their own bodies. Can you help MIMo learn two typical infant behaviors: self touch and hand regard? Make your submission and know more about the competition here.

Important dates

  • Submission deadline: September 1, 2025 September 8, 2025
  • Winner announcement at ICDL: September 19, 2025

Why Participate?

  • Winning team will receive a €150 prize
  • All finalists will receive a certificate at the IEEE ICDL
  • Contribute to the Developmental AI community!

Index

Install the software

We provide two options for installation: a local installation and a Singularity container. Both options are equivalent and will install all the necessary dependencies to participate in the competition.

Option 1: Local installation

The first option is to install BabyBench locally on your machine. This is the recommended option if you are new to Python or if you want to use BabyBench for your own research.

Pre-requisites: Python, Git, and Conda. Tested on Ubuntu 18.04 and 24.04, MS Windows 10 and 11, and MacOS 15.

  1. Create a conda environment:

    conda create --name babybench python=3.12
conda activate babybench

  2. Clone this repository:

    git clone https://github.com/babybench/BabyBench2025_Starter_Kit.git
cd BabyBench2025_Starter_Kit

  3. Install requirements:

    pip install -r requirements.txt

  4. Clone and install MIMo:

    pip install -e MIMo

All done! You are ready to start using BabyBench.

  1. Launch the installation test:

    python test_installation.py

This will run a test to check that the everything is correctly installed. If the installation is successful, you should find a new folder called test_installation in the results directory with a rendered video of the test.

Option 2: Singularity container

You can use a Singularity container to install BabyBench. This allows you to use BabyBench without installing the software on your machine. It is also the recommended option if you are using a HPC cluster.

Pre-requisites: Singularity. Tested on Ubuntu 24.04.

  1. Create the singularity container
singularity build -F babybench.sif babybench.def

This will create a singularity container called babybench.sif in the current directory.

  1. Launch the container
singularity run -c -H /home --bind "$PWD/:/home" babybench.sif

This will run a test to check that the everything is correctly installed. If the installation is successful, you should find a new folder called test_installation in the results directory with a rendered video of the test.

Troubleshooting

If you encounter any issues, visit the troubleshooting page

Getting started

If you are not sure where to begin, we recommend having a look at the examples directory and this wiki page.

The aim for BabyBench is to get MIMo to learn the target behaviors without any external supervision, i.e. without extrinsic rewards. Your goal is to train a policy that matches sensory observation (proprioception, vision, touch) to actions. To do so, we provide an API to initialize and interact with the environments.

Prepare your submission

Submissions must be made through PaperPlaza. The topic of this first BabyBench competition is how infants discover their own bodies. There are two target behaviors: self-touch and hand regard.

Submission should consist of a single zip file containing:

  • A 2-page extended abstract explaining your method,

  • The training log file generated during training,

  • The trajectory log files generated during the evaluation,

  • Optionally, you can also submit links to the following:

    • A repository with your training code,
    • A folder with videos of the learned behaviors rendered during the evaluation.

How to...?

For further information, check our Wiki.
In particular, if you want to know more about:

  • the training environments, see here
  • the target behaviors, see here
  • examples, see here
  • how to generate the submission files, see here
  • the evaluation process, see here
  • resources about intrinsic motivations and open-ended learning, see here

... or see the FAQ for common questions.

Support

Feel free to contact us for any questions about BabyBench. You can post an issue here on Github or contact the organizers via mail.

We highly encourage you to collaborate with other participants! You can submit your problems, questions or ideas in the discussion forum.

Organizing team

Acknowledgements

We thank the IEEE Robotics and Automation Society Technical Committee for Cognitive Robotics for contributing the prize money for the competition. BabyBench is supported by the cluster project ``The Adaptive Mind'' funded by the Excellence Program of the Hessian Ministry of Higher Education, Research, Science and the Arts, Germany, by the Deutsche Forschungsgemeinschaft (DFG project “Abstract REpresentations in Neural Architectures (ARENA)”), by the Czech Science Foundation (GACR), Project no. 25-18113S, and by the Johanna Quandt foundation.

License

This project is licensed under an MIT License - see LICENSE for details