RAI 2.0

RAI 2.0 is a major milestone. We have rewritten it almost from scratch to provide a simple yet efficient way to build embodied agents.

RAI 2.0 introduces a number of new packages, a cleaner architecture, and a more unified approach to agent development. The framework is now easier to extend, more modular, and better aligned with modern AI workflows. This release brings significant improvements in ROS 2 integration, a redesigned Connector API, new agent types including ReAct and ASR/TTS agents, Open set detection, and a simulation and benchmarking suite to support evaluation.

The updated documentation at robotecai.github.io/rai provides comprehensive guidance on getting started, API usage, tutorials, and demos. It covers topics such as speech-to-speech interactions, simulation and benchmarking tools, and available extensions. The documentation also includes detailed instructions for setting up RAI in various forms, tracing, and vendor configurations as well as answers questions on what is RAI and is it a good for me.

RAI 2.0 supports advanced human-robot interaction through text, speech, and multimodal interfaces. More importantly, it enables autonomous long-term reasoning, allowing robots to operate and make decisions independently over extended periods. Agents built with RAI can maintain context and adapt their behavior based on evolving goals and environmental changes. This is made possible by unified handling of perception, memory, and action streams, supporting both reactive and deliberative behavior.

Below is the full changelog of what has changed since the last release. Thanks to everyone who contributed.

What's Changed

• fix(README.md): remove sourcing poetry before first colcon build by @maciejmajek in #352
• feat: add basic implementation of the Connector interface and a StreamingAudioInputDevice Connector by @rachwalk in #350
• Update poetry dependencies by @github-actions in #340
• chore: update poetry install link with fixed version by @maciejmajek in #358
• feat(RaiBaseNode): add qos policy matching by @maciejmajek in #353
• refactor: internal ros communication by @boczekbartek in #335
• refactor(tools): remove repeated ros2 tools by @boczekbartek in #364
• fix(manipuation_demo): remove manual qos profile setting and spin the node by @boczekbartek in #366
• chore: update security to reflect releases by @rachwalk in #369
• chore: refactor pydantic v1 code by @maciejmajek in #356
• feat: unified connector API by @rachwalk in #368
• feat: high level ros2 generic api by @maciejmajek in #367
• feat: make HRIConnector and ARIConnector classes generic by @rachwalk in #375
• feat: ros2 connector by @maciejmajek in #379
• feat: RAI dockerfiles by @maciejmajek in #377
• feat: move automatic speech recognition agent to rai_core by @rachwalk in #357
• feat: ROS2ARIConnector based tools by @maciejmajek in #383
• chore: remove skipped test by @rachwalk in #392
• feat: migrating ros2 tools by @maciejmajek in #385
• feat: add kwargs to BaseConnector methods by @maciejmajek in #394
• chore: remove langchain _run method wrapper by @maciejmajek in #395
• Update poetry dependencies by @github-actions in #361
• feat: general audio connector by @rachwalk in #393
• refactor: make HRIMessage use PIL.Image and pydub.AudioSegment by @rachwalk in #401
• chore: update default openai model by @maciejmajek in #400
• chore: parametrize import test by @maciejmajek in #402
• feat: add ros2 hri connector by @rachwalk in #410
• refactor: drop ROS 2 dependency for core rai packages by @maciejmajek in #411
• feat: unified ROS2 hri message by @maciejmajek in #416
• Fix to leverage pre-built artifacts from Dockerfile by @vmayoral in #413
• feat: simulation and benchmark suite by @maciejmajek in #435
• refactor: wait for message by @maciejmajek in #447
• ci: cleanup step update by @mateuszskrobek in #454
• feat: add tts to rai core by @rachwalk in #419
• fix: redownload openset weights if corrupted by @maciejmajek in #457
• feat(ros2_tools): add timeout_sec param to get_ros2_image by @boczekbartek in #458
• feat: react agent by @maciejmajek in #450
• refactor: move asr and tts agents by @rachwalk in #469
• feat: add ground truth object positions tool by @knicked in #460
• refactor: remove unused dependencies by @jmatejcz in #463
• refactor: demos rewrite, rai_interfaces import guards by @maciejmajek in #449
• docs: Add instructions on how to setup an O3DE scene to work with rai_sim. by @knicked in #431
• feat: add more scenarios to o3de manipulation benchmark by @jmatejcz in #452
• docs: development update by @maciejmajek in #484
• feat: tool calling benchmark by @MagdalenaKotynia in #455
• fix: fix error with openset launch failed to shutdown error by @MagdalenaKotynia in #467
• feat: ROS2 tools allowlist by @maciejmajek in #486
• chore: update manipulation-demo branch by @maciejmajek in #495
• feat: ROS2 tooling improvements by @maciejmajek in #491
• feat: dynamic services by @rachwalk in #496
• feat: add conversation id to HRI message by @rachwalk in #480
• feat: add service toolkit to demo agent by @rachwalk in #498
• feat: enhance destroy_subscribers behavior by @maciejmajek in #499
• feat: various enhancements by @maciejmajek in #508
• refactor: base api by @maciejmajek in #518
• fix: add base_url parameter to ChatOpenAI initialization by @MxLearner in #519
• refactor: internal api by @maciejmajek in #520
• fix: react agent by @boczekbartek in #523
• refactor: tools by @maciejmajek in #521
• refactor: remove rai.utils module by @maciejmajek in #524
• fix: ros2 import guards by @boczekbartek in #527
• refactor: remove ROS 2 imports from non ROS 2 files in rai_core by @maciejmajek in #528
• refactor: connector api by @maciejmajek in #532
• feat: unregister callback by @maciejmajek in #534
• refactor: gdino and gsam as agents by @rachwalk in #501
• feat: BaseROS2Connector by @maciejmajek in #535
• refactor: HRIConnector remove deprecated methods by @rachwalk in #536
• refactor: micro changes by @maciejmajek in #537
• refactor: rai_bench by @jmatejcz in #517
• docs: update installation guide by @rachwalk in #541
• refactor+fix: add LangChainAgent abstraction and fix HRI communication by @boczekbartek in #538
• docs: add docstring to build_message by @rachwalk in #547
• docs: add docstring to sound device connector by @rachwalk in #548
• fix: collecting messages from agent by @jmatejcz in #544
• feat: unify pydantic models for ros2 types by @boczekbartek in #545
• chore: remove deprecated rai.runners module by @boczekbartek in #552
• fix: floats comparison in tests by @boczekbartek in #549
• docs: add shellcheck installation to developer guide by @boczekbartek in #550
• feat: add StateBaseAgent by @boczekbartek in #529
• refactor: relative imports -> absolute imports by @maciejmajek in #553
• feat: rai_whoami by @maciejmajek in #540
• feat: benchmarks - results gathering and visualization by @jmatejcz in #542
• feat: config.toml for ASR and TTS by @rachwalk in #546
• feat: demo download convenience script by @maciejmajek in #560
• chore: remove deprecated scripts by @maciejmajek in #561
• docs: rai 2.0 update by @maciejmajek in #554
• docs: update setup section and fix openset by @boczekbartek in #566
• docs: extend tools tutorial by @maciejmajek in #568
• feat(ci): semantic pull request by @maciejmajek in #572
• chore: rename package rai to rai_core by @maciejmajek in #573
• chore(rai_core): release changes by @maciejmajek in #574
• chore: rai_sim and rai_bench changes by @maciejmajek in #575

New Contributors

• @vmayoral made their first contribution in #413
• @MxLearner made their first contribution in #519

Full Changelog: 1.1.0...2.0.0