2.5.4

What's Changed

• fix: create missing vision subdirectory for weights download path by @Juliaj in #705
• fix: ROS 2 color space by @maciejmajek in #708

Full Changelog: 2.5.3...2.5.4

2.5.3

What's Changed

• refactor: preprocess_image + tests by @maciejmajek in #702

Full Changelog: 2.5.2...2.5.3

2.5.2

What's Changed

• ci: expect rai-core version bump on changes by @maciejmajek in #692
• feat: dataset creation from traces by @Juliaj in #673
• fix: redownload weights only on PytorchStreamReader error by @pawel-kotowski in #686
• fix: pass copied goal to set_message_fields as it modifies goal contents in ROS 2 Jazzy by @maciejmajek in #699
• fix: megamind steps done by @jmatejcz in #703

Full Changelog: 2.5.0...2.5.2

2.5.0

What's Changed

• chore: sync development -> main by @maciejmajek in #691

  • feat: tool calling benchmark unified across types and prompts variety… (#620)

  • feat: basic tasks extension (#644)

  • feat: tool calling custom interfaces tasks extension (#636)

  • feat: tool calling spatial reasoning tasks extension (#637)

  • refactor: remove navigation tasks (#638)

  • refactor: o3de config (#630)

  • refactor(nav2_toolkit): remove unused action_client (#670)

  • fix: manipulaiton bench fixes (#653)

  • docs: rai simbench docs update (#665)

  • feat: planning task and megamind agent (#679)

  • feat: megamind context providers (#687)

  • feat: tool calling bench - manipulation tasks extenstion (#656)

  • chore: resolving conflicts (#690)

Full Changelog: 2.4.0...2.5.0

2.4.0

What's Changed

• feat: nav2 blocking tool by @maciejmajek in #669

Full Changelog: 2.3.0...2.4.0

2.3.0

What's Changed

• ci: run poetry-test on PR to main by @maciejmajek in #626
• fix: assign level and logger when creating scenarios by @jmatejcz in #625
• fix: path convert tool calling bench by @jmatejcz in #627
• docs: add required poetry version info by @maciejmajek in #631
• chore: bump & publish rai-core by @maciejmajek in #632
• feat: add alias call service by @rachwalk in #635
• feat: kokoro tts support by @MagdalenaKotynia in #643
• fix: configurator bugs fixes by @MagdalenaKotynia in #640
• chore: langchain-ollama update to disable thinking by @jmatejcz in #648
• Update poetry dependencies by @github-actions[bot] in #624
• chore: bump rai-core version to reflect pypi by @maciejmajek in #659
• fix(react-agent-ros2): replace non-blocking run with blocking to avoid severe ROS 2 degradation by @maciejmajek in #661
• fix: add on chat start callback impl by @pawel-kotowski in #662
• fix: Transform obtained from GetROS2TransformTool can be old by @Juliaj in #651
• feat: Add tracing to LLM invoke by @Juliaj in #650
• feat: vlm benchmark with structured output by @MagdalenaKotynia in #666
• chore: deprecate create_conversational_agent by @maciejmajek in #671
• docs: improve rai_bench readme by @pawel-kotowski in #674
• fix: invoke_with_tracing breaks rai used outside of developer space by @Juliaj in #677
• fix: refactor messages to use BaseMessage.text() by @btuan in #683
• feat: service client cache by @maciejmajek in #685

New Contributors

• @pawel-kotowski made their first contribution in #662
• @Juliaj made their first contribution in #651
• @btuan made their first contribution in #683

Full Changelog: 2.0.0...2.4.0

What's Changed

• ci: run poetry-test on PR to main by @maciejmajek in #626
• fix: assign level and logger when creating scenarios by @jmatejcz in #625
• fix: path convert tool calling bench by @jmatejcz in #627
• docs: add required poetry version info by @maciejmajek in #631
• chore: bump & publish rai-core by @maciejmajek in #632
• feat: add alias call service by @rachwalk in #635
• feat: kokoro tts support by @MagdalenaKotynia in #643
• fix: configurator bugs fixes by @MagdalenaKotynia in #640
• chore: langchain-ollama update to disable thinking by @jmatejcz in #648
• Update poetry dependencies by @github-actions[bot] in #624
• chore: bump rai-core version to reflect pypi by @maciejmajek in #659
• fix(react-agent-ros2): replace non-blocking run with blocking to avoid severe ROS 2 degradation by @maciejmajek in #661
• fix: add on chat start callback impl by @pawel-kotowski in #662
• fix: Transform obtained from GetROS2TransformTool can be old by @Juliaj in #651
• feat: Add tracing to LLM invoke by @Juliaj in #650
• feat: vlm benchmark with structured output by @MagdalenaKotynia in #666
• chore: deprecate create_conversational_agent by @maciejmajek in #671
• docs: improve rai_bench readme by @pawel-kotowski in #674
• fix: invoke_with_tracing breaks rai used outside of developer space by @Juliaj in #677
• fix: refactor messages to use BaseMessage.text() by @btuan in #683
• feat: service client cache by @maciejmajek in #685

New Contributors

• @pawel-kotowski made their first contribution in #662
• @Juliaj made their first contribution in #651
• @btuan made their first contribution in #683

Full Changelog: 2.0.0...2.3.0

RAI 2.0

RAI 2.0 is a major milestone. We have rewritten it almost from scratch to provide a simple yet efficient way to build embodied agents.

RAI 2.0 introduces a number of new packages, a cleaner architecture, and a more unified approach to agent development. The framework is now easier to extend, more modular, and better aligned with modern AI workflows. This release brings significant improvements in ROS 2 integration, a redesigned Connector API, new agent types including ReAct and ASR/TTS agents, Open set detection, and a simulation and benchmarking suite to support evaluation.

The updated documentation at robotecai.github.io/rai provides comprehensive guidance on getting started, API usage, tutorials, and demos. It covers topics such as speech-to-speech interactions, simulation and benchmarking tools, and available extensions. The documentation also includes detailed instructions for setting up RAI in various forms, tracing, and vendor configurations as well as answers questions on what is RAI and is it a good for me.

RAI 2.0 supports advanced human-robot interaction through text, speech, and multimodal interfaces. More importantly, it enables autonomous long-term reasoning, allowing robots to operate and make decisions independently over extended periods. Agents built with RAI can maintain context and adapt their behavior based on evolving goals and environmental changes. This is made possible by unified handling of perception, memory, and action streams, supporting both reactive and deliberative behavior.

Below is the full changelog of what has changed since the last release. Thanks to everyone who contributed.

What's Changed

• fix(README.md): remove sourcing poetry before first colcon build by @maciejmajek in #352
• feat: add basic implementation of the Connector interface and a StreamingAudioInputDevice Connector by @rachwalk in #350
• Update poetry dependencies by @github-actions in #340
• chore: update poetry install link with fixed version by @maciejmajek in #358
• feat(RaiBaseNode): add qos policy matching by @maciejmajek in #353
• refactor: internal ros communication by @boczekbartek in #335
• refactor(tools): remove repeated ros2 tools by @boczekbartek in #364
• fix(manipuation_demo): remove manual qos profile setting and spin the node by @boczekbartek in #366
• chore: update security to reflect releases by @rachwalk in #369
• chore: refactor pydantic v1 code by @maciejmajek in #356
• feat: unified connector API by @rachwalk in #368
• feat: high level ros2 generic api by @maciejmajek in #367
• feat: make HRIConnector and ARIConnector classes generic by @rachwalk in #375
• feat: ros2 connector by @maciejmajek in #379
• feat: RAI dockerfiles by @maciejmajek in #377
• feat: move automatic speech recognition agent to rai_core by @rachwalk in #357
• feat: ROS2ARIConnector based tools by @maciejmajek in #383
• chore: remove skipped test by @rachwalk in #392
• feat: migrating ros2 tools by @maciejmajek in #385
• feat: add kwargs to BaseConnector methods by @maciejmajek in #394
• chore: remove langchain _run method wrapper by @maciejmajek in #395
• Update poetry dependencies by @github-actions in #361
• feat: general audio connector by @rachwalk in #393
• refactor: make HRIMessage use PIL.Image and pydub.AudioSegment by @rachwalk in #401
• chore: update default openai model by @maciejmajek in #400
• chore: parametrize import test by @maciejmajek in #402
• feat: add ros2 hri connector by @rachwalk in #410
• refactor: drop ROS 2 dependency for core rai packages by @maciejmajek in #411
• feat: unified ROS2 hri message by @maciejmajek in #416
• Fix to leverage pre-built artifacts from Dockerfile by @vmayoral in #413
• feat: simulation and benchmark suite by @maciejmajek in #435
• refactor: wait for message by @maciejmajek in #447
• ci: cleanup step update by @mateuszskrobek in #454
• feat: add tts to rai core by @rachwalk in #419
• fix: redownload openset weights if corrupted by @maciejmajek in #457
• feat(ros2_tools): add timeout_sec param to get_ros2_image by @boczekbartek in #458
• feat: react agent by @maciejmajek in #450
• refactor: move asr and tts agents by @rachwalk in #469
• feat: add ground truth object positions tool by @knicked in #460
• refactor: remove unused dependencies by @jmatejcz in #463
• refactor: demos rewrite, rai_interfaces import guards by @maciejmajek in #449
• docs: Add instructions on how to setup an O3DE scene to work with rai_sim. by @knicked in #431
• feat: add more scenarios to o3de manipulation benchmark by @jmatejcz in #452
• docs: development update by @maciejmajek in #484
• feat: tool calling benchmark by @MagdalenaKotynia in #455
• fix: fix error with openset launch failed to shutdown error by @MagdalenaKotynia in #467
• feat: ROS2 tools allowlist by @maciejmajek in #486
• chore: update manipulation-demo branch by @maciejmajek in #495
• feat: ROS2 tooling improvements by @maciejmajek in #491
• feat: dynamic services by @rachwalk in #496
• feat: add conversation id to HRI message by @rachwalk in #480
• feat: add service toolkit to demo agent by @rachwalk in #498
• feat: enhance destroy_subscribers behavior by @maciejmajek in #499
• feat: various enhancements by @maciejmajek in #508
• refactor: base api by @maciejmajek in #518
• fix: add base_url parameter to ChatOpenAI initialization by @MxLearner in #519
• refactor: internal api by @maciejmajek in #520
• fix: react agent by @boczekbartek in #523
• refactor: tools by @maciejmajek in #521
• refactor: remove rai.utils module by @maciejmajek in #524
• fix: ros2 import guards by @boczekbartek in #527
• refactor: remove ROS 2 imports from non ROS 2 files in rai_core by @maciejmajek in #528
• refactor: connector api by @maciejmajek in #532
• feat: unregister callback by @maciejmajek in #534
• refactor: gdino and gsam as agents by @rachwalk in #501
• feat: BaseROS2Connector by @maciejmajek in #535
• refactor: HRIConnector remove deprecated methods by @rachwalk in #536
• refactor: micro changes by @maciejmajek in #537
• refactor: rai_bench by @jmatejcz in #517
• docs: update installation guide by @rachwalk in #541
• refactor+fix: add LangChainAgent abstraction and fix HRI communication by @boczekbartek in #538
• docs: add docstring to build_message by @rachwalk in #547
• docs: add docstring to sound device connector by @rachwalk in #548
• fix: collecting messages from agent by @jmatejcz in #544
• feat: unify pydantic models for ros2 types by @boczekbartek in #545
• chore: remove deprecated rai.runners module by @boczekbartek in #552
• fix: floats comparison in tests by @boczekbartek in #549
• docs: add shellcheck installation to developer guide by @boczekbartek in #550
• feat: add StateBaseAgent by @boczekbartek in #529
• refactor: relative imports -> absolute imports by @maciejmajek in #553
• feat: rai_whoami by @maciejmajek in #540
• ...

1.1.0

Major Features

• Added Configuration GUI using Streamlit for easier setup of vendor and settings
• Implemented new Debugging Assistant for ROS 2 network inspection and troubleshooting
• Added support for base_url configuration in OpenAI section to support compatible APIs
• Integrated with rai_state_logs for improved system state monitoring

Improvements

• Converted XML launchfiles to Python for better maintainability
• Updated manipulation demo with Streamlit interface and synchronization improvements
• Improved documentation structure and demo organization
• Implemented second person perspective for identity building
• Added default value for keep_speaker_busy parameter in TTS launchfiles

Bug Fixes

• Fixed empty allowlist behavior in text_hmi (empty allowlist now allows all)
• Fixed wrong argument name and type in priority of task demos
• Fixed various documentation links and references
• Reduced system prompt logging verbosity

Documentation

• Added new debugging assistant documentation
• Updated demo documentation and organization
• Removed external demo repository links from readme
• Added ROSCon 2024 talk reference

Other Changes

• Updated Discord server link
• Added CodeRabbit configuration
• Updated poetry dependencies
• Various code style and license notice formatting improvements

What's Changed

• Talk and community group update by @adamdbrw in #299
• Embodied AI Community Group update by @adamdbrw in #306
• docs: fix link to rosbot-xl python script by @boczekbartek in #305
• chore: coderabbit config by @maciejmajek in #308
• feat: implement configuration gui by @maciejmajek in #304
• docs: update link to roscon2024 talk about rai by @boczekbartek in #310
• Update poetry dependencies by @github-actions in #300
• style: remove additional hash from license notice by @maciejmajek in #311
• fix(text_hmi): empty allowlist == allow all by @maciejmajek in #318
• refactor: add default value for keep_speaker_busy parameter to tts launchfiles by @maciejmajek in #316
• chore: reduce system prompt logging verbosity by @maciejmajek in #315
• build(deps): bump tornado from 6.4.1 to 6.4.2 by @dependabot in #319
• Update poetry dependencies by @github-actions in #321
• feat: implement launchfile for rai_state_logs by @boczekbartek in #322
• fix(demos): wrong argument name and type in priority of task by @boczekbartek in #325
• refactor: xml launchfiles to python by @boczekbartek in #324
• Manipulation demo: Set calibration parameters by @knicked in #330
• feat(manipulation demo): use pretty print of the last message & new system prompt by @maciejmajek in #333
• feat: integrate with rai_state_logs by @boczekbartek in #314
• feat: debugging assistant by @maciejmajek in #326
• feat: allow passing base_url in openai section of config.toml by @maciejmajek in #328
• tests: tool runner by @maciejmajek in #327
• Manipulation demo: Update launching steps by @knicked in #332
• Update poetry dependencies by @github-actions in #329
• chore: Update misleading acronym expansion by @maciejmajek in #342
• docs: update by @maciejmajek in #343
• docs: fix link to agriculture demo by @maciejmajek in #344
• docs: remove demos' repository links from readme by @maciejmajek in #348
• refactor: use second person for identity building by @maciejmajek in #347
• Manipulation demo: Synchronize node launching by @knicked in #341
• Add image display in streamlit tool display by @knicked in #349
• Manipulation demo: Add streamlit interface by @knicked in #337

Full Changelog: 1.0.0...1.1.0

RAI 1.0.0 release

What's Changed

New Features

• Added tools for open-set detection, manipulation, taxi, and agriculture demos
• Enhanced voice-based apps and ROS2 message tools
• Integrated Langfuse for system-wide tracing
• Introduced GenAI URDF parser

Fixes and Improvements

• Various bug fixes in text_hmi, Ros2PubMessageTool, and Turtlebot configuration
• Refactoring for better clarity and modularity (vendor separation, toolset updates)
• Improved task handling, tracing, and documentation

Documentation Updates

• New tutorials and improved docs for ROSbot XL and Turtlebot4
• Added vendor table and demo documentation

Build and CI Updates

• Updated CI setup with self-hosted runners and pre-commit configurations

Chores and Maintenance

• Dependency updates and minor improvements to launch files and extensions

What's Changed

• Update poetry dependencies by @github-actions in #234
• feat: add tool for open-set detection using grounding dino by @rachwalk in #215
• fix(docs and code): text_hmi by @maciejmajek in #238
• docs: improve whoami docs by @boczekbartek in #239
• fix(text_hmi): uids are strings by @boczekbartek in #241
• fix(Ros2PubMessageTool): not all messages has headers + unitest by @boczekbartek in #237
• feat: improve ros2 pub msg tool by @boczekbartek in #242
• build: remove CI on draft PRs by @rachwalk in #244
• fix(hri.launch): use voice_hmi_node instead of hmi_node by @maciejmajek in #246
• feat(rai_tts&rai_hmi): stable elevenlabs voice, reintroduced chunking by @maciejmajek in #247
• refactor: decouple default vendor by @maciejmajek in #248
• feat: Langfuse integration, system-wide tracing configuration (LangSmith&Langfuse) by @maciejmajek in #253
• docs: Added vendor table by @adamdbrw in #255
• refactor: separate files for ToolRunner and artifact handlers by @maciejmajek in #256
• feat: rai_hmi api for universal voice-based apps by @maciejmajek in #257
• feat: insert system prompt when not present by @maciejmajek in #259
• Update poetry dependencies by @github-actions in #254
• feat: genAI urdf parser by @maciejmajek in #260
• CI: update to self-hosted runner by @skrobinho in #245
• feat: taxi demo by @maciejmajek in #250
• chore: add sam2 and change extension name to open_set_vision by @rachwalk in #243
• docs: adapt rosbot-xl docs to adhere to new ros2 action interface by @boczekbartek in #264
• docs&refactor: clarity improvements by @maciejmajek in #267
• CI: pre-commit runner ubuntu image update by @skrobinho in #275
• feat: manipulation toolset by @maciejmajek in #271
• feat: handle background ros2 actions and refactor RaiNodes + integrate w/ open-set vision by @boczekbartek in #273
• bb/refactor/text hmi wide remove legacy by @boczekbartek in #278
• chore: reduce verbosity & feat: enhance typing by @maciejmajek in #282
• fix: adding task to the queue from text hmi by @boczekbartek in #284
• refactor: manipulation toolset by @maciejmajek in #280
• fix: add tracing to the agent in RaiStateBasedLlmNode by @boczekbartek in #288
• feat: add autoware call in taxi-demo by @maciejmajek in #263
• improve RAG formating in streamlit by @boczekbartek in #289
• feat: agriculture demo by @maciejmajek in #285
• docs: add turtlebot4 tutorial by @boczekbartek in #240
• feat/docs: add rosbot xl launchfile and improve demos docs by @boczekbartek in #291
• chore(turtlebot): update links to binaries by @boczekbartek in #294
• chore: add rviz to turtlebot launchfile by @boczekbartek in #293
• docs: release update by @maciejmajek in #283
• fix(turtlebot): vendors config and readme updates by @boczekbartek in #296
• feat: add manipulation demo by @maciejmajek in #295
• feat: manipulation and taxi demo launch files and readme by @maciejmajek in #297

New Contributors

• @skrobinho made their first contribution in #245

Full Changelog: 0.9.0-beta...1.0.0

0.9.0-beta

Summary of changes

This release brings a considerable refactoring effort to the RAI's agent system. RAI architecture now consists of two separate agents:

• HMI agent (for communication, both voice and text)
• RAI agent (for solving tasks/missions)
  The default workflow now is to add a task to the RAI agent via HMI such as text_hmi (streamlit app). The RAI Agent will pick up the task, report feedback on every iteration and return task's report at the very end (success or no).

We have also added a new, refactored version of the voice interface with much better voice detection system. Our ASR now supports (possibly) any recording device (no matter the sampling rate), as well as wake word using OpenWakeWord.

Additionally, other enhancements were introduced:

• RAI What I See (Node for intelligent image capture, based on heuristics (did anything change since the last time?))
• Extended tests
• CI license checking
• NoMaD transformer based navigation system
• Overall docs improvement
• Global configuration file (config.toml) where one can set the default model that will be used across RAI's modules
• Local ASR (Whisper)
• CI has been extended to ROS 2 Humble
• Pydantic V2 compatibility with latest Langchain version

What's Changed

• fix: check status badge by @rachwalk in #173
• hri streaming and led strip publishing by @wsiekierska in #102
• chore: remove use-case specific code by @maciejmajek in #174
• remove build badge until jenkins is setup by @rachwalk in #179
• Implementing license header checking in CI by @john-science in #184
• Update poetry dependencies by @github-actions in #185
• tests: extended coverage by @maciejmajek in #190
• chore(licensechecker.yaml): check license on pr only by @maciejmajek in #193
• ci: add support for running tests on ROS Humble and Jazzy by @maciejmajek in #192
• chore: move rai_grounding_dino to rai_extensions by @maciejmajek in #195
• docs: RAI diagram by @adamdbrw in #197
• feat(rai_cli): improved UX by @maciejmajek in #189
• fix: demo repo dir name by @pijaro in #199
• docs improvements by @adamdbrw in #200
• docs: What is rai by @adamdbrw in #198
• docs: describe development environment setup by @boczekbartek in #201
• chore: remove corrupted file by @maciejmajek in #206
• refactor (human-robot-interface): various enhancements by @maciejmajek in #180
• feat: support multiple shells in the setup script by @rachwalk in #216
• docs(README.md): fix setup section & add informations about global configuration file by @maciejmajek in #210
• Update poetry dependencies by @github-actions in #211
• Refactor of Rosbot XL demo documantation by @wsiekierska in #217
• fix: remove web uri image preprocess test by @maciejmajek in #220
• chore: update tts_clients.py by @eltociear in #221
• chore: fix typos by @maciejmajek in #223
• feat: Add NoMaD extension by @knicked in #218
• refactor: HMI Node by @maciejmajek in #143
• chore: update models' config by @maciejmajek in #226
• docs: update on openVLA by @MagdalenaKotynia in #229
• docs(voice_interface.md): fix typo in docker run command for OpenTTS by @maciejmajek in #222
• chore: Update poetry dependencies (pydantic v2) by @github-actions in #228
• First version of whatisee node by @adamdbrw in #219

New Contributors

Thank you for your contributions!

• @john-science made their first contribution in #184
• @eltociear made their first contribution in #221
• @knicked made their first contribution in #218

Full Changelog: 0.8.1-beta...0.9.0-beta