EMOS workshop recipes

Presented at ROSCon FR 2026, Paris.

Self-contained recipe scripts we walk through, in order, to show EMOS capabilities. Each recipe is a single Python file; the shared model/server addresses live in endpoints.py (fill in once).

endpoints.py                        shared model endpoints (fill in once)
01_conversational_vlm_agent.py      text Q&A about the webcam, spoken answers
02_proactive_perception_agent.py    autonomous greeter — an Event triggers an Action
03_point_navigation.py              click a goal on the map, robot drives there
04_point_navigation_with_events.py  nav that reacts — auto-dispatch goals, self-unblock
05_go_to_what_you_see.py            describe a target, robot grounds it and drives there
06_adaptive_swap.py                 hot-swap models on a GPU-load Event, with a fallback
07_semantic_routing.py              route a query to the VLM or the LLM by intent
08_cortex_agent.py                  Cortex plans and executes a task from a goal
09_cortex_with_memory.py            Cortex + navigation + spatio-temporal Memory (finale)
11_gender_reveal_participant_submission.py   a participant's quick VLM classifier (live submission)

Where the models run

Everything runs locally now — the workshop's hosted endpoint is offline, so the LLM/VLM recipes use a local Ollama. endpoints.py defaults every service to 127.0.0.1, so if you run them on your machine there's nothing to edit.

Used by recipes	Service	endpoints.py keys
LLM / VLM — 01, 02, 05, 07, 08, 09, 11	Ollama (local)	OLLAMA_HOST
Embeddings — 06, 07, 09	Ollama (local)	OLLAMA_HOST
TTS / RoboBrain — 01, 02, 05	RoboML container (needs a GPU)	ROBOML_HOST, ROBOML_PORT
Route vector store — 07	ChromaDB (local, port 8080)	CHROMA_HOST, CHROMA_PORT
Local detector / Kokoro TTS	on-device (onnxruntime / sherpa, in the install)	—
Navigation — 03, 04, 05, 08, 09	Kompass, on-device GPU	—

Navigation recipes 03 / 04 are pure on-device — no endpoints needed.

One-time setup

1. Update EMOS first: run emos update to pull the latest stack — do this before running any recipe.
2. endpoints.py already defaults every service to 127.0.0.1 — only edit it if a service runs on a different host/port than the defaults below.
3. Bring up the EMOS env — do this in every terminal you run a recipe from:

   cd ~/.local/share/emos
   pixi shell
   source install/setup.bash

4. Extra packages, installed once into the pixi env (only for the recipes that need them):

   Package	For	Install
   ros-jazzy-usb-cam	webcam (01, 02, 07, 11)	pixi add ros-jazzy-usb-cam
   emem	Memory (09)	pixi add --pypi emem
   kompass_sim	nav sim (03, 04, 05, 08, 09)	native build or Docker — see Simulation

Model services (start the ones your recipe needs)

Ollama — LLM / VLM / embeddings (01, 02, 05, 06, 07, 08, 09, 11). Install from ollama.com, then pull the models the recipes use:

ollama serve                              # runs on 127.0.0.1:11434
ollama pull qwen3.5                        # LLM + VLM (use a multimodal tag)
ollama pull nomic-embed-text-v2-moe        # embeddings (07, 09)
ollama pull llama3.2:3b && ollama pull moondream   # recipe 06 swap pair

The recipes set the model tag inline (checkpoint="qwen3.5:latest") — change it to whatever multimodal/LLM model you actually pulled.

RoboML — TTS / STT / RoboBrain (01, 02, 05). Needs a GPU. Run the container and point ROBOML_HOST/ROBOML_PORT at it (defaults 127.0.0.1:8000):

docker run --rm --gpus all -p 8000:8000 automatika/roboml:latest

No GPU? Use 01_local_tts_alternative.py, which runs TTS on-device (no RoboML).

ChromaDB — route vector store (07). Install and run it locally on port 8080 (CHROMA_HOST/CHROMA_PORT default to 127.0.0.1:8080):

pip install chromadb
chroma run --host 127.0.0.1 --port 8080

Simulation (nav recipes 03, 04, 05, 08, 09)

The navigation recipes need the kompass-sim Webots simulation. Two ways to run it.

Native — build the package in a ROS 2 workspace (see the kompass-sim README), then in its own terminal:

ros2 launch kompass_sim webots_turtlebot3.launch.py        # 2D lidar — 03, 04, 08
ros2 launch kompass_sim webots_turtlebot3_rgbd.launch.py   # RGBD     — 05, 09

Docker (recommended if you don't have ROS 2) — a prebuilt ROS 2 Jazzy image that already ships Webots. Run it with --network host so a recipe running on the host can discover the sim over DDS, plus X-server + GPU access so the windows show up.

docker pull automatika/kompass-sim:jazzy   # once
xhost +local:root                          # once per login session

# 2D lidar sim (recipes 03, 04, 08) — this is the image's default command
docker run -it --rm \
  --network host --ipc host \
  --env DISPLAY="$DISPLAY" --env QT_X11_NO_MITSHM=1 \
  --volume /tmp/.X11-unix:/tmp/.X11-unix:rw \
  --device /dev/dri:/dev/dri \
  automatika/kompass-sim:jazzy

# RGBD sim (recipes 05, 09) — override the launch file
docker run -it --rm \
  --network host --ipc host \
  --env DISPLAY="$DISPLAY" --env QT_X11_NO_MITSHM=1 \
  --volume /tmp/.X11-unix:/tmp/.X11-unix:rw \
  --device /dev/dri:/dev/dri \
  automatika/kompass-sim:jazzy ros2 launch kompass_sim webots_turtlebot3_rgbd.launch.py

Notes

• NVIDIA GPUs: install the NVIDIA Container Toolkit and add --gpus all --env NVIDIA_DRIVER_CAPABILITIES=all (you can drop --device /dev/dri).
• Discovery: with --network host the sim and your recipe talk over DDS. The recipe's pixi/python flow and the kompass-sim container both run on the default ROS 2 middleware (Fast DDS), so they discover each other with no RMW config. Just keep ROS_DOMAIN_ID the same on both (defaults to 0; pass --env ROS_DOMAIN_ID=<id> to the container if you set one on the host). Only a concern if you've manually exported RMW_IMPLEMENTATION — note the EMOS containers default to zenoh — in which case set the sim container to the same value.
• X11 trouble? --privileged is a coarser alternative that also grants display access.
• Build it yourself: docker build -t automatika/kompass-sim:jazzy . from the kompass-sim repo root instead of pulling.

Running a recipe

In the sourced pixi shell:

cd ~/emos-workshop-recipes
python 0X_<recipe>.py

Most recipes open an auto-generated dashboard at http://localhost:5001 — type input and see output there. Stop with Ctrl-C.

---

The recipes

01 — Conversational VLM agent

Type a question; the agent looks at the live webcam and answers you out loud. Two components + one enable_ui() call — a vision-grounded agent that talks back, with zero frontend code.

• Needs: Ollama (VLM), RoboML container (TTS), webcam on /image_raw.
• Run:

  ros2 run usb_cam usb_cam_node_exe &
  python 01_conversational_vlm_agent.py

  Open localhost:5001, type "what do you see?".

02 — Proactive perception agent (Events)

Autonomous, not reactive: a cheap local detector fires an Event when a person appears, which triggers the VLM to greet them — the core EMOS Event→Action idea.

• Needs: Ollama (VLM), RoboML container (TTS), webcam. First run downloads the detector model.
• Run:

  ros2 run usb_cam usb_cam_node_exe &
  python 02_proactive_perception_agent.py

  Open localhost:5001, then step into frame — it greets you.

03 — Point-to-point navigation

Click a goal on the map and the full Kompass stack (global planner + DWA controller

• drive manager + safety zones) drives there. On-device, GPU-accelerated, no cloud.

• Needs: the nav sim and a 2D map (the turtlebot3_webots.yaml here works).
• Run:

  ros2 launch kompass_sim webots_turtlebot3.launch.py    # in its own terminal
  python 03_point_navigation.py

  Open the dashboard, click a point on the map. No ROS 2 / sim won't start natively? Run the sim in Docker instead — see Simulation.

04 — Navigation that reacts (Events on the nav stack)

Recipe 03's exact stack plus two Events: auto-dispatch the goal on a map click, and self-unblock when stuck (a composed Event: emergency-stop OR controller failure → DriveManager.move_to_unblock()).

• Needs: same as 03.
• Run: same as 03, with python 04_point_navigation_with_events.py. Click to send goals and watch it recover.

05 — Go to what you see (instructions only — we won't run this live)

Describe a target in plain language; an LLM extracts the object, RoboBrain (a VLM) grounds it in the live RGBD frame, and that point becomes the Planner's goal.

• Needs: an RGBD sim (/camera/rgbd, depth camera_info, /scan, odom), Ollama (LLM), RoboML container (RoboBrain).
• Run:

  ros2 launch kompass_sim webots_turtlebot3_rgbd.launch.py
  python 05_go_to_what_you_see.py

  Open the dashboard, type a target like "go to the chair".

06 — Adaptive model swap (Events + Fallback)

A VLM answers visual questions while an Action polls GPU utilization; when load goes high an Event hot-swaps to a smaller model (in-process, via EMOS IPC), and on_algorithm_fail is a safety-net fallback if the model server dies.

• Needs: an Ollama server with both models (llama3.2:3b, moondream); pip install pynvml.
• Run:

  python 06_adaptive_swap.py
  python simulate_gpu_load.py     # separate terminal — fakes GPU throttle to trigger the swap

07 — Semantic routing

One question topic, routed by intent: "what's in front of you?" → VLM (camera), "capital of France?" → LLM. Vector mode embeds the route samples in ChromaDB and matches the query. Both routes are served by the local Ollama.

• Needs: Ollama (VLM + LLM + embeddings), a local ChromaDB server (port 8080), webcam on /image_raw.
• Run:

  ros2 run usb_cam usb_cam_node_exe &
  python 07_semantic_routing.py

  Open the dashboard, type a question and watch it land on the VLM or the LLM.

08 — Cortex agent

Cortex (an LLM task-planner that is also the system monitor) turns a goal into a plan and executes it across components: take a picture, describe it, speak the description, toggle an LED (a custom action).

• Needs: Ollama (VLM + LLM, tool-calling), local TTS, kompass_sim (for the map).
• Run: python 08_cortex_agent.py, then give it a goal from the dashboard, or:

  ros2 action send_goal /cortex_<pid>/vision_language_action \
    automatika_embodied_agents/action/VisionLanguageAction "{task: 'describe what you see'}"

09 — Cortex + Navigation + Memory (finale)

The full Physical AI agent: Vision + Memory (eMEM) build a spatio-temporal map of what was seen and where; Cortex plans, queries memory ("where did I last see X?"), and dispatches the navigation stack.

• Needs: the RGBD nav sim, Ollama (planner + VLM + embeddings), emem installed (pixi add --pypi emem).
• Run:

  ros2 launch kompass_sim webots_turtlebot3_rgbd.launch.py
  python 09_cortex_with_memory.py

  Open the dashboard and give Cortex a goal — it remembers as it goes.

---

10_robot_plugin.py is recipe 09 deployed on a DeepRobotics Lite3 quadruped via the Lite3 plugin. It needs the robot + plugin, so it's out of scope for the workshop run.

Speech input. The conversational recipes (01, 07) take the question as text, and the Cortex recipes (08, 09) take the goal as text. To drive any of them by voice instead, add a SpeechToText component in front — see the commented note at the bottom of recipe 01 (it feeds the question topic); the local RoboML container serves Whisper for it.