|
| 1 | +# Coffee Voice Agent UI |
| 2 | + |
| 3 | +A comprehensive PyQt-based monitoring dashboard for the Coffee Voice Agent system. This package provides real-time visualization and control of voice agent status, emotions, tool usage, conversation flow, and system analytics. |
| 4 | + |
| 5 | +   |
| 6 | + |
| 7 | +## 🎯 Overview |
| 8 | + |
| 9 | +The Coffee Voice Agent UI is a standalone ROS2 application that provides comprehensive monitoring and control capabilities for the coffee robot's voice interaction system. It connects to the voice agent bridge to display real-time status information, conversation transcripts, emotional states, and system performance metrics. |
| 10 | + |
| 11 | +### Key Features |
| 12 | + |
| 13 | +- **🤖 Real-time Agent Status**: Monitor connection state, behavioral mode, and session information |
| 14 | +- **🎭 Emotion Visualization**: Track current emotions, transitions, and emotional journey |
| 15 | +- **💬 Live Conversation Flow**: Real-time transcript with user speech and agent responses |
| 16 | +- **🔧 Tool Activity Monitor**: Function tool execution tracking with performance metrics |
| 17 | +- **📊 Analytics Dashboard**: Usage statistics, performance trends, and system health |
| 18 | +- **⚙️ Manual Controls**: Testing interface for virtual requests and debug commands |
| 19 | + |
| 20 | +## 📋 Prerequisites |
| 21 | + |
| 22 | +- **ROS2 Jazzy** (or compatible version) |
| 23 | +- **Python 3.8+** |
| 24 | +- **PyQt5** (installed via `python_qt_binding`) |
| 25 | +- **coffee_voice_agent_msgs** package |
| 26 | +- **coffee_voice_agent** package (for full system operation) |
| 27 | + |
| 28 | +## 🚀 Installation |
| 29 | + |
| 30 | +### 1. Clone and Build |
| 31 | + |
| 32 | +```bash |
| 33 | +# Navigate to your ROS2 workspace |
| 34 | +cd ~/your_ros2_ws/src |
| 35 | + |
| 36 | +# Build the package |
| 37 | +cd .. |
| 38 | +colcon build --packages-select coffee_voice_agent_ui |
| 39 | + |
| 40 | +# Source the workspace |
| 41 | +source install/setup.bash |
| 42 | +``` |
| 43 | + |
| 44 | +### 2. Verify Installation |
| 45 | + |
| 46 | +```bash |
| 47 | +# Check if the package is available |
| 48 | +ros2 pkg list | grep coffee_voice_agent_ui |
| 49 | + |
| 50 | +# Check available executables |
| 51 | +ros2 pkg executables coffee_voice_agent_ui |
| 52 | +``` |
| 53 | + |
| 54 | +## 🎮 Usage |
| 55 | + |
| 56 | +### Option 1: Standalone Monitor (Recommended) |
| 57 | + |
| 58 | +Launch just the monitoring UI to connect to an existing voice agent system: |
| 59 | + |
| 60 | +```bash |
| 61 | +# Direct execution |
| 62 | +ros2 run coffee_voice_agent_ui voice_agent_monitor |
| 63 | + |
| 64 | +# Or using launch file with parameters |
| 65 | +ros2 launch coffee_voice_agent_ui monitor.launch.py \ |
| 66 | + ui_node_name:=voice_monitor \ |
| 67 | + log_level:=debug |
| 68 | +``` |
| 69 | + |
| 70 | +### Option 2: Full System Launch |
| 71 | + |
| 72 | +Launch both the voice agent bridge and monitor UI together: |
| 73 | + |
| 74 | +```bash |
| 75 | +ros2 launch coffee_voice_agent_ui full_system.launch.py \ |
| 76 | + voice_agent_host:=localhost \ |
| 77 | + voice_agent_port:=8080 \ |
| 78 | + enable_ui:=true |
| 79 | +``` |
| 80 | + |
| 81 | +### Option 3: RQT Plugin (Development) |
| 82 | + |
| 83 | +For development purposes, the UI can also be loaded as an RQT plugin: |
| 84 | + |
| 85 | +```bash |
| 86 | +# Load in rqt |
| 87 | +rqt |
| 88 | + |
| 89 | +# Navigate to: Plugins → Coffee Robot → Voice Agent Monitor |
| 90 | +# Or run standalone rqt plugin |
| 91 | +rqt --standalone coffee_voice_agent_ui |
| 92 | +``` |
| 93 | + |
| 94 | +## 🏗️ Architecture |
| 95 | + |
| 96 | +### System Overview |
| 97 | + |
| 98 | +``` |
| 99 | +Voice Agent ←→ Voice Agent Bridge ←→ ROS2 Topics ←→ Monitor UI |
| 100 | +``` |
| 101 | + |
| 102 | +### Component Architecture |
| 103 | + |
| 104 | +``` |
| 105 | +VoiceAgentMonitorApp (QMainWindow) |
| 106 | +├── Agent Status Widget (connection, state, metrics) |
| 107 | +├── Emotion Display Widget (current emotion, transitions, timeline) |
| 108 | +├── Conversation Widget (live transcript, timeouts, turn tracking) |
| 109 | +├── Tool Monitor Widget (function tool usage, performance) |
| 110 | +├── Analytics Widget (usage trends, performance metrics) |
| 111 | +└── Controls Widget (manual testing, debug commands) |
| 112 | +``` |
| 113 | + |
| 114 | +### ROS2 Integration |
| 115 | + |
| 116 | +The monitor subscribes to these topics: |
| 117 | +- `voice_agent/status` (AgentStatus) - Unified agent state information |
| 118 | +- `voice_agent/tool_events` (ToolEvent) - Function tool execution events |
| 119 | +- `voice_agent/user_speech` (String) - User speech transcriptions |
| 120 | +- `voice_agent/connected` (Bool) - Connection status updates |
| 121 | + |
| 122 | +The monitor publishes to these topics: |
| 123 | +- `voice_agent/virtual_requests` (String) - Test virtual coffee requests |
| 124 | +- `voice_agent/commands` (String) - Debug and control commands |
| 125 | + |
| 126 | +## 📊 Dashboard Layout |
| 127 | + |
| 128 | +``` |
| 129 | +┌─────────────────┬─────────────────┬─────────────────┐ |
| 130 | +│ 🤖 Agent Status │ 💬 Conversation │ 📊 Analytics │ |
| 131 | +│ • State & Mode │ Flow │ • Performance │ |
| 132 | +│ • Connection │ • Live Chat │ • Usage Stats │ |
| 133 | +│ • Session Info │ • Timeouts │ • Trends │ |
| 134 | +├─────────────────┼─────────────────┼─────────────────┤ |
| 135 | +│ 🎭 Emotion │ 🔧 Tool Monitor │ ⚙️ Controls │ |
| 136 | +│ Center │ • Active Tools │ • Virtual Reqs │ |
| 137 | +│ • Current │ • Recent Calls │ • Commands │ |
| 138 | +│ • Transitions │ • Statistics │ • Debug Tools │ |
| 139 | +└─────────────────┴─────────────────┴─────────────────┘ |
| 140 | +``` |
| 141 | + |
| 142 | +## 🎛️ Features Guide |
| 143 | + |
| 144 | +### Agent Status Panel |
| 145 | +- **Real-time state indicator**: Visual status with color coding |
| 146 | +- **Connection monitoring**: WebSocket connection health |
| 147 | +- **Session tracking**: Active conversation duration and metrics |
| 148 | +- **Phase indication**: Current conversation phase (greeting, discussion, etc.) |
| 149 | + |
| 150 | +### Emotion Center |
| 151 | +- **Current emotion display**: Large visual indicator with emoji |
| 152 | +- **Transition tracking**: Previous → current emotion flow |
| 153 | +- **Emotion timeline**: Visual history of emotional journey |
| 154 | +- **Eye animation preview**: How emotions translate to robot expressions |
| 155 | + |
| 156 | +### Conversation Flow |
| 157 | +- **Live transcript**: Real-time user and agent messages |
| 158 | +- **Tool integration**: Function tool calls inline with conversation |
| 159 | +- **Timeout monitoring**: User response countdown with visual indicators |
| 160 | +- **Turn tracking**: Conversation metrics and response times |
| 161 | + |
| 162 | +### Tool Monitor |
| 163 | +- **Active tracking**: Currently executing tools with duration |
| 164 | +- **Recent activity**: Last 10 tool calls with status and timing |
| 165 | +- **Usage statistics**: Success rates, execution times, frequency |
| 166 | +- **Performance metrics**: Tool reliability and efficiency tracking |
| 167 | + |
| 168 | +### Analytics |
| 169 | +- **Session performance**: Daily conversation count, success rates |
| 170 | +- **Popular interactions**: Most requested functions and topics |
| 171 | +- **Emotion trends**: Emotional state patterns over time |
| 172 | +- **System metrics**: Message rates, connection health, queue status |
| 173 | + |
| 174 | +### Controls Panel |
| 175 | +- **Quick actions**: End conversation, reset state, pause wake word |
| 176 | +- **Virtual request testing**: Simulate coffee orders with different priorities |
| 177 | +- **Command interface**: Send debug commands with parameters |
| 178 | +- **Debug tools**: Connection testing, log export, tool triggers |
| 179 | + |
| 180 | +## 📁 Launch Files |
| 181 | + |
| 182 | +### `monitor.launch.py` |
| 183 | +Launches only the monitor UI for connecting to existing voice agent systems. |
| 184 | + |
| 185 | +**Parameters:** |
| 186 | +- `ui_node_name` (default: "voice_agent_monitor") - Name for the UI node |
| 187 | +- `log_level` (default: "info") - Logging level |
| 188 | +- `window_title` (default: "Coffee Voice Agent Monitor") - Window title |
| 189 | + |
| 190 | +### `full_system.launch.py` |
| 191 | +Launches both the voice agent bridge and monitor UI for complete system operation. |
| 192 | + |
| 193 | +**Parameters:** |
| 194 | +- `voice_agent_host` (default: "localhost") - Voice agent WebSocket host |
| 195 | +- `voice_agent_port` (default: "8080") - Voice agent WebSocket port |
| 196 | +- `enable_ui` (default: "true") - Enable/disable monitor UI |
| 197 | +- `log_level` (default: "info") - Logging level for all nodes |
| 198 | + |
| 199 | +## 🔧 Configuration |
| 200 | + |
| 201 | +### Environment Variables |
| 202 | +- `QT_AUTO_SCREEN_SCALE_FACTOR=1` - Qt high DPI scaling |
| 203 | +- `ROS_DOMAIN_ID` - ROS2 domain for multi-robot systems |
| 204 | + |
| 205 | +### Parameters |
| 206 | +The monitor can be configured through ROS2 parameters: |
| 207 | +- `use_sim_time` (default: false) - Use simulation time |
| 208 | +- `window_title` - Custom window title |
| 209 | + |
| 210 | +## 🐛 Troubleshooting |
| 211 | + |
| 212 | +### Common Issues |
| 213 | + |
| 214 | +**1. "No module named 'coffee_voice_agent_msgs'"** |
| 215 | +```bash |
| 216 | +# Ensure the messages package is built and sourced |
| 217 | +colcon build --packages-select coffee_voice_agent_msgs |
| 218 | +source install/setup.bash |
| 219 | +``` |
| 220 | + |
| 221 | +**2. Qt High DPI Warnings** |
| 222 | +```bash |
| 223 | +# Set environment variable before running |
| 224 | +export QT_AUTO_SCREEN_SCALE_FACTOR=1 |
| 225 | +ros2 run coffee_voice_agent_ui voice_agent_monitor |
| 226 | +``` |
| 227 | + |
| 228 | +**3. No Data in UI Panels** |
| 229 | +- Check if voice agent bridge is running: `ros2 node list | grep voice_agent_bridge` |
| 230 | +- Verify topic publications: `ros2 topic list | grep voice_agent` |
| 231 | +- Check topic data: `ros2 topic echo voice_agent/status` |
| 232 | + |
| 233 | +**4. UI Won't Start** |
| 234 | +```bash |
| 235 | +# Check PyQt installation |
| 236 | +python3 -c "from python_qt_binding.QtWidgets import QApplication; print('PyQt OK')" |
| 237 | + |
| 238 | +# Check ROS2 dependencies |
| 239 | +ros2 pkg deps coffee_voice_agent_ui |
| 240 | +``` |
| 241 | + |
| 242 | +### Debug Mode |
| 243 | + |
| 244 | +Run with debug logging for detailed information: |
| 245 | +```bash |
| 246 | +ros2 run coffee_voice_agent_ui voice_agent_monitor --ros-args --log-level debug |
| 247 | +``` |
| 248 | + |
| 249 | +## 🔗 Related Packages |
| 250 | + |
| 251 | +- **coffee_voice_agent** - Core voice agent implementation |
| 252 | +- **coffee_voice_agent_msgs** - ROS2 message definitions |
| 253 | +- **coffee_vision_ui** - Camera monitoring interface |
| 254 | +- **coffee_expressions** - Emotion and expression control |
| 255 | + |
| 256 | +## 📝 Development |
| 257 | + |
| 258 | +### Adding New Widgets |
| 259 | + |
| 260 | +1. Create widget in `coffee_voice_agent_ui/widgets/` |
| 261 | +2. Inherit from `QWidget` and implement required interfaces |
| 262 | +3. Add to main layout in `voice_agent_monitor_app.py` |
| 263 | +4. Connect to ROS2 data streams as needed |
| 264 | + |
| 265 | +### Extending Analytics |
| 266 | + |
| 267 | +1. Add data collection in existing widgets' `get_analytics_data()` methods |
| 268 | +2. Update `AnalyticsWidget` to process new data types |
| 269 | +3. Create visualizations in analytics panel |
| 270 | + |
| 271 | +## 📄 License |
| 272 | + |
| 273 | +TODO: License declaration |
| 274 | + |
| 275 | +## 🤝 Contributing |
| 276 | + |
| 277 | +This package is part of the Coffee Robot project. Please follow the project's coding conventions and submit pull requests for review. |
| 278 | + |
| 279 | +--- |
| 280 | + |
| 281 | +**For support or questions about the Coffee Voice Agent UI, please refer to the main project documentation or contact the development team.** |
0 commit comments