Merge branch 'development'

Author: atupem

README.md

@@ -103,7 +103,6 @@ More details in the [**Quickstart Guide**](https://docs.bytebot.ai/quickstart).
 | 💬 Chat UI    | `http://localhost:9992`     | Agent UI                 |
 | 🤖 Agent API  | `http://localhost:9991`     | REST API                 |
 | 🌐 noVNC      | `http://localhost:9990/vnc` | open in any browser      |
-| 🖥️ VNC Client | `localhost:5900`            | password‑less by default |
 
 
 

docs/core-concepts/desktop-environment.mdx

@@ -47,14 +47,6 @@ The desktop environment is configured with automation in mind:
 
 ## Remote Access
 
-### VNC Server
-
-The container runs a VNC server that allows direct access to the desktop:
-
-- Accessible on port 5900 (default VNC port)
-- Compatible with standard VNC clients (RealVNC, TightVNC, etc.)
-- Supports standard VNC authentication
-
 ### noVNC
 
 For browser-based access, noVNC is included:
@@ -76,10 +68,9 @@ Bytebot uses Xvfb (X Virtual Framebuffer) as its display server:
 
 ### Manual Interaction
 
-You can directly interact with the desktop environment using:
+You can interact with the desktop environment using your web browser:
 
-1. **VNC client** connected to `localhost:5900`
-2. **Web browser** navigated to `http://localhost:9990/vnc`
+1. Navigate to `http://localhost:9990/vnc`
 
 ### Programmatic Interaction
 

docs/quickstart.mdx

@@ -26,11 +26,8 @@ Getting started with Bytebot is simple and straightforward. You can run it as a
         docker-compose -f infrastructure/docker/docker-compose.core.yml up -d --no-build # start container
         ```
 
-        This will start Bytebot with default settings. The container exposes several ports:
+        This will start Bytebot with default settings. The container exposes one port:
         - `9990`: REST API and noVNC web access
-        - `5900`: VNC server
-        - `6080`: noVNC direct
-        - `6081`: noVNC HTTP proxy
       </Accordion>
 
       <Accordion icon="cube" title="Building the Docker Image (Alternative)">
@@ -52,10 +49,6 @@ Getting started with Bytebot is simple and straightforward. You can run it as a
       <Accordion icon="display" title="Accessing the Desktop">
         You can access the Bytebot desktop environment in two ways:
 
-        **Using a VNC Client**:
-        Connect to `localhost:5900` with any VNC client.
-
-        **Using a Web Browser**:
         Navigate to `http://localhost:9990/vnc` in your web browser for noVNC access.
       </Accordion>
     </AccordionGroup>

docs/rest-api/input-tracking.mdx

@@ -0,0 +1,41 @@
+---
+title: "Input Tracking"
+openapi: "POST /input-tracking/start"
+description: "Start and stop input tracking on the Bytebot desktop"
+---
+
+The Bytebot daemon can monitor mouse and keyboard events through the
+`InputTracking` module. Tracking is disabled by default and can be toggled
+via the REST API. Tracked actions are streamed over WebSockets so that the
+agent can store them as messages.
+
+## Start Tracking
+
+`POST /input-tracking/start`
+
+Begins capturing input events. The endpoint returns a simple status object:
+
+```json
+{
+  "status": "started"
+}
+```
+
+## Stop Tracking
+
+`POST /input-tracking/stop`
+
+Stops capturing events and clears any internal buffers. The response is
+similar to the start endpoint:
+
+```json
+{
+  "status": "stopped"
+}
+```
+
+## WebSocket Stream
+
+When tracking is active, actions are emitted on the `input_action` channel of
+the WebSocket server running on the daemon. Clients can connect to the daemon
+and listen for these events to persist them as needed.

docs/rest-api/introduction.mdx

@@ -64,6 +64,9 @@ Common HTTP status codes:
     Execute desktop automation actions like mouse movements, clicks, keyboard
     input, and screenshots
   </Card>
+  <Card title="Input Tracking" icon="eye" href="/rest-api/input-tracking">
+    Control and stream keyboard and mouse events from the desktop
+  </Card>
   <Card title="Usage Examples" icon="code" href="/rest-api/examples">
     Code examples and snippets for common automation scenarios
   </Card>

infrastructure/docker/Dockerfile

@@ -203,17 +203,28 @@ RUN git clone https://github.com/novnc/noVNC.git /opt/noVNC \
 # -----------------------------------------------------------------------------
 # Copy package files first to leverage Docker cache
 
-# Install dependencies required to build libnut-core
+# Install dependencies required to build libnut-core and uiohook-napi
 RUN apt-get update && \
-    apt-get install -y cmake libx11-dev libxtst-dev libxinerama-dev libxi-dev libxrandr-dev git build-essential && \
+    apt-get install -y \
+        cmake \
+        libx11-dev \
+        libxtst-dev \
+        libxinerama-dev \
+        libxi-dev \
+        libxt-dev \
+        libxrandr-dev \
+        libxkbcommon-dev \
+        libxkbcommon-x11-dev \
+        git build-essential && \
     rm -rf /var/lib/apt/lists/*
 
 
 COPY ./packages/bytebotd/ /bytebotd/
 WORKDIR /bytebotd
 RUN npm install --build-from-source
+RUN npm rebuild uiohook-napi --build-from-source
 
-RUN npm run build 
+RUN npm run build
 
 WORKDIR /compile
 
@@ -387,11 +398,8 @@ WORKDIR /home/bytebot
 # -----------------------------------------------------------------------------
 # 8. Port configuration and runtime
 # -----------------------------------------------------------------------------
-# - Port 9990: bytebotd
-# - Port 5900: VNC display for the Ubuntu VM
-# - Port 6080: noVNC client
-# - Port 6081: noVNC HTTP proxy
-EXPOSE 9990 5900 6080 6081
+# - Port 9990: bytebotd and external noVNC websocket
+EXPOSE 9990
 
 # Start supervisor to manage all services
 CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf", "-n"]

infrastructure/docker/docker-compose.core.yml

@@ -14,9 +14,6 @@ services:
     hostname: computer
     privileged: true
     ports:
-      - "9990:9990" # bytebotd service
-      - "5900:5900" # VNC display
-      - "6080:6080" # noVNC client
-      - "6081:6081" # noVNC HTTP proxy
+      - "9990:9990" # bytebotd service & noVNC
     environment:
       - DISPLAY=:0

infrastructure/docker/docker-compose.yml

@@ -14,10 +14,7 @@ services:
     hostname: computer
     privileged: true
     ports:
-      - "9990:9990" # bytebotd service
-      - "5900:5900" # VNC display
-      - "6080:6080" # noVNC client
-      - "6081:6081" # noVNC HTTP proxy
+      - "9990:9990" # bytebotd service & noVNC
     environment:
       - DISPLAY=:0
     networks:
@@ -60,14 +57,16 @@ services:
       context: ../../packages/
       dockerfile: bytebot-ui/Dockerfile
       args:
-        - NEXT_PUBLIC_BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://localhost:9991}
-        - NEXT_PUBLIC_BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-ws://localhost:6080}
+        - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent:9991}
+        - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}
     container_name: bytebot-ui
     restart: unless-stopped
     ports:
       - "9992:9992"
     environment:
       - NODE_ENV=production
+      - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent:9991}
+      - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}
     depends_on:
       - bytebot-agent
     networks:

infrastructure/docker/supervisord.conf

@@ -68,17 +68,6 @@ stdout_logfile_maxbytes=0
 redirect_stderr=true
 depends_on=x11vnc
 
-[program:novnc-http]
-command=python3 -m http.server 6081 --directory /opt/noVNC
-autostart=true
-autorestart=true
-startsecs=5
-priority=50
-stdout_logfile=/dev/stdout
-stdout_logfile_maxbytes=0
-redirect_stderr=true
-depends_on=websockify
-
 [program:bytebotd]
 command=node /bytebotd/dist/main.js
 directory=/bytebotd
@@ -90,7 +79,7 @@ environment=DISPLAY=":0"
 stdout_logfile=/dev/stdout
 stdout_logfile_maxbytes=0
 redirect_stderr=true
-depends_on=novnc-http
+depends_on=websockify
 
 [eventlistener:startup]
 command=echo "All services started successfully"