[#315] Updates: add advanced audio support

Author: pennycoders

.devcontainer/devcontainer.json

@@ -1,10 +1,15 @@
 {
 	"name": "JetKVM",
-	"image": "mcr.microsoft.com/devcontainers/go:1-1.23-bookworm",
+	"image": "mcr.microsoft.com/devcontainers/base:ubuntu-22.04",
+	"runArgs": ["--platform=linux/amd64" ],
 	"features": {
 		"ghcr.io/devcontainers/features/node:1": {
 			// Should match what is defined in ui/package.json
 			"version": "22.15.0"
+		},
+		"ghcr.io/devcontainers/features/go:1": {
+			// Should match what is defined in go.mod
+			"version": "latest"
 		}
 	},
 	"mounts": [

.gitignore

@@ -1,6 +1,12 @@
 bin/*
 static/*
+.vscode/
+tmp/
+.devcontainer/devcontainer-lock.json
 .idea
 .DS_Store
+*.log
+*.tmp
+*.code-workspace
 
 device-tests.tar.gz

DEVELOPMENT.md

@@ -11,28 +11,47 @@
 
 </div>
 
+
 # JetKVM Development Guide
 
+
 Welcome to JetKVM development! This guide will help you get started quickly, whether you're fixing bugs, adding features, or just exploring the codebase.
 
 ## Get Started
 
+
 ### Prerequisites
 - **A JetKVM device** (for full development)
 - **[Go 1.24.4+](https://go.dev/doc/install)** and **[Node.js 22.15.0](https://nodejs.org/en/download/)**
 - **[Git](https://git-scm.com/downloads)** for version control
 - **[SSH access](https://jetkvm.com/docs/advanced-usage/developing#developer-mode)** to your JetKVM device
+- **Audio build dependencies:**
+   - **New in this release:** The audio pipeline is now fully in-process using CGO, ALSA, and Opus. You must run the provided scripts in `tools/` to set up the cross-compiler and build static ALSA/Opus libraries for ARM. See below.
+
 
 ### Development Environment
 
-**Recommended:** Development is best done on **Linux** or **macOS**. 
+**Recommended:** Development is best done on **Linux** or **macOS**.
+
+#### Apple Silicon (M1/M2/M3) Mac Users
+
+If you are developing on an Apple Silicon Mac, you should use a devcontainer to ensure compatibility with the JetKVM build environment (which targets linux/amd64 and ARM). There are two main options:
+
+- **VS Code Dev Containers**: Open the project in VS Code and use the built-in Dev Containers support. The configuration is in `.devcontainer/devcontainer.json`.
+- **Devpod**: [Devpod](https://devpod.sh/) is a fast, open-source tool for running devcontainers anywhere. If you use Devpod, go to **Settings → Experimental → Additional Environmental Variables** and add:
+   - `DOCKER_DEFAULT_PLATFORM=linux/amd64`
+   This ensures all builds run in the correct architecture.
+- **devcontainer CLI**: You can also use the [devcontainer CLI](https://github.com/devcontainers/cli) to launch the devcontainer from the terminal.
+
+This approach ensures compatibility with all shell scripts, build tools, and cross-compilation steps used in the project.
 
 If you're using Windows, we strongly recommend using **WSL (Windows Subsystem for Linux)** for the best development experience:
 - [Install WSL on Windows](https://docs.microsoft.com/en-us/windows/wsl/install)
 - [WSL Setup Guide](https://docs.microsoft.com/en-us/windows/wsl/setup/environment)
 
 This ensures compatibility with shell scripts and build tools used in the project.
 
+
 ### Project Setup
 
 1. **Clone the repository:**
@@ -46,16 +65,25 @@ This ensures compatibility with shell scripts and build tools used in the projec
    go version && node --version
    ```
 
-3. **Find your JetKVM IP address** (check your router or device screen)
+3. **Set up the cross-compiler and audio dependencies:**
+   ```bash
+   make dev_env
+   # This will run tools/setup_rv1106_toolchain.sh and tools/build_audio_deps.sh
+   # It will clone the cross-compiler and build ALSA/Opus static libs in $HOME/.jetkvm
+   #
+   # **Note:** This is required for the new in-process audio pipeline. If you skip this step, audio will not work.
+   ```
+
+4. **Find your JetKVM IP address** (check your router or device screen)
 
-4. **Deploy and test:**
+5. **Deploy and test:**
    ```bash
    ./dev_deploy.sh -r 192.168.1.100  # Replace with your device IP
    ```
 
-5. **Open in browser:** `http://192.168.1.100`
+6. **Open in browser:** `http://192.168.1.100`
 
-That's it! You're now running your own development version of JetKVM.
+That's it! You're now running your own development version of JetKVM, **with in-process audio streaming for the first time.**
 
 ---
 
@@ -71,13 +99,15 @@ npm install
 
 Now edit files in `ui/src/` and see changes live in your browser!
 
-### Modify the backend
+
+### Modify the backend (including audio)
 
 ```bash
-# Edit Go files (config.go, web.go, etc.)
+# Edit Go files (config.go, web.go, internal/audio, etc.)
 ./dev_deploy.sh -r 192.168.1.100 --skip-ui-build
 ```
 
+
 ### Run tests
 
 ```bash
@@ -93,21 +123,26 @@ tail -f /var/log/jetkvm.log
 
 ---
 
+
 ## Project Layout
 
 ```
 /kvm/
 ├── main.go              # App entry point
-├── config.go           # Settings & configuration
-├── web.go              # API endpoints
-├── ui/                 # React frontend
-│   ├── src/routes/     # Pages (login, settings, etc.)
-│   └── src/components/ # UI components
-└── internal/           # Internal Go packages
+├── config.go            # Settings & configuration
+├── web.go               # API endpoints
+├── ui/                  # React frontend
+│   ├── src/routes/      # Pages (login, settings, etc.)
+│   └── src/components/  # UI components
+├── internal/            # Internal Go packages
+│   └── audio/           # In-process audio pipeline (CGO, ALSA, Opus) [NEW]
+├── tools/               # Toolchain and audio dependency setup scripts
+└── Makefile             # Build and dev automation (see audio targets)
 ```
 
 **Key files for beginners:**
 
+- `internal/audio/` - [NEW] In-process audio pipeline (CGO, ALSA, Opus)
 - `web.go` - Add new API endpoints here
 - `config.go` - Add new settings here
 - `ui/src/routes/` - Add new pages here
@@ -136,9 +171,10 @@ npm install
 ./dev_device.sh <YOUR_DEVICE_IP>
 ```
 
+
 ### Quick Backend Changes
 
-*Best for: API or backend logic changes*
+*Best for: API, backend, or audio logic changes (including audio pipeline)*
 
 ```bash
 # Skip frontend build for faster deployment
@@ -206,7 +242,8 @@ curl -X POST http://<IP>/auth/password-local \
 
 ---
 
-## Common Issues & Solutions
+
+### Common Issues & Solutions
 
 ### "Build failed" or "Permission denied"
 
@@ -218,6 +255,8 @@ ssh root@<IP> chmod +x /userdata/jetkvm/bin/jetkvm_app_debug
 go clean -modcache
 go mod tidy
 make build_dev
+# If you see errors about missing ALSA/Opus or toolchain, run:
+make dev_env  # Required for new audio support
 ```
 
 ### "Can't connect to device"
@@ -230,6 +269,15 @@ ping <IP>
 ssh root@<IP> echo "Connection OK"
 ```
 
+
+### "Audio not working"
+
+```bash
+# Make sure you have run:
+make dev_env
+# If you see errors about ALSA/Opus, check logs and re-run the setup scripts in tools/.
+```
+
 ### "Frontend not updating"
 
 ```bash
@@ -244,18 +292,21 @@ npm install
 
 ## Next Steps
 
+
 ### Adding a New Feature
 
-1. **Backend:** Add API endpoint in `web.go`
+1. **Backend:** Add API endpoint in `web.go` or extend audio in `internal/audio/`
 2. **Config:** Add settings in `config.go`
 3. **Frontend:** Add UI in `ui/src/routes/`
 4. **Test:** Deploy and test with `./dev_deploy.sh`
 
+
 ### Code Style
 
 - **Go:** Follow standard Go conventions
 - **TypeScript:** Use TypeScript for type safety
 - **React:** Keep components small and reusable
+- **Audio/CGO:** Keep C/Go integration minimal, robust, and well-documented. Use zerolog for all logging.
 
 ### Environment Variables
 

Makefile

@@ -1,3 +1,20 @@
+# --- JetKVM Audio/Toolchain Dev Environment Setup ---
+.PHONY: setup_toolchain build_audio_deps dev_env
+
+# Clone the rv1106-system toolchain to $HOME/.jetkvm/rv1106-system
+setup_toolchain:
+	bash tools/setup_rv1106_toolchain.sh
+
+# Build ALSA and Opus static libs for ARM in $HOME/.jetkvm/audio-libs
+build_audio_deps: setup_toolchain
+	bash tools/build_audio_deps.sh
+
+# Prepare everything needed for local development (toolchain + audio deps)
+dev_env: build_audio_deps
+	@echo "Development environment ready."
+JETKVM_HOME ?= $(HOME)/.jetkvm
+TOOLCHAIN_DIR ?= $(JETKVM_HOME)/rv1106-system
+AUDIO_LIBS_DIR ?= $(JETKVM_HOME)/audio-libs
 BRANCH    ?= $(shell git rev-parse --abbrev-ref HEAD)
 BUILDDATE ?= $(shell date -u +%FT%T%z)
 BUILDTS   ?= $(shell date -u +%s)
@@ -25,9 +42,14 @@ TEST_DIRS := $(shell find . -name "*_test.go" -type f -exec dirname {} \; | sort
 hash_resource:
 	@shasum -a 256 resource/jetkvm_native | cut -d ' ' -f 1 > resource/jetkvm_native.sha256
 
-build_dev: hash_resource
+build_dev: build_audio_deps hash_resource
 	@echo "Building..."
-	$(GO_CMD) build \
+	GOOS=linux GOARCH=arm GOARM=7 \
+	CC=$(TOOLCHAIN_DIR)/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/bin/arm-rockchip830-linux-uclibcgnueabihf-gcc \
+	CGO_ENABLED=1 \
+	CGO_CFLAGS="-I$(AUDIO_LIBS_DIR)/alsa-lib-1.2.14/include -I$(AUDIO_LIBS_DIR)/opus-1.5.2/include -I$(AUDIO_LIBS_DIR)/opus-1.5.2/celt" \
+	CGO_LDFLAGS="-L$(AUDIO_LIBS_DIR)/alsa-lib-1.2.14/src/.libs -lasound -L$(AUDIO_LIBS_DIR)/opus-1.5.2/.libs -lopus -lm -ldl -static" \
+	go build \
 		-ldflags="$(GO_LDFLAGS) -X $(KVM_PKG_NAME).builtAppVersion=$(VERSION_DEV)" \
 		$(GO_RELEASE_BUILD_ARGS) \
 		-o $(BIN_DIR)/jetkvm_app cmd/main.go
@@ -70,9 +92,14 @@ dev_release: frontend build_dev
 	rclone copyto bin/jetkvm_app r2://jetkvm-update/app/$(VERSION_DEV)/jetkvm_app
 	rclone copyto bin/jetkvm_app.sha256 r2://jetkvm-update/app/$(VERSION_DEV)/jetkvm_app.sha256
 
-build_release: frontend hash_resource
+build_release: frontend build_audio_deps hash_resource
 	@echo "Building release..."
-	$(GO_CMD) build \
+	GOOS=linux GOARCH=arm GOARM=7 \
+	CC=$(TOOLCHAIN_DIR)/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/bin/arm-rockchip830-linux-uclibcgnueabihf-gcc \
+	CGO_ENABLED=1 \
+	CGO_CFLAGS="-I$(AUDIO_LIBS_DIR)/alsa-lib-1.2.14/include -I$(AUDIO_LIBS_DIR)/opus-1.5.2/include -I$(AUDIO_LIBS_DIR)/opus-1.5.2/celt" \
+	CGO_LDFLAGS="-L$(AUDIO_LIBS_DIR)/alsa-lib-1.2.14/src/.libs -lasound -L$(AUDIO_LIBS_DIR)/opus-1.5.2/.libs -lopus -lm -ldl -static" \
+	go build \
 		-ldflags="$(GO_LDFLAGS) -X $(KVM_PKG_NAME).builtAppVersion=$(VERSION)" \
 		$(GO_RELEASE_BUILD_ARGS) \
 		-o bin/jetkvm_app cmd/main.go

README.md

@@ -11,13 +11,20 @@
 
 </div>
 
-JetKVM is a high-performance, open-source KVM over IP (Keyboard, Video, Mouse) solution designed for efficient remote management of computers, servers, and workstations. Whether you're dealing with boot failures, installing a new operating system, adjusting BIOS settings, or simply taking control of a machine from afar, JetKVM provides the tools to get it done effectively.
+
+
+JetKVM is a high-performance, open-source KVM over IP (Keyboard, Video, Mouse, **Audio**) solution designed for efficient remote management of computers, servers, and workstations. Whether you're dealing with boot failures, installing a new operating system, adjusting BIOS settings, or simply taking control of a machine from afar, JetKVM provides the tools to get it done effectively.
+
+
+
+
 
 ## Features
 
-- **Ultra-low Latency** - 1080p@60FPS video with 30-60ms latency using H.264 encoding. Smooth mouse and keyboard interaction for responsive remote control.
+- **Ultra-low Latency** - 1080p@60FPS video with 30-60ms latency using H.264 encoding. Smooth mouse, keyboard, and audio for responsive remote control.
+- **First-Class Audio Support** - JetKVM now supports in-process, low-latency audio streaming using ALSA and Opus, fully integrated via CGO. No external audio binaries or IPC required—audio is delivered directly from the device to your browser.
 - **Free & Optional Remote Access** - Remote management via JetKVM Cloud using WebRTC.
-- **Open-source software** - Written in Golang on Linux. Easily customizable through SSH access to the JetKVM device.
+- **Open-source software** - Written in Golang (with CGO for audio) on Linux. Easily customizable through SSH access to the JetKVM device.
 
 ## Contributing
 
@@ -31,20 +38,23 @@ The best place to search for answers is our [Documentation](https://jetkvm.com/d
 
 If you've found an issue and want to report it, please check our [Issues](https://github.com/jetkvm/kvm/issues) page. Make sure the description contains information about the firmware version you're using, your platform, and a clear explanation of the steps to reproduce the issue.
 
+
+
 # Development
 
-JetKVM is written in Go & TypeScript. with some bits and pieces written in C. An intermediate level of Go & TypeScript knowledge is recommended for comfortable programming.
+JetKVM is written in Go & TypeScript, with some C for low-level integration. **Audio support is now fully in-process using CGO, ALSA, and Opus—no external audio binaries required.**
 
-The project contains two main parts, the backend software that runs on the KVM device and the frontend software that is served by the KVM device, and also the cloud.
+The project contains two main parts: the backend software (Go, CGO) that runs on the KVM device, and the frontend software (React/TypeScript) that is served by the KVM device and the cloud.
 
 For comprehensive development information, including setup, testing, debugging, and contribution guidelines, see **[DEVELOPMENT.md](DEVELOPMENT.md)**.
 
 For quick device development, use the `./dev_deploy.sh` script. It will build the frontend and backend and deploy them to the local KVM device. Run `./dev_deploy.sh --help` for more information.
 
+
 ## Backend
 
-The backend is written in Go and is responsible for the KVM device management, the cloud API and the cloud web.
+The backend is written in Go and is responsible for KVM device management, audio/video streaming, the cloud API, and the cloud web. **Audio is now captured and encoded in-process using ALSA and Opus via CGO, with no external processes or IPC.**
 
 ## Frontend
 
-The frontend is written in React and TypeScript and is served by the KVM device. It has three build targets: `device`, `development` and `production`. Development is used for development of the cloud version on your local machine, device is used for building the frontend for the KVM device and production is used for building the frontend for the cloud.
+The frontend is written in React and TypeScript and is served by the KVM device. It has three build targets: `device`, `development`, and `production`. Development is used for the cloud version on your local machine, device is used for building the frontend for the KVM device, and production is used for building the frontend for the cloud.

config.go

@@ -130,6 +130,7 @@ var defaultConfig = &Config{
 		RelativeMouse: true,
 		Keyboard:      true,
 		MassStorage:   true,
+		Audio:         true,
 	},
 	NetworkConfig:   &network.NetworkConfig{},
 	DefaultLogLevel: "INFO",

internal/audio/api.go

@@ -0,0 +1,11 @@
+package audio
+
+// StartAudioStreaming launches the in-process audio stream and delivers Opus frames to the provided callback.
+func StartAudioStreaming(send func([]byte)) error {
+	return StartCGOAudioStream(send)
+}
+
+// StopAudioStreaming stops the in-process audio stream.
+func StopAudioStreaming() {
+	StopCGOAudioStream()
+}