Skip to content

docs: add UTM VM setup guide for macOS developers #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rgarcia merged 3 commits into from
Dec 22, 2025
Merged

docs: add UTM VM setup guide for macOS developers #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b54eb7e
docs: add UTM VM setup guide for macOS developers
rgarcia Dec 22, 2025
2fda8e9
refactor: rename setup-vm.sh to bootstrap-dev-environment.sh
rgarcia Dec 22, 2025
d8074b8
bind to 0.0.0.0 option for lgtm
rgarcia Dec 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,6 @@ lib/system/exec_agent/exec-agent
# Envoy binaries
lib/ingress/binaries/**
dist/**

# UTM VM - downloaded ISO files
scripts/utm/images/
95 changes: 60 additions & 35 deletions DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ This document covers development setup, configuration, and contributing to Hypem

## Prerequisites

> **macOS Users:** Hypeman requires KVM, which is only available on Linux. See [scripts/utm/README.md](scripts/utm/README.md) for instructions on setting up a Linux VM with nested virtualization on Apple Silicon Macs.

**Go 1.25.4+**, **KVM**, **erofs-utils**, **dnsmasq**

```bash
Expand All @@ -13,17 +15,19 @@ dnsmasq --version
```

**Install on Debian/Ubuntu:**

```bash
sudo apt-get install erofs-utils dnsmasq
```

**KVM Access:** User must be in `kvm` group for VM access:

```bash
sudo usermod -aG kvm $USER
# Log out and back in, or use: newgrp kvm
```

**Network Capabilities:**
**Network Capabilities:**

Before running or testing Hypeman, ensure IPv4 forwarding is enabled:

Expand All @@ -39,6 +43,7 @@ sudo sysctl -p
**Why:** Required for routing traffic between VM network and external network.

The hypeman binary needs network administration capabilities to create bridges and TAP devices:

```bash
# After building, grant network capabilities
sudo setcap 'cap_net_admin,cap_net_bind_service=+eip' /path/to/hypeman
Expand Down Expand Up @@ -78,34 +83,34 @@ root hard nofile 65536

Hypeman can be configured using the following environment variables:

| Variable | Description | Default |
|----------|-------------|---------|
| `PORT` | HTTP server port | `8080` |
| `DATA_DIR` | Directory for storing VM images, volumes, and other data | `/var/lib/hypeman` |
| `BRIDGE_NAME` | Name of the network bridge for VM networking | `vmbr0` |
| `SUBNET_CIDR` | CIDR notation for the VM network subnet (gateway derived automatically) | `10.100.0.0/16` |
| `UPLINK_INTERFACE` | Host network interface to use for VM internet access | _(auto-detect)_ |
| `JWT_SECRET` | Secret key for JWT authentication (required for production) | _(empty)_ |
| `DNS_SERVER` | DNS server IP address for VMs | `1.1.1.1` |
| `MAX_CONCURRENT_BUILDS` | Maximum number of concurrent image builds | `1` |
| `MAX_OVERLAY_SIZE` | Maximum size for overlay filesystem | `100GB` |
| `ENV` | Deployment environment (filters telemetry, e.g. your name for dev) | `unset` |
| `OTEL_ENABLED` | Enable OpenTelemetry traces/metrics | `false` |
| `OTEL_ENDPOINT` | OTLP gRPC endpoint | `127.0.0.1:4317` |
| `OTEL_SERVICE_INSTANCE_ID` | Instance ID for telemetry (differentiates multiple servers) | hostname |
| `LOG_LEVEL` | Default log level (debug, info, warn, error) | `info` |
| `LOG_LEVEL_<SUBSYSTEM>` | Per-subsystem log level (API, IMAGES, INSTANCES, NETWORK, VOLUMES, VMM, SYSTEM, EXEC, CADDY) | inherits default |
| `CADDY_LISTEN_ADDRESS` | Address for Caddy ingress listeners | `0.0.0.0` |
| `CADDY_ADMIN_ADDRESS` | Address for Caddy admin API | `127.0.0.1` |
| `CADDY_ADMIN_PORT` | Port for Caddy admin API | `2019` |
| `CADDY_STOP_ON_SHUTDOWN` | Stop Caddy when hypeman shuts down (set to `true` for dev) | `false` |
| `ACME_EMAIL` | Email for ACME certificate registration (required for TLS ingresses) | _(empty)_ |
| `ACME_DNS_PROVIDER` | DNS provider for ACME challenges: `cloudflare` | _(empty)_ |
| `ACME_CA` | ACME CA URL (empty = Let's Encrypt production) | _(empty)_ |
| `TLS_ALLOWED_DOMAINS` | Comma-separated allowed domains for TLS (e.g., `*.example.com,api.other.com`) | _(empty)_ |
| `DNS_PROPAGATION_TIMEOUT` | Max time to wait for DNS propagation (e.g., `2m`) | _(empty)_ |
| `DNS_RESOLVERS` | Comma-separated DNS resolvers for propagation checking | _(empty)_ |
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token (when using `cloudflare` provider) | _(empty)_ |
| Variable | Description | Default |
| -------------------------- | -------------------------------------------------------------------------------------------- | ------------------ |
| `PORT` | HTTP server port | `8080` |
| `DATA_DIR` | Directory for storing VM images, volumes, and other data | `/var/lib/hypeman` |
| `BRIDGE_NAME` | Name of the network bridge for VM networking | `vmbr0` |
| `SUBNET_CIDR` | CIDR notation for the VM network subnet (gateway derived automatically) | `10.100.0.0/16` |
| `UPLINK_INTERFACE` | Host network interface to use for VM internet access | _(auto-detect)_ |
| `JWT_SECRET` | Secret key for JWT authentication (required for production) | _(empty)_ |
| `DNS_SERVER` | DNS server IP address for VMs | `1.1.1.1` |
| `MAX_CONCURRENT_BUILDS` | Maximum number of concurrent image builds | `1` |
| `MAX_OVERLAY_SIZE` | Maximum size for overlay filesystem | `100GB` |
| `ENV` | Deployment environment (filters telemetry, e.g. your name for dev) | `unset` |
| `OTEL_ENABLED` | Enable OpenTelemetry traces/metrics | `false` |
| `OTEL_ENDPOINT` | OTLP gRPC endpoint | `127.0.0.1:4317` |
| `OTEL_SERVICE_INSTANCE_ID` | Instance ID for telemetry (differentiates multiple servers) | hostname |
| `LOG_LEVEL` | Default log level (debug, info, warn, error) | `info` |
| `LOG_LEVEL_<SUBSYSTEM>` | Per-subsystem log level (API, IMAGES, INSTANCES, NETWORK, VOLUMES, VMM, SYSTEM, EXEC, CADDY) | inherits default |
| `CADDY_LISTEN_ADDRESS` | Address for Caddy ingress listeners | `0.0.0.0` |
| `CADDY_ADMIN_ADDRESS` | Address for Caddy admin API | `127.0.0.1` |
| `CADDY_ADMIN_PORT` | Port for Caddy admin API | `2019` |
| `CADDY_STOP_ON_SHUTDOWN` | Stop Caddy when hypeman shuts down (set to `true` for dev) | `false` |
| `ACME_EMAIL` | Email for ACME certificate registration (required for TLS ingresses) | _(empty)_ |
| `ACME_DNS_PROVIDER` | DNS provider for ACME challenges: `cloudflare` | _(empty)_ |
| `ACME_CA` | ACME CA URL (empty = Let's Encrypt production) | _(empty)_ |
| `TLS_ALLOWED_DOMAINS` | Comma-separated allowed domains for TLS (e.g., `*.example.com,api.other.com`) | _(empty)_ |
| `DNS_PROPAGATION_TIMEOUT` | Max time to wait for DNS propagation (e.g., `2m`) | _(empty)_ |
| `DNS_RESOLVERS` | Comma-separated DNS resolvers for propagation checking | _(empty)_ |
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token (when using `cloudflare` provider) | _(empty)_ |

**Important: Subnet Configuration**

Expand All @@ -114,10 +119,12 @@ The default subnet `10.100.0.0/16` is chosen to avoid common conflicts. Hypeman
If you need a different subnet, set `SUBNET_CIDR` in your environment. The gateway is automatically derived as the first IP in the subnet (e.g., `10.100.0.0/16` → `10.100.0.1`).

**Alternative subnets if needed:**

- `172.30.0.0/16` - Private range between common Docker (172.17.x.x) and cloud provider (172.31.x.x) ranges
- `10.200.0.0/16` - Another private range option

**Example:**

```bash
# In your .env file
SUBNET_CIDR=172.30.0.0/16
Expand All @@ -128,23 +135,30 @@ SUBNET_CIDR=172.30.0.0/16
`UPLINK_INTERFACE` tells Hypeman which host interface to use for routing VM traffic to the outside world (for iptables MASQUERADE rules). On many hosts this is `eth0`, but laptops and more complex setups often use Wi‑Fi or other names.

**Quick way to discover it:**

```bash
# Ask the kernel which interface is used to reach the internet
ip route get 1.1.1.1
```

Look for the `dev` field in the output, for example:

```text
1.1.1.1 via 192.168.12.1 dev wlp2s0 src 192.168.12.98
```

In this case, `wlp2s0` is the uplink interface, so you would set:

```bash
UPLINK_INTERFACE=wlp2s0
```

You can also inspect all routes:

```bash
ip route show
```

Pick the interface used by the default route (usually the line starting with `default`). Avoid using local bridges like `docker0`, `br-...`, `virbr0`, or `vmbr0` as the uplink; those are typically internal virtual networks, not your actual internet-facing interface.

### TLS Ingress (HTTPS)
Expand All @@ -154,6 +168,7 @@ Hypeman uses Caddy with automatic ACME certificates for TLS termination. Certifi
To enable TLS ingresses:

1. Configure ACME credentials in your `.env`:

```bash
# Required for any TLS ingress
[email protected]
Expand All @@ -164,6 +179,7 @@ CLOUDFLARE_API_TOKEN=your-api-token
```

2. Create an ingress with TLS enabled:

```bash
curl -X POST http://localhost:8080/v1/ingresses \
-H "Content-Type: application/json" \
Expand Down Expand Up @@ -199,6 +215,7 @@ sudo chown $USER:$USER /var/lib/hypeman
### Dockerhub login

Requires Docker Hub authentication to avoid rate limits when running the tests:

```bash
docker login
```
Expand All @@ -214,14 +231,17 @@ make build
## Running the Server

1. Generate a JWT token for testing (optional):

```bash
make gen-jwt
```

2. Start the server with hot-reload for development:

```bash
make dev
```

The server will start on port 8080 (configurable via `PORT` environment variable).

### Local OpenTelemetry (optional)
Expand All @@ -232,15 +252,19 @@ To collect traces and metrics locally, run the Grafana LGTM stack (Loki, Grafana
# Start Grafana LGTM (UI at http://localhost:3000, login: admin/admin)
# Note, if you are developing on a shared server, you can use the same LGTM stack as your peer(s)
# You will be able to sort your metrics, traces, and logs using the ENV configuration (see below)
BIND=127.0.0.1
# YOLO=1 # Uncomment to expose ports externally
if [ -n "$YOLO" ]; then BIND=0.0.0.0; fi

docker run -d --name lgtm \
-p 127.0.0.1:3000:3000 \
-p 127.0.0.1:4317:4317 \
-p 127.0.0.1:4318:4318 \
-p 127.0.0.1:9090:9090 \
-p 127.0.0.1:4040:4040 \
-p $BIND:3000:3000 \
-p $BIND:4317:4317 \
-p $BIND:4318:4318 \
-p $BIND:9090:9090 \
-p $BIND:4040:4040 \
grafana/otel-lgtm:latest

# If developing on a remote server, forward the port to your local machine:
# If developing on a remote server, forward the port to your local machine (or YOLO):
# ssh -L 3001:localhost:3000 your-server (then open http://localhost:3001)

# Enable OTel in .env (set ENV to your name to filter your telemetry)
Expand All @@ -254,6 +278,7 @@ make dev
Open http://localhost:3000 to view traces (Tempo), metrics (Mimir), and logs (Loki) in Grafana.

**Import the Hypeman dashboard:**

1. Go to Dashboards → New → Import
2. Upload `dashboards/hypeman.json` or paste its contents
3. Select the Prometheus datasource and click Import
Expand Down
155 changes: 155 additions & 0 deletions scripts/utm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# UTM VM for Hypeman Development

Run Hypeman in a UTM virtual machine with nested KVM virtualization on Apple Silicon Macs.

> **Note:** In the future, this dev environment setup may be automated using Ansible or similar tooling.

## Requirements

- **macOS 15 (Sequoia)** or later
- **Apple M3 or newer** chip (required for nested virtualization)
- **UTM 4.6+**: `brew install --cask utm`

## Setup

### 1. Download Ubuntu Server ISO

Download Ubuntu Server for ARM from [Ubuntu's website](https://ubuntu.com/download/server/arm).

Or use the helper script:

```bash
./download-iso.sh
```

### 2. Create VM in UTM

Follow the [official UTM Ubuntu guide](https://docs.getutm.app/guides/ubuntu/):

1. Open UTM and click **+** to **Create a New Virtual Machine**
1. Start: select **Virtualize**
1. Operating System: select **Linux**
1. Hardware: Set RAM to **8192 MiB** and CPU cores to **4**
1. Linux: check the box to **Use Apple Virtualization**. **Boot from ISO** should be the default. Click **Browse** and select the Ubuntu Server ISO.
1. Storage: Set disk size to **100 GiB**
1. Shared Directory: skip this. The `bootstrap-dev-environment.sh` script will clone Hypeman onto the VM.
1. Summary: Name the VM **Hypeman** and click **Save**

**Important:** Before starting the VM:

- Right-click **HypemanDev** → **Edit**
- Go to **System** → Check **Use Hypervisor** (enables nested virtualization)
- Save

### 3. Install Ubuntu Server

Start the VM and follow the Ubuntu installer. You can select the defaults for everything except the following:

1. Profile configuration: set up with your name, a hostname like `hypeman` and a linux username and password you can remember.
1. **Install OpenSSH Server** ← Important to select this and then **Import SSH key** → **from GitHub** → Enter your GitHub username to import your public keys. The result is that, assuming your host machine has a private key that is uploaded to GitHub, you can re-use this for sshing into the VM.

Once installation is complete, select **Reboot now**.

### 4. Find VM IP Address

After reboot, the VM console will ask you to log in with the Linux username and password you set up.
Once you do that and have a shell, run:

```bash
ip addr
```

Look for an IP like `192.168.x.x` on the `enp0s1` interface.

Open a terminal window on your host machine and export it: `export HYPEMAN_VM_IP=192.168.x.x`

### 5. SSH into the VM

From your Mac terminal:

```bash
ssh -A -i ~/.ssh/id_ed25519 yourusername@${HYPEMAN_VM_IP}
```

### 6. Configure SSH (Recommended)

Add this to your `~/.ssh/config`, replacing `192.168.x.x` with your VM's IP and `yourusername` with your Ubuntu username:

```
Host hypeman
HostName 192.168.x.x
User yourusername
IdentityFile ~/.ssh/id_ed25519
ForwardAgent yes
```

Now you can simply run:

```bash
ssh hypeman
```

## Setting up the VM for Hypeman Development

Copy the `bootstrap-dev-environment.sh` script into the VM and run it:

```bash
scp bootstrap-dev-environment.sh hypeman:
ssh hypeman
./bootstrap-dev-environment.sh
```

This installs Go, erofs-utils, dnsmasq, and other dependencies needed for Hypeman.

It also clones the main Hypeman repositories into ~/code.

## Development Workflow

### Cursor Remote-SSH

Cursor can run on your host machine and connect to the VM to edit files and run commands.

1. Open Command Palette (`Cmd+Shift+P`)
2. Run **Remote-SSH: Connect to Host...**
3. Type **hypeman** (from your SSH config)
4. Open the `~/code/hypeman` directory.

### Run Hypeman

Follow the directions in [../../DEVELOPMENT.md](../../DEVELOPMENT.md).

You might want to set up your host machine's `/etc/hosts` file to resolve `hypeman.local` to your VM's IP:

```
192.168.x.x hypeman.local
```

This will let you, for example, run the LGTM stack in the VM (according to the instructions in [../../DEVELOPMENT.md](../../DEVELOPMENT.md)) and then on your host machine, open http://hypeman.local:3000 to view the Grafana dashboard.

It will also let you test ingresses with subdomain routing.

### Hypeman CLI on host -> VM

The Hypeman CLI can be installed on your host machine and used to interact with the VM.

1. Install the CLI on your host machine (or build from source):

```bash
brew install onkernel/tap/hypeman
```

2. Set up the API key on your host machine:

```bash
# in the vm, generate a token:
make gen-jwt

# on the host machine
export HYPEMAN_API_KEY="<token-from-vm>"
export HYPEMAN_BASE_URL="http://hypemman.local:8080"

# then try out some commands
hypeman pull debian:13-slim
hypeman run --name test-debian debian:13-slim
hypeman exec test-debian -- uname -a
```
Loading
Loading