Skip to content

Commit 8d70731

Browse files
authored
docs: add UTM VM setup guide for macOS developers (#42)
* docs: add UTM VM setup guide for macOS developers Add instructions and scripts for running Hypeman in a UTM virtual machine with nested KVM virtualization on Apple Silicon Macs (M3+ required). - Add scripts/utm/README.md with step-by-step UTM VM setup guide - Add scripts/utm/download-iso.sh to download Ubuntu Server ARM64 ISO - Add scripts/utm/setup-vm.sh to provision VM with Hypeman dependencies - Update DEVELOPMENT.md with note directing macOS users to UTM guide - Add .gitignore entry for downloaded ISO files - Minor markdown formatting improvements in DEVELOPMENT.md * refactor: rename setup-vm.sh to bootstrap-dev-environment.sh Clarifies that this script sets up a dev environment for working on Hypeman, not a VM running inside Hypeman. * bind to 0.0.0.0 option for lgtm
1 parent 08ac241 commit 8d70731

File tree

5 files changed

+393
-35
lines changed

5 files changed

+393
-35
lines changed

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,6 @@ lib/system/exec_agent/exec-agent
2424
# Envoy binaries
2525
lib/ingress/binaries/**
2626
dist/**
27+
28+
# UTM VM - downloaded ISO files
29+
scripts/utm/images/

DEVELOPMENT.md

Lines changed: 60 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ This document covers development setup, configuration, and contributing to Hypem
44

55
## Prerequisites
66

7+
> **macOS Users:** Hypeman requires KVM, which is only available on Linux. See [scripts/utm/README.md](scripts/utm/README.md) for instructions on setting up a Linux VM with nested virtualization on Apple Silicon Macs.
8+
79
**Go 1.25.4+**, **KVM**, **erofs-utils**, **dnsmasq**
810

911
```bash
@@ -13,17 +15,19 @@ dnsmasq --version
1315
```
1416

1517
**Install on Debian/Ubuntu:**
18+
1619
```bash
1720
sudo apt-get install erofs-utils dnsmasq
1821
```
1922

2023
**KVM Access:** User must be in `kvm` group for VM access:
24+
2125
```bash
2226
sudo usermod -aG kvm $USER
2327
# Log out and back in, or use: newgrp kvm
2428
```
2529

26-
**Network Capabilities:**
30+
**Network Capabilities:**
2731

2832
Before running or testing Hypeman, ensure IPv4 forwarding is enabled:
2933

@@ -39,6 +43,7 @@ sudo sysctl -p
3943
**Why:** Required for routing traffic between VM network and external network.
4044

4145
The hypeman binary needs network administration capabilities to create bridges and TAP devices:
46+
4247
```bash
4348
# After building, grant network capabilities
4449
sudo setcap 'cap_net_admin,cap_net_bind_service=+eip' /path/to/hypeman
@@ -78,34 +83,34 @@ root hard nofile 65536
7883

7984
Hypeman can be configured using the following environment variables:
8085

81-
| Variable | Description | Default |
82-
|----------|-------------|---------|
83-
| `PORT` | HTTP server port | `8080` |
84-
| `DATA_DIR` | Directory for storing VM images, volumes, and other data | `/var/lib/hypeman` |
85-
| `BRIDGE_NAME` | Name of the network bridge for VM networking | `vmbr0` |
86-
| `SUBNET_CIDR` | CIDR notation for the VM network subnet (gateway derived automatically) | `10.100.0.0/16` |
87-
| `UPLINK_INTERFACE` | Host network interface to use for VM internet access | _(auto-detect)_ |
88-
| `JWT_SECRET` | Secret key for JWT authentication (required for production) | _(empty)_ |
89-
| `DNS_SERVER` | DNS server IP address for VMs | `1.1.1.1` |
90-
| `MAX_CONCURRENT_BUILDS` | Maximum number of concurrent image builds | `1` |
91-
| `MAX_OVERLAY_SIZE` | Maximum size for overlay filesystem | `100GB` |
92-
| `ENV` | Deployment environment (filters telemetry, e.g. your name for dev) | `unset` |
93-
| `OTEL_ENABLED` | Enable OpenTelemetry traces/metrics | `false` |
94-
| `OTEL_ENDPOINT` | OTLP gRPC endpoint | `127.0.0.1:4317` |
95-
| `OTEL_SERVICE_INSTANCE_ID` | Instance ID for telemetry (differentiates multiple servers) | hostname |
96-
| `LOG_LEVEL` | Default log level (debug, info, warn, error) | `info` |
97-
| `LOG_LEVEL_<SUBSYSTEM>` | Per-subsystem log level (API, IMAGES, INSTANCES, NETWORK, VOLUMES, VMM, SYSTEM, EXEC, CADDY) | inherits default |
98-
| `CADDY_LISTEN_ADDRESS` | Address for Caddy ingress listeners | `0.0.0.0` |
99-
| `CADDY_ADMIN_ADDRESS` | Address for Caddy admin API | `127.0.0.1` |
100-
| `CADDY_ADMIN_PORT` | Port for Caddy admin API | `2019` |
101-
| `CADDY_STOP_ON_SHUTDOWN` | Stop Caddy when hypeman shuts down (set to `true` for dev) | `false` |
102-
| `ACME_EMAIL` | Email for ACME certificate registration (required for TLS ingresses) | _(empty)_ |
103-
| `ACME_DNS_PROVIDER` | DNS provider for ACME challenges: `cloudflare` | _(empty)_ |
104-
| `ACME_CA` | ACME CA URL (empty = Let's Encrypt production) | _(empty)_ |
105-
| `TLS_ALLOWED_DOMAINS` | Comma-separated allowed domains for TLS (e.g., `*.example.com,api.other.com`) | _(empty)_ |
106-
| `DNS_PROPAGATION_TIMEOUT` | Max time to wait for DNS propagation (e.g., `2m`) | _(empty)_ |
107-
| `DNS_RESOLVERS` | Comma-separated DNS resolvers for propagation checking | _(empty)_ |
108-
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token (when using `cloudflare` provider) | _(empty)_ |
86+
| Variable | Description | Default |
87+
| -------------------------- | -------------------------------------------------------------------------------------------- | ------------------ |
88+
| `PORT` | HTTP server port | `8080` |
89+
| `DATA_DIR` | Directory for storing VM images, volumes, and other data | `/var/lib/hypeman` |
90+
| `BRIDGE_NAME` | Name of the network bridge for VM networking | `vmbr0` |
91+
| `SUBNET_CIDR` | CIDR notation for the VM network subnet (gateway derived automatically) | `10.100.0.0/16` |
92+
| `UPLINK_INTERFACE` | Host network interface to use for VM internet access | _(auto-detect)_ |
93+
| `JWT_SECRET` | Secret key for JWT authentication (required for production) | _(empty)_ |
94+
| `DNS_SERVER` | DNS server IP address for VMs | `1.1.1.1` |
95+
| `MAX_CONCURRENT_BUILDS` | Maximum number of concurrent image builds | `1` |
96+
| `MAX_OVERLAY_SIZE` | Maximum size for overlay filesystem | `100GB` |
97+
| `ENV` | Deployment environment (filters telemetry, e.g. your name for dev) | `unset` |
98+
| `OTEL_ENABLED` | Enable OpenTelemetry traces/metrics | `false` |
99+
| `OTEL_ENDPOINT` | OTLP gRPC endpoint | `127.0.0.1:4317` |
100+
| `OTEL_SERVICE_INSTANCE_ID` | Instance ID for telemetry (differentiates multiple servers) | hostname |
101+
| `LOG_LEVEL` | Default log level (debug, info, warn, error) | `info` |
102+
| `LOG_LEVEL_<SUBSYSTEM>` | Per-subsystem log level (API, IMAGES, INSTANCES, NETWORK, VOLUMES, VMM, SYSTEM, EXEC, CADDY) | inherits default |
103+
| `CADDY_LISTEN_ADDRESS` | Address for Caddy ingress listeners | `0.0.0.0` |
104+
| `CADDY_ADMIN_ADDRESS` | Address for Caddy admin API | `127.0.0.1` |
105+
| `CADDY_ADMIN_PORT` | Port for Caddy admin API | `2019` |
106+
| `CADDY_STOP_ON_SHUTDOWN` | Stop Caddy when hypeman shuts down (set to `true` for dev) | `false` |
107+
| `ACME_EMAIL` | Email for ACME certificate registration (required for TLS ingresses) | _(empty)_ |
108+
| `ACME_DNS_PROVIDER` | DNS provider for ACME challenges: `cloudflare` | _(empty)_ |
109+
| `ACME_CA` | ACME CA URL (empty = Let's Encrypt production) | _(empty)_ |
110+
| `TLS_ALLOWED_DOMAINS` | Comma-separated allowed domains for TLS (e.g., `*.example.com,api.other.com`) | _(empty)_ |
111+
| `DNS_PROPAGATION_TIMEOUT` | Max time to wait for DNS propagation (e.g., `2m`) | _(empty)_ |
112+
| `DNS_RESOLVERS` | Comma-separated DNS resolvers for propagation checking | _(empty)_ |
113+
| `CLOUDFLARE_API_TOKEN` | Cloudflare API token (when using `cloudflare` provider) | _(empty)_ |
109114

110115
**Important: Subnet Configuration**
111116

@@ -114,10 +119,12 @@ The default subnet `10.100.0.0/16` is chosen to avoid common conflicts. Hypeman
114119
If you need a different subnet, set `SUBNET_CIDR` in your environment. The gateway is automatically derived as the first IP in the subnet (e.g., `10.100.0.0/16``10.100.0.1`).
115120

116121
**Alternative subnets if needed:**
122+
117123
- `172.30.0.0/16` - Private range between common Docker (172.17.x.x) and cloud provider (172.31.x.x) ranges
118124
- `10.200.0.0/16` - Another private range option
119125

120126
**Example:**
127+
121128
```bash
122129
# In your .env file
123130
SUBNET_CIDR=172.30.0.0/16
@@ -128,23 +135,30 @@ SUBNET_CIDR=172.30.0.0/16
128135
`UPLINK_INTERFACE` tells Hypeman which host interface to use for routing VM traffic to the outside world (for iptables MASQUERADE rules). On many hosts this is `eth0`, but laptops and more complex setups often use Wi‑Fi or other names.
129136

130137
**Quick way to discover it:**
138+
131139
```bash
132140
# Ask the kernel which interface is used to reach the internet
133141
ip route get 1.1.1.1
134142
```
143+
135144
Look for the `dev` field in the output, for example:
145+
136146
```text
137147
1.1.1.1 via 192.168.12.1 dev wlp2s0 src 192.168.12.98
138148
```
149+
139150
In this case, `wlp2s0` is the uplink interface, so you would set:
151+
140152
```bash
141153
UPLINK_INTERFACE=wlp2s0
142154
```
143155

144156
You can also inspect all routes:
157+
145158
```bash
146159
ip route show
147160
```
161+
148162
Pick the interface used by the default route (usually the line starting with `default`). Avoid using local bridges like `docker0`, `br-...`, `virbr0`, or `vmbr0` as the uplink; those are typically internal virtual networks, not your actual internet-facing interface.
149163

150164
### TLS Ingress (HTTPS)
@@ -154,6 +168,7 @@ Hypeman uses Caddy with automatic ACME certificates for TLS termination. Certifi
154168
To enable TLS ingresses:
155169

156170
1. Configure ACME credentials in your `.env`:
171+
157172
```bash
158173
# Required for any TLS ingress
159174
@@ -164,6 +179,7 @@ CLOUDFLARE_API_TOKEN=your-api-token
164179
```
165180

166181
2. Create an ingress with TLS enabled:
182+
167183
```bash
168184
curl -X POST http://localhost:8080/v1/ingresses \
169185
-H "Content-Type: application/json" \
@@ -199,6 +215,7 @@ sudo chown $USER:$USER /var/lib/hypeman
199215
### Dockerhub login
200216

201217
Requires Docker Hub authentication to avoid rate limits when running the tests:
218+
202219
```bash
203220
docker login
204221
```
@@ -214,14 +231,17 @@ make build
214231
## Running the Server
215232

216233
1. Generate a JWT token for testing (optional):
234+
217235
```bash
218236
make gen-jwt
219237
```
220238

221239
2. Start the server with hot-reload for development:
240+
222241
```bash
223242
make dev
224243
```
244+
225245
The server will start on port 8080 (configurable via `PORT` environment variable).
226246

227247
### Local OpenTelemetry (optional)
@@ -232,15 +252,19 @@ To collect traces and metrics locally, run the Grafana LGTM stack (Loki, Grafana
232252
# Start Grafana LGTM (UI at http://localhost:3000, login: admin/admin)
233253
# Note, if you are developing on a shared server, you can use the same LGTM stack as your peer(s)
234254
# You will be able to sort your metrics, traces, and logs using the ENV configuration (see below)
255+
BIND=127.0.0.1
256+
# YOLO=1 # Uncomment to expose ports externally
257+
if [ -n "$YOLO" ]; then BIND=0.0.0.0; fi
258+
235259
docker run -d --name lgtm \
236-
-p 127.0.0.1:3000:3000 \
237-
-p 127.0.0.1:4317:4317 \
238-
-p 127.0.0.1:4318:4318 \
239-
-p 127.0.0.1:9090:9090 \
240-
-p 127.0.0.1:4040:4040 \
260+
-p $BIND:3000:3000 \
261+
-p $BIND:4317:4317 \
262+
-p $BIND:4318:4318 \
263+
-p $BIND:9090:9090 \
264+
-p $BIND:4040:4040 \
241265
grafana/otel-lgtm:latest
242266

243-
# If developing on a remote server, forward the port to your local machine:
267+
# If developing on a remote server, forward the port to your local machine (or YOLO):
244268
# ssh -L 3001:localhost:3000 your-server (then open http://localhost:3001)
245269

246270
# Enable OTel in .env (set ENV to your name to filter your telemetry)
@@ -254,6 +278,7 @@ make dev
254278
Open http://localhost:3000 to view traces (Tempo), metrics (Mimir), and logs (Loki) in Grafana.
255279

256280
**Import the Hypeman dashboard:**
281+
257282
1. Go to Dashboards → New → Import
258283
2. Upload `dashboards/hypeman.json` or paste its contents
259284
3. Select the Prometheus datasource and click Import

scripts/utm/README.md

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# UTM VM for Hypeman Development
2+
3+
Run Hypeman in a UTM virtual machine with nested KVM virtualization on Apple Silicon Macs.
4+
5+
> **Note:** In the future, this dev environment setup may be automated using Ansible or similar tooling.
6+
7+
## Requirements
8+
9+
- **macOS 15 (Sequoia)** or later
10+
- **Apple M3 or newer** chip (required for nested virtualization)
11+
- **UTM 4.6+**: `brew install --cask utm`
12+
13+
## Setup
14+
15+
### 1. Download Ubuntu Server ISO
16+
17+
Download Ubuntu Server for ARM from [Ubuntu's website](https://ubuntu.com/download/server/arm).
18+
19+
Or use the helper script:
20+
21+
```bash
22+
./download-iso.sh
23+
```
24+
25+
### 2. Create VM in UTM
26+
27+
Follow the [official UTM Ubuntu guide](https://docs.getutm.app/guides/ubuntu/):
28+
29+
1. Open UTM and click **+** to **Create a New Virtual Machine**
30+
1. Start: select **Virtualize**
31+
1. Operating System: select **Linux**
32+
1. Hardware: Set RAM to **8192 MiB** and CPU cores to **4**
33+
1. Linux: check the box to **Use Apple Virtualization**. **Boot from ISO** should be the default. Click **Browse** and select the Ubuntu Server ISO.
34+
1. Storage: Set disk size to **100 GiB**
35+
1. Shared Directory: skip this. The `bootstrap-dev-environment.sh` script will clone Hypeman onto the VM.
36+
1. Summary: Name the VM **Hypeman** and click **Save**
37+
38+
**Important:** Before starting the VM:
39+
40+
- Right-click **HypemanDev****Edit**
41+
- Go to **System** → Check **Use Hypervisor** (enables nested virtualization)
42+
- Save
43+
44+
### 3. Install Ubuntu Server
45+
46+
Start the VM and follow the Ubuntu installer. You can select the defaults for everything except the following:
47+
48+
1. Profile configuration: set up with your name, a hostname like `hypeman` and a linux username and password you can remember.
49+
1. **Install OpenSSH Server** ← Important to select this and then **Import SSH key****from GitHub** → Enter your GitHub username to import your public keys. The result is that, assuming your host machine has a private key that is uploaded to GitHub, you can re-use this for sshing into the VM.
50+
51+
Once installation is complete, select **Reboot now**.
52+
53+
### 4. Find VM IP Address
54+
55+
After reboot, the VM console will ask you to log in with the Linux username and password you set up.
56+
Once you do that and have a shell, run:
57+
58+
```bash
59+
ip addr
60+
```
61+
62+
Look for an IP like `192.168.x.x` on the `enp0s1` interface.
63+
64+
Open a terminal window on your host machine and export it: `export HYPEMAN_VM_IP=192.168.x.x`
65+
66+
### 5. SSH into the VM
67+
68+
From your Mac terminal:
69+
70+
```bash
71+
ssh -A -i ~/.ssh/id_ed25519 yourusername@${HYPEMAN_VM_IP}
72+
```
73+
74+
### 6. Configure SSH (Recommended)
75+
76+
Add this to your `~/.ssh/config`, replacing `192.168.x.x` with your VM's IP and `yourusername` with your Ubuntu username:
77+
78+
```
79+
Host hypeman
80+
HostName 192.168.x.x
81+
User yourusername
82+
IdentityFile ~/.ssh/id_ed25519
83+
ForwardAgent yes
84+
```
85+
86+
Now you can simply run:
87+
88+
```bash
89+
ssh hypeman
90+
```
91+
92+
## Setting up the VM for Hypeman Development
93+
94+
Copy the `bootstrap-dev-environment.sh` script into the VM and run it:
95+
96+
```bash
97+
scp bootstrap-dev-environment.sh hypeman:
98+
ssh hypeman
99+
./bootstrap-dev-environment.sh
100+
```
101+
102+
This installs Go, erofs-utils, dnsmasq, and other dependencies needed for Hypeman.
103+
104+
It also clones the main Hypeman repositories into ~/code.
105+
106+
## Development Workflow
107+
108+
### Cursor Remote-SSH
109+
110+
Cursor can run on your host machine and connect to the VM to edit files and run commands.
111+
112+
1. Open Command Palette (`Cmd+Shift+P`)
113+
2. Run **Remote-SSH: Connect to Host...**
114+
3. Type **hypeman** (from your SSH config)
115+
4. Open the `~/code/hypeman` directory.
116+
117+
### Run Hypeman
118+
119+
Follow the directions in [../../DEVELOPMENT.md](../../DEVELOPMENT.md).
120+
121+
You might want to set up your host machine's `/etc/hosts` file to resolve `hypeman.local` to your VM's IP:
122+
123+
```
124+
192.168.x.x hypeman.local
125+
```
126+
127+
This will let you, for example, run the LGTM stack in the VM (according to the instructions in [../../DEVELOPMENT.md](../../DEVELOPMENT.md)) and then on your host machine, open http://hypeman.local:3000 to view the Grafana dashboard.
128+
129+
It will also let you test ingresses with subdomain routing.
130+
131+
### Hypeman CLI on host -> VM
132+
133+
The Hypeman CLI can be installed on your host machine and used to interact with the VM.
134+
135+
1. Install the CLI on your host machine (or build from source):
136+
137+
```bash
138+
brew install onkernel/tap/hypeman
139+
```
140+
141+
2. Set up the API key on your host machine:
142+
143+
```bash
144+
# in the vm, generate a token:
145+
make gen-jwt
146+
147+
# on the host machine
148+
export HYPEMAN_API_KEY="<token-from-vm>"
149+
export HYPEMAN_BASE_URL="http://hypemman.local:8080"
150+
151+
# then try out some commands
152+
hypeman pull debian:13-slim
153+
hypeman run --name test-debian debian:13-slim
154+
hypeman exec test-debian -- uname -a
155+
```

0 commit comments

Comments
 (0)