gpu passthrough (#17)

* feat(devices): add lib/devices package types, errors, and paths

Add foundational types for GPU/PCI device passthrough:
- Device, AvailableDevice, CreateDeviceRequest structs
- Error types (ErrNotFound, ErrInUse, ErrAlreadyExists, etc.)
- Device path helpers in lib/paths

* feat(devices): add PCI device discovery and VFIO binding

Add low-level device operations:
- discovery.go: Scan PCI bus, detect IOMMU groups, identify GPU devices
- vfio.go: Bind/unbind devices to vfio-pci driver for VM passthrough

* feat(devices): add device manager core

Add the main device management logic:
- Manager interface with CRUD operations for devices
- CreateDevice, GetDevice, DeleteDevice, ListDevices
- MarkAttached/MarkDetached for instance lifecycle
- BindToVFIO/UnbindFromVFIO for driver management
- Persistence via JSON metadata files

* feat(system): add kernel/initrd NVIDIA GPU support

Add support for NVIDIA GPU passthrough in the VM boot chain:
- versions.go: Add Kernel_20251213 with NVIDIA module/driver lib URLs
- initrd.go: Download and extract NVIDIA kernel modules and driver libs
- init_script.go: Load NVIDIA modules at boot, inject driver libs into containers

This enables containers to use CUDA without bundling driver versions.

* feat(instances): add instance liveness checker for device reconciliation

Add InstanceLivenessChecker adapter to allow the devices package to query
instance state without circular imports. Used during startup to detect
orphaned device attachments from crashed VMs.

- liveness.go: Adapter implementing devices.InstanceLivenessChecker
- liveness_test.go: Unit tests
- reconcile_test.go: Device reconciliation tests
- types.go: Add Devices field to StoredMetadata and CreateInstanceRequest

* feat(instances): integrate devices with instance lifecycle

Wire up device management throughout the instance lifecycle:
- create.go: Validate devices, auto-bind to VFIO, pass to VM config
- delete.go: Detach devices, auto-unbind from VFIO
- configdisk.go: Add HAS_GPU config flag for GPU instances
- manager.go: Add deviceManager dependency
- providers.go: Add ProvideDeviceManager
- wire.go/wire_gen.go: Wire up DeviceManager in DI
- api.go: Add DeviceManager to ApiService struct

* feat(api): add devices API endpoints and documentation

Add REST API for device management and supporting documentation:

API endpoints:
- GET/POST /devices - List and register devices
- GET/DELETE /devices/{id} - Get and delete devices
- GET /devices/available - Discover passthrough-capable devices
- instances.go: Accept devices param in CreateInstance

Documentation:
- GPU.md: GPU passthrough architecture and driver injection
- README.md: Device management usage guide
- scripts/gpu-reset.sh: GPU reset utility

Tests and fixtures:
- gpu_e2e_test.go, gpu_inference_test.go, gpu_module_test.go
- testdata/ollama-cuda/ - CUDA test container

Also adds build-preview-cli Makefile target.

* test: increase VM memory to 2GB to accommodate large initrd

The initrd now includes NVIDIA kernel modules, firmware, and driver
libraries (~238MB total). With 512MB VMs, the kernel couldn't unpack
the initrd into tmpfs without running out of space.

Increase test VM memory from 512MB to 2GB to provide sufficient room
for the initrd contents plus normal VM operation.

* remove slop test

* remove outdated comment

* markattached bug

* remove preview script

* fix(configdisk): only set HAS_GPU=1 for actual GPU devices

The HAS_GPU flag was being set unconditionally when any device was
attached, regardless of device type. This would trigger NVIDIA module
loading in the VM init script even for non-GPU PCI devices.

Now iterates through attached devices and checks each device's type,
only setting HAS_GPU=1 if at least one device is DeviceTypeGPU.

* fix(devices): prevent false positive warnings for instances without GPU devices

detectSuspiciousVMMProcesses was using ListAllInstanceDevices to build the
set of known running instances, but that method only returns instances with
devices attached. This caused legitimate cloud-hypervisor processes for
instances without GPU passthrough to be incorrectly flagged as 'untracked'
with misleading advice to run gpu-reset.sh.

Fix: Call IsInstanceRunning directly for each discovered process instead of
pre-building a map from ListAllInstanceDevices. This correctly identifies
all running instances regardless of device attachment.

* devices: add startup validation warnings for GPU prerequisites

Check and warn on startup if:
- IOMMU is not enabled (no groups in /sys/kernel/iommu_groups)
- VFIO modules not loaded (vfio_pci, vfio_iommu_type1)
- Huge pages not configured (info hint when devices exist)

* instances: move detectSuspiciousVMMProcesses to liveness.go

This function is about instance lifecycle, not device management.
Moving it to the instances module where it belongs.

The implementation uses IsInstanceRunning (which queries all instances)
rather than ListAllInstanceDevices (which only returns instances with
devices) to avoid false positives for non-GPU VMs.

* system: use context loggers in initrd building

Replace fmt.Printf calls with proper context loggers so messages
appear in structured logs with consistent formatting.

---------

Co-authored-by: Rafael Garcia <[email protected]>

Author: rgarcia

Makefile

@@ -1,5 +1,5 @@
 SHELL := /bin/bash
-.PHONY: oapi-generate generate-vmm-client generate-wire generate-all dev build test install-tools gen-jwt download-ch-binaries download-ch-spec ensure-ch-binaries build-caddy-binaries build-caddy ensure-caddy-binaries release-prep clean
+.PHONY: oapi-generate generate-vmm-client generate-wire generate-all dev build test install-tools gen-jwt download-ch-binaries download-ch-spec ensure-ch-binaries build-caddy-binaries build-caddy ensure-caddy-binaries build-preview-cli release-prep clean
 
 # Directory where local binaries will be installed
 BIN_DIR ?= $(CURDIR)/bin

cmd/api/api/api.go

@@ -2,6 +2,7 @@ package api
 
 import (
 	"github.com/onkernel/hypeman/cmd/api/config"
+	"github.com/onkernel/hypeman/lib/devices"
 	"github.com/onkernel/hypeman/lib/images"
 	"github.com/onkernel/hypeman/lib/ingress"
 	"github.com/onkernel/hypeman/lib/instances"
@@ -17,6 +18,7 @@ type ApiService struct {
 	InstanceManager instances.Manager
 	VolumeManager   volumes.Manager
 	NetworkManager  network.Manager
+	DeviceManager   devices.Manager
 	IngressManager  ingress.Manager
 }
 
@@ -29,6 +31,7 @@ func New(
 	instanceManager instances.Manager,
 	volumeManager volumes.Manager,
 	networkManager network.Manager,
+	deviceManager devices.Manager,
 	ingressManager ingress.Manager,
 ) *ApiService {
 	return &ApiService{
@@ -37,6 +40,7 @@ func New(
 		InstanceManager: instanceManager,
 		VolumeManager:   volumeManager,
 		NetworkManager:  networkManager,
+		DeviceManager:   deviceManager,
 		IngressManager:  ingressManager,
 	}
 }

cmd/api/api/api_test.go

@@ -9,6 +9,7 @@ import (
 	"time"
 
 	"github.com/onkernel/hypeman/cmd/api/config"
+	"github.com/onkernel/hypeman/lib/devices"
 	"github.com/onkernel/hypeman/lib/images"
 	"github.com/onkernel/hypeman/lib/instances"
 	mw "github.com/onkernel/hypeman/lib/middleware"
@@ -34,11 +35,12 @@ func newTestService(t *testing.T) *ApiService {
 
 	systemMgr := system.NewManager(p)
 	networkMgr := network.NewManager(p, cfg, nil)
+	deviceMgr := devices.NewManager(p)
 	volumeMgr := volumes.NewManager(p, 0, nil) // 0 = unlimited storage
 	limits := instances.ResourceLimits{
 		MaxOverlaySize: 100 * 1024 * 1024 * 1024, // 100GB
 	}
-	instanceMgr := instances.NewManager(p, imageMgr, systemMgr, networkMgr, volumeMgr, limits, nil, nil)
+	instanceMgr := instances.NewManager(p, imageMgr, systemMgr, networkMgr, deviceMgr, volumeMgr, limits, nil, nil)
 
 	// Register cleanup for orphaned Cloud Hypervisor processes
 	t.Cleanup(func() {
@@ -50,6 +52,7 @@ func newTestService(t *testing.T) *ApiService {
 		ImageManager:    imageMgr,
 		InstanceManager: instanceMgr,
 		VolumeManager:   volumeMgr,
+		DeviceManager:   deviceMgr,
 	}
 }
 

cmd/api/api/devices.go

@@ -0,0 +1,167 @@
+package api
+
+import (
+	"context"
+	"errors"
+
+	"github.com/onkernel/hypeman/lib/devices"
+	"github.com/onkernel/hypeman/lib/oapi"
+)
+
+// ListDevices returns all registered devices
+func (s *ApiService) ListDevices(ctx context.Context, request oapi.ListDevicesRequestObject) (oapi.ListDevicesResponseObject, error) {
+	deviceList, err := s.DeviceManager.ListDevices(ctx)
+	if err != nil {
+		return oapi.ListDevices500JSONResponse{
+			Code:    "internal_error",
+			Message: err.Error(),
+		}, nil
+	}
+
+	result := make([]oapi.Device, len(deviceList))
+	for i, d := range deviceList {
+		result[i] = deviceToOAPI(d)
+	}
+
+	return oapi.ListDevices200JSONResponse(result), nil
+}
+
+// ListAvailableDevices discovers passthrough-capable devices on the host
+func (s *ApiService) ListAvailableDevices(ctx context.Context, request oapi.ListAvailableDevicesRequestObject) (oapi.ListAvailableDevicesResponseObject, error) {
+	available, err := s.DeviceManager.ListAvailableDevices(ctx)
+	if err != nil {
+		return oapi.ListAvailableDevices500JSONResponse{
+			Code:    "internal_error",
+			Message: err.Error(),
+		}, nil
+	}
+
+	result := make([]oapi.AvailableDevice, len(available))
+	for i, d := range available {
+		result[i] = availableDeviceToOAPI(d)
+	}
+
+	return oapi.ListAvailableDevices200JSONResponse(result), nil
+}
+
+// CreateDevice registers a new device for passthrough
+func (s *ApiService) CreateDevice(ctx context.Context, request oapi.CreateDeviceRequestObject) (oapi.CreateDeviceResponseObject, error) {
+	var name string
+	if request.Body.Name != nil {
+		name = *request.Body.Name
+	}
+	req := devices.CreateDeviceRequest{
+		Name:       name,
+		PCIAddress: request.Body.PciAddress,
+	}
+
+	device, err := s.DeviceManager.CreateDevice(ctx, req)
+	if err != nil {
+		switch {
+		case errors.Is(err, devices.ErrInvalidName):
+			return oapi.CreateDevice400JSONResponse{
+				Code:    "invalid_name",
+				Message: err.Error(),
+			}, nil
+		case errors.Is(err, devices.ErrInvalidPCIAddress):
+			return oapi.CreateDevice400JSONResponse{
+				Code:    "invalid_pci_address",
+				Message: err.Error(),
+			}, nil
+		case errors.Is(err, devices.ErrDeviceNotFound):
+			return oapi.CreateDevice404JSONResponse{
+				Code:    "device_not_found",
+				Message: err.Error(),
+			}, nil
+		case errors.Is(err, devices.ErrAlreadyExists), errors.Is(err, devices.ErrNameExists):
+			return oapi.CreateDevice409JSONResponse{
+				Code:    "conflict",
+				Message: err.Error(),
+			}, nil
+		default:
+			return oapi.CreateDevice500JSONResponse{
+				Code:    "internal_error",
+				Message: err.Error(),
+			}, nil
+		}
+	}
+
+	return oapi.CreateDevice201JSONResponse(deviceToOAPI(*device)), nil
+}
+
+// GetDevice returns a device by ID or name
+func (s *ApiService) GetDevice(ctx context.Context, request oapi.GetDeviceRequestObject) (oapi.GetDeviceResponseObject, error) {
+	device, err := s.DeviceManager.GetDevice(ctx, request.Id)
+	if err != nil {
+		if errors.Is(err, devices.ErrNotFound) {
+			return oapi.GetDevice404JSONResponse{
+				Code:    "not_found",
+				Message: "device not found",
+			}, nil
+		}
+		return oapi.GetDevice500JSONResponse{
+			Code:    "internal_error",
+			Message: err.Error(),
+		}, nil
+	}
+
+	return oapi.GetDevice200JSONResponse(deviceToOAPI(*device)), nil
+}
+
+// DeleteDevice unregisters a device
+func (s *ApiService) DeleteDevice(ctx context.Context, request oapi.DeleteDeviceRequestObject) (oapi.DeleteDeviceResponseObject, error) {
+	err := s.DeviceManager.DeleteDevice(ctx, request.Id)
+	if err != nil {
+		switch {
+		case errors.Is(err, devices.ErrNotFound):
+			return oapi.DeleteDevice404JSONResponse{
+				Code:    "not_found",
+				Message: "device not found",
+			}, nil
+		case errors.Is(err, devices.ErrInUse):
+			return oapi.DeleteDevice409JSONResponse{
+				Code:    "in_use",
+				Message: "device is attached to an instance",
+			}, nil
+		default:
+			return oapi.DeleteDevice500JSONResponse{
+				Code:    "internal_error",
+				Message: err.Error(),
+			}, nil
+		}
+	}
+
+	return oapi.DeleteDevice204Response{}, nil
+}
+
+// Helper functions
+
+func deviceToOAPI(d devices.Device) oapi.Device {
+	deviceType := oapi.DeviceType(d.Type)
+	return oapi.Device{
+		Id:          d.Id,
+		Name:        &d.Name,
+		Type:        deviceType,
+		PciAddress:  d.PCIAddress,
+		VendorId:    d.VendorID,
+		DeviceId:    d.DeviceID,
+		IommuGroup:  d.IOMMUGroup,
+		BoundToVfio: d.BoundToVFIO,
+		AttachedTo:  d.AttachedTo,
+		CreatedAt:   d.CreatedAt,
+	}
+}
+
+func availableDeviceToOAPI(d devices.AvailableDevice) oapi.AvailableDevice {
+	return oapi.AvailableDevice{
+		PciAddress:    d.PCIAddress,
+		VendorId:      d.VendorID,
+		DeviceId:      d.DeviceID,
+		VendorName:    &d.VendorName,
+		DeviceName:    &d.DeviceName,
+		IommuGroup:    d.IOMMUGroup,
+		CurrentDriver: d.CurrentDriver,
+	}
+}
+
+

cmd/api/api/instances.go

@@ -96,6 +96,12 @@ func (s *ApiService) CreateInstance(ctx context.Context, request oapi.CreateInst
 		networkEnabled = *request.Body.Network.Enabled
 	}
 
+	// Parse devices (GPU passthrough)
+	var deviceRefs []string
+	if request.Body.Devices != nil {
+		deviceRefs = *request.Body.Devices
+	}
+
 	// Parse volumes
 	var volumes []instances.VolumeAttachment
 	if request.Body.Volumes != nil {
@@ -139,6 +145,7 @@ func (s *ApiService) CreateInstance(ctx context.Context, request oapi.CreateInst
 		Vcpus:          vcpus,
 		Env:            env,
 		NetworkEnabled: networkEnabled,
+		Devices:        deviceRefs,
 		Volumes:        volumes,
 	}
 

cmd/api/main.go

@@ -172,6 +172,18 @@ func run() error {
 	}
 	logger.Info("Network manager initialized")
 
+	// Reconcile device state (clears orphaned attachments from crashed VMs)
+	// Set up liveness checker so device reconciliation can accurately detect orphaned attachments
+	logger.Info("Reconciling device state...")
+	livenessChecker := instances.NewLivenessChecker(app.InstanceManager)
+	if livenessChecker != nil {
+		app.DeviceManager.SetLivenessChecker(livenessChecker)
+	}
+	if err := app.DeviceManager.ReconcileDevices(app.Ctx); err != nil {
+		logger.Error("failed to reconcile device state", "error", err)
+		return fmt.Errorf("reconcile device state: %w", err)
+	}
+
 	// Initialize ingress manager (starts Caddy daemon and DNS server for dynamic upstreams)
 	logger.Info("Initializing ingress manager...")
 	if err := app.IngressManager.Initialize(app.Ctx); err != nil {

cmd/api/wire.go

@@ -9,6 +9,7 @@ import (
 	"github.com/google/wire"
 	"github.com/onkernel/hypeman/cmd/api/api"
 	"github.com/onkernel/hypeman/cmd/api/config"
+	"github.com/onkernel/hypeman/lib/devices"
 	"github.com/onkernel/hypeman/lib/images"
 	"github.com/onkernel/hypeman/lib/ingress"
 	"github.com/onkernel/hypeman/lib/instances"
@@ -27,6 +28,7 @@ type application struct {
 	ImageManager    images.Manager
 	SystemManager   system.Manager
 	NetworkManager  network.Manager
+	DeviceManager   devices.Manager
 	InstanceManager instances.Manager
 	VolumeManager   volumes.Manager
 	IngressManager  ingress.Manager
@@ -44,6 +46,7 @@ func initializeApp() (*application, func(), error) {
 		providers.ProvideImageManager,
 		providers.ProvideSystemManager,
 		providers.ProvideNetworkManager,
+		providers.ProvideDeviceManager,
 		providers.ProvideInstanceManager,
 		providers.ProvideVolumeManager,
 		providers.ProvideIngressManager,