DRA (Dynamic Resource Allocation) Demo

This demo showcases Kubernetes Dynamic Resource Allocation (DRA) by implementing a simple driver that:

• Publishes a file as a resource (/etc/dra/file1)
• Writes the pod name to the file during NodePrepareResources
• Removes the pod name during NodeUnprepareResources

Understanding DRA Architecture

DRA follows a publish-subscribe pattern where resource drivers advertise their capabilities and Kubernetes components consume that information to make scheduling and allocation decisions.

Key Patterns

┌─────────────────────────────────────────────────────────────────────────────┐
│                           PUBLISH-SUBSCRIBE FLOW                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌──────────────┐         ResourceSlice            ┌──────────────────┐    │
│   │  DRA Driver  │ ──────────────────────────────▶  │   API Server     │    │
│   │  (Publisher) │   "I have these devices          │     (Broker)     │    │
│   └──────────────┘    on this node"                 └────────┬─────────┘    │
│                                                              │              │
│                              ┌───────────────────────────────┤              │
│                              │                               │              │
│                              ▼                               ▼              │
│                    ┌──────────────────┐           ┌─────────────────┐       │
│                    │   kube-scheduler │           │     kubelet     │       │
│                    │   (Subscriber)   │           │   (Subscriber)  │       │
│                    └──────────────────┘           └─────────────────┘       │
│                              │                               │              │
│                              │ "Find node with               │ "Prepare     │
│                              │  matching devices"            │  device"     │
│                              ▼                               ▼              │
│                    ┌──────────────────┐           ┌─────────────────┐       │
│                    │  ResourceClaim   │           │   DRA Driver    │       │
│                    │  (allocated)     │◀──────────│   (gRPC call)   │       │
│                    └──────────────────┘           └─────────────────┘       │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Components and Their Roles

Component	Role	Pub-Sub Analogy
DRA Driver	Publishes available devices via ResourceSlices	Publisher
API Server	Stores ResourceSlices, ResourceClaims, DeviceClasses	Broker
kube-scheduler	Watches ResourceSlices to find nodes with available devices	Subscriber
kubelet	Calls driver to prepare/unprepare devices for pods	Subscriber
ResourceClaim	User's request for a device (flows through the system)	Request
DeviceClass	Selector/filter for device types	Topic Filter

Note: These are analogies to help understand DRA's architecture, not formal pattern names.

The Publish-Subscribe Flow

1. Publishing (Driver → API Server)

   • DRA driver discovers local devices (GPUs, FPGAs, files, etc.)
   • Creates/updates ResourceSlice objects describing available devices
   • Each ResourceSlice contains: driver name, node name, pool info, device list

2. Subscribing (Scheduler watches ResourceSlices)

   • kube-scheduler watches all ResourceSlices across the cluster
   • Builds an in-memory view of which nodes have which devices
   • When a pod needs a device, scheduler filters nodes that can satisfy the claim

3. Allocation (Scheduler updates ResourceClaim)

   • Scheduler picks a node and writes allocation result to ResourceClaim
   • Specifies which exact device(s) from which pool are allocated

4. Preparation (Kubelet → Driver via gRPC)

   • Kubelet sees pod scheduled to its node with an allocated claim
   • Calls driver's NodePrepareResources gRPC method
   • Driver sets up the device (mounts, permissions, config files, etc.)

5. Cleanup (Kubelet → Driver via gRPC)

   • When pod terminates, kubelet calls NodeUnprepareResources
   • Driver cleans up device state

Why This Pattern?

• Decoupling: Drivers don't need to know about scheduler internals
• Scalability: Scheduler aggregates info from many drivers efficiently
• Extensibility: New device types just need a new driver
• Consistency: Standard API for all resource types (GPUs, FPGAs, network devices, etc.)

Prerequisites

• Docker
• Kind (Kubernetes in Docker)
• kubectl
• Go 1.25+

Quick Start

# Create kind cluster with DRA enabled
make cluster

# Build and load the DRA driver image
make build load

# Deploy the DRA driver and resources
make deploy

# Test with a sample pod
make test

# Check the file content
make check

# Cleanup
make clean

Project Structure

.
├── cmd/
│   └── dra-driver/
│       └── main.go           # DRA driver entrypoint
├── pkg/
│   ├── driver/
│   │   ├── driver.go         # DRA driver implementation (gRPC server)
│   │   └── publisher.go      # ResourceSlice publisher (pub/sub producer)
│   └── plugin/
│       └── registration.go   # Kubelet plugin registration
├── deploy/
│   ├── namespace.yaml        # dra-system namespace
│   ├── driver.yaml           # DRA driver DaemonSet
│   ├── resourceclass.yaml    # DeviceClass definition (topic filter)
│   └── deployment.yaml       # Deployment + ResourceClaimTemplate (consumer)
├── Dockerfile
├── Makefile
├── go.mod
└── README.md

How It Works

Components in This Demo

1. DRA Driver (pkg/driver/driver.go): Implements the DRA v1 gRPC interface

   • NodePrepareResources: Called by kubelet when pod needs the device
   • NodeUnprepareResources: Called by kubelet when pod releases the device
   • Generates CDI (Container Device Interface) specs for device injection

2. ResourceSlice Publisher (pkg/driver/publisher.go): Publishes device inventory

   • Creates ResourceSlice objects advertising available "file" devices
   • Periodically refreshes to maintain presence in the cluster

3. Plugin Registration (pkg/plugin/registration.go): Registers with kubelet

   • Uses kubelet's plugin registration mechanism
   • Tells kubelet where to find the driver's gRPC socket

4. DeviceClass (deploy/resourceclass.yaml): Defines which devices to select

   • Uses CEL expressions to filter devices by driver name

5. ResourceClaimTemplate (deploy/deployment.yaml): Template for per-pod claims

   • References the DeviceClass
   • Each pod replica gets its own ResourceClaim

6. Deployment (deploy/deployment.yaml): Consumes allocated devices

   • References the ResourceClaimTemplate
   • Each replica gets its own claim, enabling multi-node scaling

End-to-End Flow

1. Driver starts → Registers with kubelet → Publishes ResourceSlice
                                            "node-x has device 'file1'"

2. User creates ResourceClaim → "I need a device from DeviceClass 'file-resource'"

3. User creates Pod → References the ResourceClaim

4. kube-scheduler:
   - Watches ResourceSlices (knows node-x has 'file1')
   - Filters nodes that can satisfy the claim
   - Allocates 'file1' on node-x to the claim
   - Binds pod to node-x

5. kubelet on node-x:
   - Sees pod needs prepared claim
   - Calls driver.NodePrepareResources(claim)
   - Driver writes pod name to /etc/dra/file1

6. Pod runs with access to the file

7. Pod terminates:
   - kubelet calls driver.NodeUnprepareResources(claim)
   - Driver removes pod name from the file

Manual Testing

# Watch the DRA driver logs
kubectl logs -n dra-system -l app=dra-file-driver -f

# Check ResourceSlices (see what the driver published)
kubectl get resourceslices

# Check ResourceClaims (see allocation status)
kubectl get resourceclaims

# Check the file on a node (using a debug pod)
kubectl debug node/<node-name> -it --image=busybox -- cat /host/etc/dra/file1

# Or exec into a deployment pod
kubectl exec -it deploy/dra-test-deployment -- cat /etc/dra/file1

Cleanup

# Remove all resources
make undeploy

# Delete the kind cluster
make cluster-delete