Skip to content

Commit 5bc9b18

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #51481 from sunya-ch/kep-5075-blog
[en] KEP-5075: DRA Consumable Capacity Blog
2 parents 2f54504 + 00d4de7 commit 5bc9b18

File tree

1 file changed

+200
-0
lines changed

1 file changed

+200
-0
lines changed

‎content/en/blog/_posts/XX-dra-consumable-capacity.md‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes v1.34: DRA Consumable Capacity"
4+
draft: true # will be changed to date: YYYY-MM-DD before publication
5+
slug: kubernetes-v1-34-dra-consumable-capacity # optional
6+
author: >
7+
Sunyanan Choochotkaew (IBM),
8+
Lionel Jouin (Ericsson Software Technology)
9+
John Belamaric (Google)
10+
---
11+
12+
Dynamic Resource Allocation (DRA) is a Kubernetes API for managing scarce resources across Pods and containers.
13+
It enables flexible resource requests, going beyond simply allocating *N* number of devices to support more granular usage scenarios.
14+
With DRA, users can request specific types of devices based on their attributes, define custom configurations tailored to their workloads, and even share the same resource among multiple containers or Pods.
15+
16+
In this blog, we focus on the device sharing feature and dive into a new capability introduced in Kubernetes 1.34: _DRA consumable capacity_,
17+
which extends DRA to support finer-grained device sharing.
18+
19+
## Background: device sharing via ResourceClaims
20+
21+
From the beginning, DRA introduced the ability for multiple Pods to share a device by referencing the same ResourceClaim.
22+
This design decouples resource allocation from specific hardware, allowing for more dynamic and reusable provisioning of devices.
23+
24+
In Kubernetes 1.33, the new support for _partitionable devices_ allowed resource drivers to advertise slices of a device that are available, rather than exposing the entire device as an all-or-nothing resource.
25+
This enabled Kubernetes to model shareable hardware more accurately.
26+
27+
But there was still a missing piece: it didn't yet support scenarios
28+
where the device driver manages fine-grained, dynamic portions of a device resource — like network bandwidth — based on user demand,
29+
or to share those resources independently of ResourceClaims, which are restricted by their spec and namespace.
30+
31+
That’s where _consumable capacity_ for DRA comes in.
32+
33+
## Benefits of DRA consumable capacity support
34+
35+
Here's a taste of what you get in a cluster with the `DRAConsumableCapacity`
36+
[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled.
37+
38+
### Device sharing across multiple ResourceClaims or DeviceRequests
39+
40+
Resource drivers can now support sharing the same device — or even a slice of a device — across multiple ResourceClaims or across multiple DeviceRequests.
41+
42+
This means that Pods from different namespaces can simultaneously share the same device,
43+
if permitted and supported by the specific DRA driver.
44+
45+
### Device resource allocation
46+
47+
Kubernetes extends the allocation algorithm in the scheduler to support allocating a portion of a device's resources, as defined in the `capacity` field.
48+
The scheduler ensures that the total allocated capacity across all consumers never exceeds the device’s total capacity, even when shared across multiple ResourceClaims or DeviceRequests.
49+
This is very similar to the way the scheduler allows Pods and containers to share allocatable resources on Nodes;
50+
in this case, it allows them to share allocatable (consumable) resources on Devices.
51+
52+
This feature expands support for scenarios where the device driver is able to manage resources **within** a device and on a per-process basis — for example,
53+
allocating a specific amount of memory (e.g., 8 GiB) from a virtual GPU,
54+
or setting bandwidth limits on virtual network interfaces allocated to specific Pods. This aims to provide safe and efficient resource sharing.
55+
56+
### DistinctAttribute constraint
57+
58+
This feature also introduces a new constraint: `DistinctAttribute`, which is the complement of the existing `MatchAttribute` constraint.
59+
60+
The primary goal of `DistinctAttribute` is to prevent the same underlying device from being allocated multiple times within a single ResourceClaim, which could happen since we are allocating shares (or subsets) of devices.
61+
This constraint ensures that each allocation refers to a distinct resource, even if they belong to the same device class.
62+
63+
It is useful for use cases such as allocating network devices connecting to different subnets to expand coverage or provide redundancy across failure domains.
64+
65+
## How to use consumable capacity?
66+
67+
`DRAConsumableCapacity` is introduced as an alpha feature in Kubernetes 1.34. The [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) `DRAConsumableCapacity` must be enabled in kubelet, kube-apiserver, kube-scheduler and kube-controller-manager.
68+
69+
```bash
70+
--feature-gates=...,DRAConsumableCapacity=true
71+
```
72+
73+
### As a DRA driver developer
74+
75+
As a DRA driver developer writing in Golang, you can make a device within a ResourceSlice allocatable to multiple ResourceClaims (or `devices.requests`) by setting `AllowMultipleAllocations` to `true`.
76+
77+
```go
78+
Device {
79+
...
80+
AllowMultipleAllocations: ptr.To(true),
81+
...
82+
}
83+
```
84+
85+
Additionally, you can define a policy to restrict how each device's `Capacity` should be consumed by each `DeviceRequest` by defining `RequestPolicy` field in the `DeviceCapacity`.
86+
The example below shows how to define a policy that requires a GPU with 40 GiB of memory to allocate at least 5 GiB per request, with each allocation in multiples of 5 GiB.
87+
88+
```go
89+
DeviceCapacity{
90+
Value: resource.MustParse("40Gi"),
91+
RequestPolicy: &CapacityRequestPolicy{
92+
Default: ptr.To(resource.MustParse("5Gi")),
93+
ValidRange: &CapacityRequestPolicyRange {
94+
Min: ptr.To(resource.MustParse("5Gi")),
95+
Step: ptr.To(resource.MustParse("5Gi")),
96+
}
97+
}
98+
}
99+
```
100+
101+
This will be published to the ResourceSlice, as partially shown below:
102+
103+
```yaml
104+
apiVersion: resource.k8s.io/v1
105+
kind: ResourceSlice
106+
...
107+
spec:
108+
devices:
109+
- name: gpu0
110+
allowMultipleAllocations: true
111+
capacity:
112+
memory:
113+
value: 40Gi
114+
requestPolicy:
115+
default: 5Gi
116+
validRange:
117+
min: 5Gi
118+
step: 5Gi
119+
```
120+
121+
An allocated device with a specified portion of consumed capacity will have a `ShareID` field set in the allocation status.
122+
123+
```go
124+
claim.Status.Allocation.Devices.Results[i].ShareID
125+
```
126+
127+
This `ShareID` allows the driver to distinguish between different allocations that refer to the **same device or same statically-partitioned slice** but come from **different `ResourceClaim` requests**.
128+
It acts as a unique identifier for each shared slice, enabling the driver to manage and enforce resource limits independently across multiple consumers.
129+
130+
### As a consumer
131+
132+
As a consumer (or user), the device resource can be requested with a ResourceClaim like this:
133+
134+
```yaml
135+
apiVersion: resource.k8s.io/v1
136+
kind: ResourceClaim
137+
...
138+
spec:
139+
devices:
140+
requests: # for devices
141+
- name: req0
142+
exactly:
143+
- deviceClassName: resource.example.com
144+
capacity:
145+
requests: # for resources which must be provided by those devices
146+
memory: 10Gi
147+
```
148+
149+
This configuration ensures that the requested device can provide at least 10GiB of `memory`.
150+
151+
Notably that **any** `resource.example.com` device that has at least 10GiB of memory can be allocated.
152+
If a device that does not support multiple allocations is chosen, the allocation would consume the entire device.
153+
To filter only devices that support multiple allocations, you can define a selector like this:
154+
155+
```yaml
156+
selectors:
157+
- cel:
158+
expression: |-
159+
device.allowMultipleAllocations == true
160+
```
161+
162+
## Integration with DRA device status
163+
164+
In device sharing, general device information is provided through the resource slice.
165+
However, some details are set dynamically after allocation.
166+
These can be conveyed using the [`.status.devices`](/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resourceclaim-device-status) field of a ResourceClaim.
167+
That field is only published in clusters where the `DRAResourceClaimDeviceStatus`
168+
feature gate is enabled.
169+
170+
If you do have _device status_ support available, a driver can expose additional device-specific information beyond the `ShareID`.
171+
One particularly useful use case is for virtual networks, where a driver can include the assigned IP address(es) in the status.
172+
This is valuable for both network service operations and troubleshooting.
173+
174+
You can find more information by watching our recording at: [KubeCon Japan 2025 - Reimagining Cloud Native Networks: The Critical Role of DRA](https://sched.co/1x71v).
175+
176+
## What can you do next?
177+
* **Check out the [CNI DRA Driver project](https://github.com/kubernetes-sigs/cni-dra-driver)** for an example of DRA integration in Kubernetes networking. Try integrating with network resources like `macvlan`, `ipvlan`, or smart NICs.
178+
* Start enabling the `DRAConsumableCapacity` feature gate and experimenting with virtualized or partitionable devices. Specify your workloads with *consumable capacity* (for example: fractional bandwidth or memory).
179+
* Let us know your feedback:
180+
* ✅ What worked well?
181+
* ⚠️ What didn’t?
182+
183+
If you encountered issues to fix or opportunities to enhance,
184+
please [file a new issue](https://github.com/kubernetes/enhancements/issues)
185+
and reference [KEP-5075](https://github.com/kubernetes/enhancements/issues/5075) there,
186+
or reach out via [Slack (#wg-device-management)](https://kubernetes.slack.com/archives/C0409NGC1TK).
187+
188+
### Conclusion
189+
190+
Consumable capacity support enhances the device sharing capability of DRA by allowing effective device sharing across namespaces, across claims, and tailored to each Pod’s actual needs.
191+
It also empowers drivers to enforce capacity limits, improves scheduling accuracy, and unlocks new use cases like bandwidth-aware networking and multi-tenant device sharing.
192+
193+
Try it out, experiment with consumable resources, and help shape the future of dynamic resource allocation in Kubernetes!
194+
195+
### Further Reading
196+
* [DRA in the Kubernetes documentation](/docs/concepts/scheduling-eviction/dynamic-resource-allocation/)
197+
* [KEP for DRA Partitionable Devices](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4815-dra-partitionable-devices)
198+
* [KEP for DRA Device Status](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4817-resource-claim-device-status)
199+
* [KEP for DRA Consumable Capacity](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5075-dra-consumable-capacity)
200+
* [Kubernetes 1.34 Release Notes](https://www.kubernetes.dev/resources/release/#kubernetes-v134)

0 commit comments

Comments
 (0)