Add MPS control daemon support to k8s device plugin #789
+121
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add support for NVIDIA Multi-Process Service (MPS) control daemon, including service configuration and device plugin updates. Signed-off-by: Matthew Yeazel <[email protected]>
This was referenced Dec 31, 2025
vigh-m
reviewed
Jan 6, 2026
| ExecStart=/usr/bin/mps-control-daemon --config-file /etc/nvidia-k8s-device-plugin/settings.yaml | ||
| {{else}} | ||
| ExecStart= | ||
| ExecStart=/usr/bin/sleep infinity |
Contributor
There was a problem hiding this comment.
As discussed. Options to remove the sleep are limited since a combination of our tooling and SystemD behaviour is preventing us from:
- leaving the service in an
exitedstate since SystemD will nottry-restartit - letting the service fail till mps is available since the system will be in a degraded state
- having the service without an
[Install]section since we don't have logic to conditionallystartandstopa service - having the entire service file be rendered on enabling mps since that would require a
daemon reloadwhich is not possible by the existing logic.
Without us writing new logic to conditionally enable and disable services, this would be the way forward to enable mps support
bcressey
reviewed
Jan 9, 2026
|
|
||
| [Service] | ||
| Type=simple | ||
| ExecStart=/bin/true |
Contributor
There was a problem hiding this comment.
I'd prefer /usr/bin/false here so that the start fails in a more obvious way.
| [Service] | ||
| {{#if (eq settings.kubelet-device-plugins.nvidia.device-sharing-strategy "mps")}} | ||
| ExecStart= | ||
| ExecStart=/usr/bin/mps-control-daemon --config-file /etc/nvidia-k8s-device-plugin/settings.yaml |
Contributor
There was a problem hiding this comment.
In the code, there's this log message:
klog.Info("No devices are configured for MPS sharing; Waiting indefinitely.")
Which seems like the infinite sleep we want. Can we render the config in a way that triggers this?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
Related to: bottlerocket-os/bottlerocket#4673
Description of changes:
This builds the mps-control-daemon binary from the device plugin that allows MPS support. We have to patch the hardcoded paths for Bottlerocket usage since the device plugin assumes it can write to / which doesn't work with Bottlerocket.
This change also adds a new service to start this binary when settings request it. Otherwise it daemonizes
sleep infinityto let systemdtry-restartupon changing the settings for MPS.The change should be safe to take without the bottlerocket-os/bottlerocket-kernel-kit#347 change or the upcoming settings change but the daemon will not work without the kmod update and the settings being properly set.
Testing done:
Build images with the kernel change, settings changes, and validated that a node will come up with MPS working if set in user data, and the services are restarted and MPS can be enabled at runtime as well.
Setting in userdata for a g6.2xlarge which only has one GPU
Details
eksctlconfig snippet for setting it at the beginning:Results in a node reporting nvidia.com/gpu.shared:
Setting the MPS after boot
Details
Start with a node with no configuration for MPS:
The node shows one GPU:
Then set MPS:
Now check the rest of the system:
And the node shows the empty nvidia.com/gpu offering but now a shared one:
This is a known edge case and is similar to how timeslicing works. In order to avoid old resources, you'd need to start with the user-data approach.
Shifting to
rename-by-default=false(apiclient set settings.kubelet-device-plugins.nvidia.mps.rename-by-default=false) will have the original nvidia.com/gpu resource instead:And finally, setting sharing to
nonedisables MPS:And the resource goes back down to 1.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.