[WIP][cuebot][FIX] Fix GPU Resource Reservation Preventing Non-GPU Frame Dispatch#2095
Draft
anton-ubi wants to merge 7 commits intoAcademySoftwareFoundation:masterfrom
Draft
[WIP][cuebot][FIX] Fix GPU Resource Reservation Preventing Non-GPU Frame Dispatch#2095anton-ubi wants to merge 7 commits intoAcademySoftwareFoundation:masterfrom
anton-ubi wants to merge 7 commits intoAcademySoftwareFoundation:masterfrom
Conversation
Collaborator
|
Looking good so far. It looks like you might have forgotten to add the new property to the test/resource's opencue.properties: |
Contributor
Author
|
It's now ready for review. Thanks for the hint. |
DiegoTavares
approved these changes
Jan 5, 2026
Collaborator
|
I think you might have accidentally marked this PR as draft again. |
Contributor
Author
|
It's actually on purpose. I've been testing it again on our end lately and I'm not seeing the expected behavior it was supposed to fix. I'll keep you posted of any advancement. Apologies for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Hosts with GPU capabilities (ALLOW_GPU=true) incorrectly reject frames that have zero GPU memory requirements. This prevents efficient resource utilization where GPU-enabled hosts should accept both GPU and CPU-only workloads.
Root Cause
The getGpuJobs() method in CoreUnitDispatcher calls removeGpu() when no GPU-specific jobs are found. This method reduces all host resources (CPU cores, memory, GPU resources) to "reserve space for future GPU frames." When normal frames are subsequently evaluated, they fail resource checks due to these artificially reduced limits.
The problematic flow:
Solution
Add a configuration property dispatcher.gpu.skip_resource_reservation that disables GPU resource reservation when set to true. Default is false to maintain backward compatibility.
Configuration
The new behavior is controlled by environment variable CUEBOT_DISPATCHER_SKIP_GPU_RESERVATION:
Recap
Backward compatibility is preserved.
This resolves the counter-intuitive behavior where GPU-capable hosts reject valid CPU-only frames.