Feature request: support sliced GPU

[andyzhangx]

Is your feature request related to a problem? Please describe.
Thus we could use slice GPU for multi Model Hosting

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[faheem047]

Hey! @andyzhangx New contributor here, interested in working on this feature.

I was thinking we could use GPU time-slicing to enable fractional GPU allocation. Kubernetes supports this through the NVIDIA device plugin - you can request fractional GPUs like 500m (half a GPU) or 250m (quarter GPU). Multiple pods can then share the same physical GPU.

Docs I found:

• NVIDIA Device Plugin Time-Slicing
• Kubernetes Resource Model

From what I can see in the codebase:

• api/v1alpha1/workspace_types.go / api/v1beta1/workspace_types.go - ResourceSpec defines GPU requirements but only handles whole GPUs via InstanceType and Count
• pkg/workspace/inference/preset_inferences.go - It sets GPU resources using nvidia.com/gpu as integer quantities (like resource.NewQuantity(int64(gpuConfig.GPUCount), resource.DecimalSI))
• api/v1beta1/workspace_validation.go - Validation logic checks GPU requirements but assumes whole GPU allocation
• pkg/utils/resources/nodes.go - Checks for nvidia.com/gpu capacity on nodes

To support time-slicing, we'd need to:

• Add a fractional GPU field to ResourceSpec (maybe gpuSliceFraction?)
• Update the resource allocation in preset_inferences.go to use milli-GPU format (like 500m for 0.5 GPU instead of integer 1)
• Update validation to handle fractional GPU requirements
• Maybe add node capability detection to check if time-slicing is enabled

Questions:

• Does this approach make sense for KAITO?
• Where should the fractional GPU setting go in the API? What format?
• How do we validate that nodes support time-slicing?
• Should it default to whole GPU if not specified?

I don't have GPU hardware to test with, but I can work on the code changes, unit tests, and docs. Would need help with E2E testing.

Should I start with a proposal doc or a small PR? Thanks!