Investigate GPU allocation performance: device plugin vs. DRA

[jgehrcke]

Time it takes to complete the test suite, roughly:

• 7 minutes with nvidia.com/gpu allocation done by the device plugin
• 12 minutes with nvidia.com/gpu allocation done via DRAExtendedResource

The difference is significant, and generally reproducible.

Example:

To a certain extent, a difference is expected: the DRA flow involves more work, and more components to coordinate.

And maybe everything already flows as fast as it can, with DRA. However, I think it will be very interesting to at some point make a proper distributed profiling exercise, to see precisely where we spend how much time in the information flow. There's probably a way to make some hops more snappy, with better event propagation or some tweaks here and there.

Performance optimization is clearly less important than achieving correctness and doing simplification work. However, to support wide adoption of GPU allocation via DRA we have to understand performance implications when users migrate from the device plugin world to DRA.

[guptaNswati]

Oh this very interesting. thank you for sharing this. I wonder if it’s because scheduler has to generate the claim and store additional metadata and coordinate with the right drivers of a device class. Rest will be the similar process as for dra based claims.

I wonder how it will pan out in case of mig devices.

[johnbelamaric]

@alaypatel07 @gauravkghildiyal have worked on the scalability testing for DRA pretty extensively and probably have some insight.

[gauravkghildiyal]

In our tests conducted for DRANET in comparison with a device-plugin implementation, we observed the same thing as what Alay documented for the OSS scale tests, viz. that DRA required a higher QPS both on the client side of scheduler and kcm and similarly on the kube-apiserver side.

(Recall that kcm also has to create ResourceClaims from templates as one additional things, assuming that one is using ResourceClaimTemplates for tests)

[jgehrcke]

@alaypatel07 @gauravkghildiyal have worked on the scalability testing for DRA pretty extensively and probably have some insight.

Ack. I just want to add that here it's more about the time it takes to prepare a single claim, and not so much about how that latency scales with e.g. pod count or resource claim count. Although these dimensions are not completely independent.