CUDA: Changing the CUDA scheduling strategy to spin

[JTischbein]

The PR #16308 sets as device property by default integrated = false to disable host buffers. While this change is needed, the additional memory copies introduce multiple cudaStreamSynchronize calls. With the default scheduling strategy, each synchronization has a small latency between kernel termination and freeing the CPU thread on sm121, leading to a ~15% performance regression on gpt-oss-20b-mxfp4. This can be fixed by setting the default scheduling strategy to cudaDeviceScheduleSpin. With the missing latency for each synchronization, the performance is roughly equal as handling the device as integrated.

This code change checks whether the device has compute capability 12.1 and then sets the CUDA flag cudaDeviceScheduleSpin.

[JohannesGaessler]

From cuda_runtime_api.h:

 * - ::cudaDeviceScheduleAuto: The default value if the \p flags parameter is
 * zero, uses a heuristic based on the number of active CUDA contexts in the
 * process \p C and the number of logical processors in the system \p P. If
 * \p C \> \p P, then CUDA will yield to other OS threads when waiting for the
 * device, otherwise CUDA will not yield while waiting for results and
 * actively spin on the processor. Additionally, on Tegra devices,
 * ::cudaDeviceScheduleAuto uses a heuristic based on the power profile of
 * the platform and may choose ::cudaDeviceScheduleBlockingSync for low-powered
 * devices.
 * - ::cudaDeviceScheduleSpin: Instruct CUDA to actively spin when waiting for
 * results from the device. This can decrease latency when waiting for the
 * device, but may lower the performance of CPU threads if they are performing
 * work in parallel with the CUDA thread.
 * - ::cudaDeviceScheduleBlockingSync: Instruct CUDA to block the CPU thread 
 * on a synchronization primitive when waiting for the device to finish work.

Is this change still needed if you set the operating systems power settings to something like "prefer maximum performance"?

[ggerganov]

I can confirm that this patch improves generation performance on NVIDIA DGX Spark with gpt-oss:

Model	Test	t/s master	t/s pr/16585	Speedup
gpt-oss 120B MXFP4 MoE	pp2048	1640.04	1694.05	1.03
gpt-oss 120B MXFP4 MoE	tg32	38.31	45.78	1.19
gpt-oss 20B MXFP4 MoE	pp2048	3610.01	3614.70	1.00
gpt-oss 20B MXFP4 MoE	tg32	54.72	79.95	1.46

[ggerganov]

@JTischbein Need to fix the HIP CI: https://github.com/ggml-org/llama.cpp/actions/runs/18523017783/job/52787191842?pr=16585#step:6:252

[ggerganov]

Suggested change

	// This fix is temporary, as the strategy will be the default in later drivers.
	// This fix is temporary, as the strategy will be the default in later drivers.

[JohannesGaessler]

To my knowledge there are no mobile devices using compute capability 12.1 so it should be fine to set cudaDeviceScheduleSpin unconditionally.

[JohannesGaessler]

Suggested change

	bool is_cc121 = false;
	bool device_schedule_spin = false;

[JohannesGaessler]

Suggested change

	is_cc121 |= prop.major == 12 && prop.minor == 1;
	// Depending on the CUDA drivers the DGX Spark can run with a device schedule that prefers low power use.
	// However, as it is plugged into a wall it should prefer maximum performance.
	// TODO: add a check for a future driver version where this is fixed to avoid thrashing for > 20 CUDA contexts.
	device_schedule_spin = prop.major == 12 && prop.minor == 1;

[JohannesGaessler]

Suggested change

	// Setting device scheduling strategy for iGPUs to "spinning" to avoid delays in cuda synchronize calls.
	// This fix is temporary, as the strategy will be the default in later drivers.
	if (is_cc121) {
	CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin));
	}
	if (device_schedule_spin) {
	CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin));
	}

[JohannesGaessler]

HIP CI

In that case, I would say to just do away with the boolean and to just call cudaSetDeviceFlags where you are currently setting it.

[JTischbein]

I specified the comment a bit more and removed the boolean check. I wanted to avoid multiple calls, but as sm121 is an iGPU only one device should match the condition. Thank you for the feedback.