Auto-tuning workgroupsize when localmem consumption depends on it

[tkf]

Does KernelAbstractions.jl support auto-setting workgroupsize when the kernel has local memory size that depends on groupsize? For example, CUDA.launch_configuration takes a shmem callback that maps a number of threads to shared memory used. This is used for implementing mapreduce in CUDA.jl. Since shmem argument for CUDA.launch_configuration is not used in Kernel{CUDADevice}, I guess it's not implemented yet? Is it related to #19?

[vchuravy]

This is #11 KA doesn't support dynamic shared memory.

[tkf]

Does #11 have auto-tuning? I skimmed the code but I couldn't find any. Or it's planned but not implemented?

[vchuravy]

No #11 was started before we added auto-tuning, and stalled since no-one had a clear need for it.

[tkf]

oh, that sounds like I need to give a shot at it if I want it 😂

I still am not clear how to implement auto-tuning with #11, though. If I write @dynamic_localmem T (workgroupsize) -> expression_with(T, worksize), I also need to have a way to compute T from the arguments to the kernel, which can be arbitrarily complex. Since Cassette operates on untyped IR, isn't it impossible to get T given kernel arguments (types)? Doing this at the macro level is even more hopeless. Also, how about @dynamic_localmem behind an inlinable function call?

If these concerns are legit, maybe we still need the explicit shmem callback-like approach?

[tkf]

I'm in particular interested in the use case combined with pre-launch workgroupsize auto-tuning #216.

[bjarthur]

before we added auto-tuning...

is auto tuning documented? if so, i can't find it.

[vchuravy]

When the workgroupsize=nothing the backend is free to pick a size. Most of the GPU backends have a way to ask for the appropriate size of a compiled kernel (nee Auto-Tuning) and the CPU picks 1024