Our default ABI for amdgpu is undocumented and doesn't exist on Debian. The error diagnostics are dreadful.

[JonChesterfield]

If we compile with clang today and run on Debian today, the error message is HSA_STATUS_ERROR_INVALID_CODE_OBJECT. I've raised ROCm/ROCR-Runtime#321 in the hope that HSA might produce a more useful diagnostic.

The primary context is #118515, "Use COV6 by default". I think that was premature, partly because Debian is still on rocm 6.1 which can't run that, partly because we seem to be waiting on information on what cov6 actually is relative to cov5. It seems better to not default to cov6 until those change.

AMD's documentation on calling conventions is pretty sparse in general. There isn't a "stable ABI" as such, more a continual drift at the IR level and jumpy changes at the binary level. There's some information at https://llvm.org/docs/AMDGPUUsage.html#amdgpu-amdhsa-code-object-metadata-v5 for previous binary ABI versions. The only references to v6 are in the context of a "generic processor" which refers to a "generic code object version number" but doesn't say what that might be. I've been trying to find information on what changed for v6 and found nothing.

For non-amd-people, the "code object version" nomenclature here means "ABI version", particularly as relates to the boundary between x64 and amdgpu, i.e. the runtime code that calls functions on the gpu from a host ("launches kernels"). Code built for version N and for version M will have some things in common and some things different. Number and order of parameters passed to kernel functions, whether an x64 machine is expected to have allocated a heap or not, how other information is encoded in side channels. Probably more things. This is designed carefully then thrown over the wall. This isn't encoded in the triple or the data layout. I don't seek to change any of that here, merely to ask whether we can roll back to a version that runs on most Linux boxes out there and thus unbreak everyone that was broken by 118515, or at least document what v6 is.

Thanks

(edit: if it turns out that all v6 contains is a "generic ISA" approximation thing, all of the users of which would have to be deliberately opting into, we should default to v5 and leave it there until v7 comes along, and so should rocm, and also hsa probably shouldn't be refusing to run v6)

[llvmbot]

@llvm/issue-subscribers-backend-amdgpu

Author: Jon Chesterfield (JonChesterfield)

If we compile with clang today and run on Debian today, the error message is `HSA_STATUS_ERROR_INVALID_CODE_OBJECT`. I've raised https://github.com/ROCm/ROCR-Runtime/issues/321 in the hope that HSA might produce a more useful diagnostic.

The primary context is #118515, "Use COV6 by default". I think that was premature, partly because Debian is still on 6.1, partly because we seem to be waiting on information on what cov6 actually is relative to cov5. It seems better to not default to six until those change.

AMD's documentation on calling conventions is pretty sparse in general. There isn't a "stable ABI" as such, more a continual drift at the IR level and jumpy changes at the binary level. There's some information at https://llvm.org/docs/AMDGPUUsage.html#amdgpu-amdhsa-code-object-metadata-v5 for previous binary ABI versions. The only references to v6 are in the context of a "generic processor" which refers to a "generic code object version number" but doesn't say what that might be. I've been trying to find information on what changed for v6 and found nothing.

For non-amd-people, the "code object version" nomenclature here means "ABI version", particularly as relates to the boundary between x64 and amdgpu, i.e. the runtime code that calls functions on the gpu from a host ("launches kernels"). Code built for version N and for version M will have some things in common and some things different. Number and order of parameters passed to kernel functions, whether an x64 machine is expected to have allocated a heap or not, how other information is encoded in side channels. Probably more things. This is designed carefully then thrown over the wall. This isn't encoded in the triple or the data layout. I don't seek to change any of that here, merely to ask whether we can roll back to a version that runs on most Linux boxes out there and thus unbreak everyone that was broken by 118515, or at least document what v6 is.

Thanks

[JonChesterfield]

@cgmb might have context from the Debian side on whether packaging a newer rocm is a feasible thing to do. I'd like clang to build programs that run on Debian once more, preferably before clang's next release cycle, though my preference is to have clang target v5 rather than try to get debian to run v6

[cgmb]

@JonChesterfield, the progress of ROCm on Debian was somewhat stalled over the past year but I think you will find that it will be catching up quite soon. AMD has made significant new investments to help. I'm hoping to upload an update from LLVM 17 to LLVM 19 or 20 this week. We should then be able to backport to Trixie. The update to HIP Runtime 6.4.3 should not be far behind.

The generic targets that are enabled by COV6 are an enormous improvement over the status quo and should allow me to drop some big and complex patches from the Debian packages. However, there are many libraries that are written using brittle design patterns that only work on GPU architectures that they have explicitly considered. As such, in order for Debian to adopt generic targets, those upstream libraries need to actually be written to work them. That will only happen if the upstream developers use generic targets.

In my opinion, LLVM is doing the right thing by moving to COV6 by default. It's important to move the ecosystem to using generic targets when possible. The only alternative I can think of would be to default to COV5 unless generic targets are used, in which case COV6 would be default. I suppose the LLVM documentation could perhaps be improved, too.

The fact that COV6 does not work on Debian is not LLVM's fault and they should not wait for us. At least, not in this case. The ecosystem improvements enabled by COV6 are significant.