AreTomo2/3 custom compile for RTX5000s

[iainharley]

When I compile AreTomo2 outwith nextpyp, I needed to add "-gencode arch=compute_120,code=sm_120 " to the makefile11 in order for it to work with my RTX5090s (Blackwell). I cannot get AreTomo working within nextpyp, and I guess that this is the issue. Is it possible to recompile it after installation or edit it before installing nextpyp? I can't seem to find the nextpyp internal aretomo files to do this, or is it possible to point nextpyp aretomo commands to my own installation? Any advice is appreciated :)

[abartesaghi]

In earlier versions of nextPYP, it was possible to configure the path to external programs. However, we've since moved away from that approach, and currently, there's no (easy) way to override the version of AreTomo that comes pre-installed with nextPYP.

That said, we should be able to add the new gencode option, rebuild AreTomo, and update the corresponding container.

What version of CUDA are you using when building AreTomo2 with the -gencode arch=compute_120,code=sm_120 option? Also, which version of nextPYP are you currently using?

[iainharley]

Currently using CUDA 12.8 and nextPYP v0.7.1

[abartesaghi]

@iainharley, we've updated the CUDA version in nextPYP to 12.8 and rebuilt AreTomo2/3 using the flag suggested by @cuchaz (-gencode=arch=compute_xxx,code=compute_xxx). Since there have been numerous updates and bug fixes since version 0.7.1, these changes are included in the upcoming release candidate 0.7.3 (not officially released yet). However, you're welcome to try it out and let us know if AreTomo runs correctly on your Blackwell cards.

You can download the installer using wget https://nextpyp.app/files/pyp/0.7.3/install. Then follow the standard installation/upgrade instructions.

[iainharley]

Great thanks, I will give this a try this coming week and get back to you.

[iainharley]

Installed 0.7.3 and AreTomo is working now, thanks!

[cuchaz]

If it helps at all, you can add PTX codegen to your build to get automatic forward compatibility with future Nvidia architectures. See this part of the Cuda developer guide.

Here's a more specific guide for Blackwell architectures.

In short, add an option like -gencode=arch=compute_xxx,code=compute_xxx to generate PTX (a kind of bytecode, rather than machine instructions). Note how the code=compute_xxx part is different than the code=sm_xxx part. The code=sm_xxx generates machine code for a specific architecture, while the code=compute_xxx generates device-independent PTX bytecode. At runtime, the Nvidia driver will compile the PTX bytecode into machine instructions for the local hardware before running the GPU kernels.

Edit: Looks like you need Cuda toolkit v12.8 or newer to support the latest Blackwell archetectures directly. See Cuda release features here.

And here's a better mapping between GPU architectures, compute capabilities, and Cuda versions.