CUDA Version Mismatch and running GPU searches

[xLinkKnight]

What happened?

I'm attempting to use an RTX 5000 ADA card for some crosslink searches using PD 3.1 SP1 with MS Annika 3.0.7.

We had the latest CUDA Toolkit installed (v12.9) and it gave an error which stated 12.2 was needed.

Version 12.2 isn't officially supported for our OS (Windows Server 2025) by Nvidia. We selected the Windows 11 installer as that is the closest in kernel similarity. Now we get an error which states that 12.2 is needed and that we have 12.6 installed.

As you can see in the attached pictures, we do in fact have 12.2 installed, not 12.6.

I'm not sure how the version information is getting fetched or where the issue might originate from.

MS Annika Version

MS Annika 3.0.7

Proteome Discoverer Version

Proteome Discoverer 3.1 SP1

Contact Details

No response

[michabirklbauer]

Hi @xLinkKnight

MS Annika checks the CUDA version by calling nvcc --version and checking if release 12.2 is a sub-string of the output of nvcc --version. You can check what the output of that is on your machine and see if maybe the default nvcc still links to the 12.6 version?

In any case, it's also possible to run MS Annika with newer CUDA versions. You have to enable "Show Advanced Parameters" in Proteome Discoverer and in the MS Annika Search node under "Non-Cleavable Search Settings" there should be an option "Disable CUDA Check" and set that to "Yes". This is basically just a safe-guard rail so people don't accidentally enable this when they don't know what they are doing.

Please also make sure to select "f32DV" or "f32DM" in the parameter "Candidate Search Method".

If would strongly recommend to at least run one reference search on the CPU to compare results to - I have had the experience that the output of the matrix multiplication with CUDA deviates quite a bit between different versions (because people in machine learning don't care about floating point in-accuracies that much - is my guess). Just to make sure nothing fishy is going on.

Please let me know how it goes or if I can help in any other way!

Best, Micha

[xLinkKnight]

Thank you for the tips. The error was from other installations (v12.6, v12.9) not getting properly uninstalled. Cleaning these directories and making a few environment variable changes fixed the problem of v12.2 not getting called.

May I ask what is the typical GPU utilization like for searches? I see hardly any GPU activity (no processing or vRAM loading) based on task manager or the Nvidia usage tool. I've double checked my Nvidia settings to allow for the 5000 card for compute purposes. Since this is the my first time attempting to make use of a GPU, I'm not sure what to expect.

[michabirklbauer]

From my experience GPU utilization in terms of processing power does not go very high, you will probably see the ocassional spike where it goes up for short bursts but then goes back to low - that's mostly because only part of the MS Annika algorithm is run on the GPU (e.g. looking for candidate peptides) while the scoring and finding the correct peptides is run on the CPU. During the scoring and identification of the peptides the GPU is basically idle because it needs to wait for the next pack of mass spectra from the CPU. The VRAM usage mostly depends on the size of your protein database, but will probably stay low too because even larger proteomes can be compressed into sparse matrix format quite well (e.g. a few GB). I have attached a screenshot of a human proteome-wide search that I have run on my GPU (which isn't as powerful, so if you are running on an RTX 5000 card you will probably see even lower numbers). The main benefit of higher-end cards for MS Annika is that communication between CPU and GPU is usually much faster, which I have seen is often the bottleneck.

Long story short, I would not worry about low GPU usage, that is quite normal.
What you can check if you want to make sure that everything works correctly is in the PD log (turn on verbose messages) that it says at the beginning of your MS Annika search that the GPU backend is used, then it should display the GPU algorithm it is using and finally you should see that the candidate search exited with code 0 - which means everything went correctly. If you see a non-zero exit code that means something went wrong (you would also receive a warning at the end of the search by Proteome Discoverer).

[xLinkKnight]

Thank you for the thorough breakdown of how it should function.

I've verified by reading the logs, and all seems to be in working order. It matches nicely with your pictures.

I thought I would be clever and set up two difference searches to maximize our system resources. One CPU bound (DSSO crosslinker) and the second GPU enabled (SDA crosslinker). But as you describe, the bulk of the algorithm is still CPU intensive and so more queued searches introduce a bottleneck.

[michabirklbauer]

Yeah unfortunately I have also never been able to figure out how to maximize system resources fully. But from my experience it is still usually much faster to run multiple searches in parallel rather than sequentially - as long as you have enough RAM of course.

[xLinkKnight]

May I ask if you've noticed considerable search times when working with either timsTOF or Astral data? Our searches with data from a timsTOF take exponentially longer to search, even with the recommended search parameter tweaks. I just want to know if we might have something poorly configured, or is it the nature of working with the sheer number of spectra these new instruments acquire.

May I ask about your thoughts on treating the encoded spectra (empirical and in silico digested peptide sequences) with a locality sensitive hashing function? The objective would be use a relaxed routine like LSH to trim down the search space by identifying candidate crosslinks for further scrutiny via approximate nearest neighbor determination. I just foresee difficulties with how robust the algorithm would be at handling incomplete fragmentation series and spectra with noise. Just trying to figure ways to make a more GPU favored search path.

Please feel free to close this thread or move it to a more appropriate section.

[michabirklbauer]

I don't have much experience with TOF data but for Astral - yes, definitely! Even for smaller protein databases I have seen increases up to 2-3x compared to standard orbitrap data. It's both the number of spectra but also Astral spectra are much more complex (and noisy). And if you use SDA the problem will become even bigger because it's unspecific, so you get many more possibilities.

Yeah I have both considered hashing and approximate nearest neighbor search for candidate selection, there was a pretty good paper on how good that approach is for spectral library search - but there were a few reasons why I ended up deciding against it: 1) hashing makes scoring hard to understand - which is often something really important for end users that don't have a computer science background, 2) afaik approximate nearest neighbor search is not 100% deterministic - you might get slightly different results for every search (bad for reproducibility), and 3) we wanted to make use of our in-house search engine MS Amanda for the scoring, so I was mostly looking for a pre-selection step and not a full scoring [and 4) as so often in research, we wanted to do something new]. From all the tests I did, the candidate selection that is now in MS Annika was never the problem actually, neither in qualitative nor in computational performance. I have also run the algorithm on some non-crosslinked data to test it's performance in recovering the correct peptide candidates, and you reach 95+% coverage pretty easily within the first few 100 candidates - even for modified peptides. But obviously you benefit from de-isotoping/de-convolution/de-noising.

But I'd be happy to give some guidance if you want to change the candidate selection to something that utilizes the GPU more but I unfortunately can't do it myself anymore as the project is over and I have no more time (and funding) to implement new features in MS Annika :(
I don't know whether you are a developer as well but in theory if you have experience you could swap out the code in any of the functions here with hashing and approximate nearest neighbor search and return the top n nearest neighbors. Then compile the DLL and drop it in the Thermo.Magellan.Server directory and it would replace the current GPU backend.