PETSc.jl and GPUs

[thomasgibson]

In order to use PETSc on GPUs, we'll also need to wrap the necessary data types for CUDA and OpenCL. We'll need to provide the compiled binaries as well; https://www.mcs.anl.gov/petsc/features/gpus.html

[ViralBShah]

@boriskaus I believe we do not build PETSc_jll with GPU support right? Is it something that users expect nowadays? Again, a first step towards getting this eventually is having support in the JLL.

[boriskaus]

No we don't build it with GPU support (I imagine that doing that with BinaryBuilder is a nightmare).

If people are interested in this (which more and more are), they should perhaps build the correct version of PETSc themselves and link that one to PETSc.jl, instead of PETSc_jll. A crucial factor here is that we must ensure that the correct version of PETSc is being linked

[ViralBShah]

We do it increasingly in BinaryBuilder, but it definitely increases complexity. I agree that for now, this is a future feature, and people who need it should use their own PETSc build.

Of course, contributions to do this are welcome (there are several Yggdrasil examples with GPU enabled packages now).

[b-fg]

Hi, I was looking into this package as an alternative option for our GMG in WaterLily.jl. GPU support in PETSc.jl would be very much appreciated, so I was wondering if there are plans in this direction for the short/mid term?

[boriskaus]

PETSc itself works with GPU support and we recently managed to scale one of our codes to a large GPU machine. This was done using our own build of PETSc. I imagine that installing GOPU support though binarybuilder is rather complicated, so I don't think I will touch that anywhere soon.
Also, this has many layers of abstraction (PETSc, for example, creates a copy of every vector on the GPU and CPU), and you certainly loose efficiency with every abstraction layer.

Unless you really need to use the nonlinear solvers of PETSc, I therefore imagine that it will be much simpler to use either ParallelStencil.jl or Chmy.jl basis for your GMG solvers, which both have excellent (parallel) GPU support on different flavors of GPUs. Since this is written in Julia it is closer to the GPU chip, so I imagine that you loose less efficiency (although I haven't measured that)

[b-fg]

Thanks for the details. I was looking for an alternative to our GMG, which already works with GPUs. The only constraints I had in mind were support for MPI, automatic differentiation, and GPU. The packages you mention offer support for finite differences/general stencils, but not as linear algebra solvers, right?

[ViralBShah]

Actually it may not be too hard to add gpu support. Lots of packages do that nowadays. It is worth a shot at least.

[ViralBShah]

The GPU support in BB is now pretty good with lots of libraries built with GPU support. So this should be pretty straightforward now.

[sloisel]

Hi, just to let you know that GPU support (and Strumpack specifically) would be kindly appreciated in this. I tried to set up Github builder CI to use self-made PETSc builds and it's been a nightmare. I think this is beyond the reach of mere mortals.