Switch VecOfArrays for RecursiveArrayTools's VectorOfArray

[benegee]

Recent updates made RecursiveArrayTools's VectorOfArray do the right thing when used with Adapt.jl. Hence we can avoid using our own VecOfArrays.

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md with its PR number.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[vchuravy]

This seems counter what I expect. For GPU support the send and receive buffers likely need to be CuArrays.

[benegee]

Shouldn't they get Adapt.jled as well?

julia> mpicache = Trixi.P4estMPICache(Float32)

julia> typeof(mpicache.mpi_send_buffers)
RecursiveArrayTools.VectorOfArray{Float32, 2, Vector{Vector{Float32}}}

julia> mpicache.mpi_send_buffers =  VectorOfArray(Vector{Vector{Float32}}(undef, 10))
VectorOfArray{Float32,2}:
10-element Vector{Vector{Float32}}:
 #undef
 #undef
 #undef
 #undef
 #undef
 #undef
 #undef
 #undef
 #undef
 #undef

julia> for index in 1:length(mpicache.mpi_send_buffers)
         mpicache.mpi_send_buffers.u[index] = Vector{Float32}(undef, 10)
       end

julia> Adapt.adapt_structure(CuArray, mpicache.mpi_send_buffers)
VectorOfArray{Float32,2}:
10-element Vector{CuArray{Float32, 1, CUDA.DeviceMemory}}:

[vchuravy]

Likely! Are we missing a rule?

[benegee]

I think this is now doing what we expect:

julia> using Trixi, CUDA
julia> trixi_include("../examples/p4est_3d_dgsem/elixir_advection_basic.jl")
julia> semi_adapted = Trixi.trixi_adapt(CuArray, Float32, semi)
julia> typeof(semi.cache.fstar_primary_threaded)
RecursiveArrayTools.VectorOfArray{Float64, 5, Vector{Array{Float64, 4}}}
julia> typeof(semi_adapted.cache.fstar_primary_threaded)
RecursiveArrayTools.VectorOfArray{Float32, 5, Vector{CuArray{Float32, 4, CUDA.DeviceMemory}}}

Should we try to merge @vchuravy ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.70%. Comparing base (b9376e8) to head (d234c52).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2491      +/-   ##
==========================================
+ Coverage   96.68%   96.70%   +0.02%     
==========================================
  Files         512      511       -1     
  Lines       42302    42284      -18     
==========================================
- Hits        40899    40890       -9     
+ Misses       1403     1394       -9     

Flag	Coverage Δ
unittests	96.70% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.