Skip to content

Switch DGMulti to Matrix{<:SVector} solution storage #1952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed

Switch DGMulti to Matrix{<:SVector} solution storage #1952

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
54f55e0
remove Octavian and matmul! (#1950)
jlchan May 23, 2024
d44f41a
bump StartUpDG compat to 1
jlchan May 23, 2024
c7b38bb
Merge branch 'main' into jc/dev_DGMulti
jlchan May 23, 2024
291f487
switch from StructArrays
jlchan May 23, 2024
a6170c6
Merge branch 'main' into jc/dev_DGMulti
jlchan May 24, 2024
99c9864
Merge branch 'main' into jc/dev_DGMulti
jlchan May 30, 2024
9a14f12
Merge branch 'main' into jc/dev_DGMulti
jlchan Jun 4, 2024
5bbf9e4
Merge branch 'main' into jc/dev_DGMulti
jlchan Sep 10, 2024
97c4c5d
Merge branch 'main' into jc/dev_DGMulti
jlchan Sep 15, 2024
c2fa8e9
Merge branch 'main' into jc/dev_DGMulti
jlchan Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ LinearMaps = "7a12625a-238d-50fd-b39a-03d52299707e"
LoopVectorization = "bdcacae8-1622-11e9-2a5c-532679323890"
MPI = "da04e1cc-30fd-572f-bb4f-1f8673147195"
MuladdMacro = "46d2c3a1-f734-5fdb-9937-b9b9aeba4221"
Octavian = "6fd5a793-0b7e-452c-907f-f8bfe9c57db4"
OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
P4est = "7d669430-f675-4ae7-b43e-fab78ec5a902"
Polyester = "f517fe37-dbe3-4b94-8317-1923a5111588"
Expand Down Expand Up @@ -79,7 +78,6 @@ LoopVectorization = "0.12.151"
MPI = "0.20"
Makie = "0.19, 0.20"
MuladdMacro = "0.2.2"
Octavian = "0.3.21"
OffsetArrays = "1.12"
P4est = "0.4.9"
Polyester = "0.7.10"
Expand All @@ -92,7 +90,7 @@ Requires = "1.1"
SciMLBase = "1.90, 2"
SimpleUnPack = "1.1"
SparseArrays = "1"
StartUpDG = "0.17.7"
StartUpDG = "1"
Static = "0.8.7"
StaticArrayInterface = "1.4"
StaticArrays = "1.5"
Expand Down
1 change: 0 additions & 1 deletion src/Trixi.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ using LinearMaps: LinearMap
using LoopVectorization: LoopVectorization, @turbo, indices
using StaticArrayInterface: static_length # used by LoopVectorization
using MuladdMacro: @muladd
using Octavian: Octavian, matmul!
using Polyester: Polyester, @batch # You know, the cheapest threads you can find...
using OffsetArrays: OffsetArray, OffsetVector
using P4est
Expand Down
15 changes: 8 additions & 7 deletions src/solvers/dgmulti/dg.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@
#! format: noindent

# out <- A*x
mul_by!(A) = @inline (out, x) -> matmul!(out, A, x)
mul_by!(A) = @inline (out, x) -> mul!(out, A, x)
mul_by!(A::T) where {T <: SimpleKronecker} = @inline (out, x) -> mul!(out, A, x)
mul_by!(A::AbstractSparseMatrix) = @inline (out, x) -> mul!(out, A, x)
function mul_by!(A::LinearAlgebra.AdjOrTrans{T, S}) where {T, S <: AbstractSparseMatrix}
@inline (out, x) -> mul!(out, A, x)
end

# out <- out + α * A * x
mul_by_accum!(A, α) = @inline (out, x) -> matmul!(out, A, x, α, One())
mul_by_accum!(A, α) = @inline (out, x) -> mul!(out, A, x, α, One())
function mul_by_accum!(A::AbstractSparseMatrix, α)
@inline (out, x) -> mul!(out, A, x, α, One())
end
Expand All @@ -33,6 +33,11 @@ mul_by_accum!(A::UniformScaling) = MulByAccumUniformScaling()
StructArrays.foreachfield(f, args...)
end

# Matrix{<:SVector} fallback
@inline function apply_to_each_field(f::F, args::Vararg{Any, N}) where {F, N}
f(args...)
end

# specialize for UniformScaling types: works for either StructArray{SVector} or Matrix{SVector}
# solution storage formats.
@inline apply_to_each_field(f::MulByUniformScaling, out, x, args...) = copy!(out, x)
Expand Down Expand Up @@ -136,12 +141,8 @@ function digest_boundary_conditions(boundary_conditions::NamedTuple{Keys, ValueT
return boundary_conditions
end

# Allocate nested array type for DGMulti solution storage.
function allocate_nested_array(uEltype, nvars, array_dimensions, dg)
# store components as separate arrays, combine via StructArrays
return StructArray{SVector{nvars, uEltype}}(ntuple(_ -> zeros(uEltype,
array_dimensions...),
nvars))
Comment on lines -139 to -144
Copy link

@chriselrod chriselrod May 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what's the motivation for the change and how does it perform?
The StructArray version should be SIMD-friendlier, while the Matrix{<:SVector} would be better if SIMD isn't possible.

Copy link
Contributor Author

@jlchan jlchan May 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The motivation is due mainly to #1789. This impacts my specific Trixi solver DGMulti and locks Trixi.jl to a specific version of OrdinaryDiffEq. I looked at the array wrappers in RecursiveArrayTools.jl to try to address this, but it takes a lot more energy for me to figure out customizing broadcasting over custom arrays than just using Base.Array. I'm also concerned about changes like SciML/RecursiveArrayTools.jl#343.

My plan is to swap between Matrix{<:SVector{N, T}} and Array{T, 3} using unsafe_wrap, which is currently what Trixi does for other solvers too. My guess is that it won't be nearly as performant once we make these changes, since we have a few pieces of code which are accelerated by Octavian.jl and LoopVec.jl.

However, I'm less worried about efficiency at the moment and more about maintainability, which I think this would help with.

return zeros(SVector{nvars, uEltype}, array_dimensions...)
end

function reset_du!(du, dg::DGMulti, other_args...)
Expand Down
6 changes: 0 additions & 6 deletions src/solvers/dgmulti/flux_differencing.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,12 +294,6 @@ function compute_flux_differencing_SBP_matrices(dg::DGMultiFluxDiffSBP,
return Qrst_skew
end

# For flux differencing SBP-type approximations, store solutions in Matrix{SVector{nvars}}.
# This results in a slight speedup for `calc_volume_integral!`.
function allocate_nested_array(uEltype, nvars, array_dimensions, dg::DGMultiFluxDiffSBP)
return zeros(SVector{nvars, uEltype}, array_dimensions...)
end

function create_cache(mesh::DGMultiMesh, equations, dg::DGMultiFluxDiffSBP, RealT,
uEltype)
rd = dg.basis
Expand Down