Fallback serialization for any `AbstractArray` subtype that is convertable from `Array`?

[lassepe]

There are some special array types, such as CUDA.CuArray which currently silently fail when writing to disk (e.g., in the case of CuArray, it writes the array meta-data and device pointer to disk which is no longer valid when de-serializing).

One way to fix this is to add extensions to those packages to customize the serialization logic. For example, I opened such a PR here: JuliaGPU/CUDA.jl#2789

However, I am opening this issue to discuss if it may make sense to provide a more general fallback directly in JLD2. Here is what I had in mind:

function JLD2.writeas(t::Type{<:AbstractArray{T, N}}) where {T, N}
   if hasmethod(convert, Tuple{Type{t}, Array{T, N}})
       Array{T,N}
   else
       t
   end
end

The reason for checking for convertability here rather than doing this for all AbstractArray is that some abstract arrays (such as those from ComponentArrays.jl) may have additional meta-data that does not allow them to be constructed from AbstractArray alone.

[JonasIsensee]

Hi @lassepe,

I'm happy to see that you (i.e. someone that is not me) is interested in these kinds of things :)
I'm all for making serialization more widely functional and easy to add for package authors, if they need special treatment.

However, I fear that this fallback makes more assumptions about the nature of AbstractArrays than we can safely make.
Example:

julia> using SparseMatrices
julia> spzeros(10000,10000)
10000×10000 SparseMatrixCSC{Float64, Int64} with 0 stored entries:
⎡⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⎤

This perfectly valid implementation of an AbstractArray (from a stdlib) that supports huge sizes and memory usage that is approximately proportional to the number of non-zero values.
Your suggested writeas returns Matrix{Float64} and JLD2 would proceed to write the full instantiated array, whereas the current behaviour of writing the struct fields is efficient.

In other words: I don't think it is a good idea to make the default assumption for AbstractArray that the memory layout is garbage and we should convert to a regular array first.

I also seem to remember that JLD2 used to throw warnings when encountering Ptr during serialization.
That would at least warn users (instead of a completely silent failure).

[lassepe]

Thanks for your swift reply, @JonasIsensee.

I see your point about introducing potentially awkward side-effects (as outlined with the sparse arrays example above).
Unfortunately, the pointers used by CUDA.jl and friends are not regular pointers but GPU-specific data structures that escape the current checks in JLD2:

https://github.com/JuliaGPU/CUDA.jl/blob/8b6a2a06c8f9d247081d7c92d8f8a82ab68153d5/src/refpointer.jl#L40

Given your example above, I wonder if SparseMatricCSC should be the default fallback (offering some implicit compression as well). However, that conversion may of course also add overhead and I can see how having that the default for all dense matrices may have other unwanted side-effects.

If you conclude that this kind of default is fundamentally problematic, please feel free to close the issue :) In that case, I will open a PR on the CUDA.jl / GPUArrays.jl package instead.