reading data with `jldopen` did not work

[Beforerr]

I feel like there may be a bug that the following codes did not work

jldopen(path)["Data/Table Layout"]

while JLD2.load(path, "Data/Table Layout") would work.

The data file is available at https://github.com/JouleCai/geospacelab/blob/48696ca43aa53721d1ca9414caffed3c4d9048f4/geospacelab/datahub/sources/madrigal/isr/eiscat/examples/MAD6400_2021-03-10_beata_ant%40uhfa.hdf5

Stacktrace

ERROR: ArgumentError: There is no UInt type with 0 bytes
Stacktrace:
  [1] uintofsize
    @ ~/.julia/packages/JLD2/WDhXU/src/misc.jl:132 [inlined]
  [2] jlread(io::JLD2.MmapIO, ::Type{JLD2.EnumerationDatatype})
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/datatypes.jl:392
  [3] jlread(io::JLD2.MmapIO, ::Type{JLD2.H5Datatype})
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/datatypes.jl:73
  [4] jlread(io::JLD2.MmapIO, ::Type{JLD2.CompoundDatatype})
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/datatypes.jl:234
  [5] jlread(io::JLD2.MmapIO, ::Type{JLD2.H5Datatype})
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/datatypes.jl:63
  [6] getproperty(tw::JLD2.HmWrap{JLD2.HmDatatype, JLD2.MmapIO}, s::Symbol)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/macros_utils.jl:248
  [7] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/datasets.jl:25
  [8] getindex(g::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, name::String)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/groups.jl:99
  [9] getindex(f::JLD2.JLDFile{JLD2.MmapIO}, name::String)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/JLD2.jl:369

[JonasIsensee]

Hi @Beforerr,
thanks for opening this bug report.
The issue here is that JLD2 cannot read this dataset currently. I'll try to fix that.

This JLD2.load(path, "Data/Table Layout") works because load is the FileIO.load function and your file has extension .hdf5. Therefore, FileIO will use HDF5.jl instead of JLD2....

[JonasIsensee]

#693 fixes it for me locally. Will merge if tests pass.

[Beforerr]

Thanks for the quick fix. No wonder JLD2.load felt like magic😂

[Beforerr]

I ran a comparison, it seems that JLD2 allocates quite a lot compared to HDF5, is it expected? And if we know the type, is there any way to pass this information to improve performance?

julia> @b h5open(path)["Data/Table Layout"][:]
34.660 ms (124 allocs: 14.241 MiB)

julia> @b jldopen(path)["Data/Table Layout"]
43.922 ms (54029 allocs: 131.949 MiB, 10.52% gc time)

[JonasIsensee]

Hm, that is a lot of allocations.

First of all, though:

HDF5.jl is a wrapper around the C library. I will also be allocating and freeing quite a bit of memory in a way that is invisible to julia.

What's more interesting, that it is significantly slower than hdf5. One could check what is going on there.

Regarding your question: no, type information (about your struct) isn't really the problem here.

[JonasIsensee]

I did merge some allocation improvements and locally, the runtime is approximately matched with hdf5:

julia> @time begin f = jldopen(path); a=f["Data/Table Layout"]; close(f); end
  0.016474 seconds (34.18 k allocations: 12.497 MiB, 22.06% gc time)

julia> @time begin f = h5open(path); f["Data/Table Layout"][:]; close(f); end
  0.014361 seconds (130 allocations: 3.563 MiB)

According to profiling, the largest remaining portion of runtime is spent in decompressing the compressed arrays in the file, so this is not something I can improve on the JLD2 side.