[NixIO] huge runtime and file size when saving long channel indexes

[rgutzen]

It seems that in particular ChannelIndex objects are not saved very efficiently by the IO.
To illustrate this I create minimal neo structures with an increasing number of channels in a ChannelIndex.

import neo
import quantities as pq
import numpy as np

for N in [10, 100, 1000, 10000, 100000]:
    chidx = neo.ChannelIndex(name='chidx',
                             index=np.arange(N),
                             coordinates=np.zeros((N,2))*pq.mm)
    blk = neo.Block()
    blk.channel_indexes = [chidx]
    with neo.NixIO('temp_N{}.nix'.format(N)) as io:
        io.write(blk)

Here, the time it took to create the files and the corresponding file sizes compared to other formats/IOs (Matlab, python pickle):

NIX
N: 10		 000.16s  677 kB
N: 100		 000.88s    5 MB
N: 1000		 008.44s   48 MB
N: 10000	 084.09s  480 MB
N: 100000	 923.70s  2.4 GB
    
MAT
N: 10		 0.0016s  272 B
N: 100		 0.0014s  272 B
N: 1000		 0.0010s  272 B
N: 10000	 0.0010s  272 B
N: 100000	 0.0029s  272 B

Pickle
N: 10		 0.0010s  1.5 KB
N: 100		 0.0005s  3.7 KB
N: 1000		 0.0006s   25 KB
N: 10000	 0.0011s  240 KB
N: 100000	 0.0042s  2.4 MB

neo: 0.9.0.dev
nixio: v1.5.0b4

I know that it is unusual to have this many channels, but as for example with the new functionalities for imaging data, this issue would become more relevant when interpreting each pixel as a channel, which then easily adds up to 10k.

[achilleas-k]

This has been a long running issue that can't easily be solved (if at all). I've investigated ways to shrink both the write times and the file sizes in the past.

With file sizes, a big part of what we see here is the size overhead of HDF5 Group objects. Each index in a ChannelIndex object gets converted to an individual Source object in NIX, which in turn is an HDF5 Group. Each of these holds the information for a given channel in the ChannelIndex (index, name, ID, coordinates). The mapping was designed this way because we considered that semantically this was the closest interpretation of what a Neo ChannelIndex means in terms of NIX objects. We could store these in the metadata of the top-level Source object that's mapped to the ChannelIndex, but even these new metadata properties would be HDF5 Groups at the end of the day (though there's smaller overhead). A quick test of this solution shows the file size reduces to about half of the original (using your test). I haven't measured write times.

When it came up in the past, we discussed the possibility of using HDF5 Attributes to represent metadata properties. This would remove the Group overhead but wouldn't allow us to store things like units alongside a property value. It's also something that would need to change in the NIX specification so it's a very big decision and it would need major changes in all NIX libraries.

The write time seems to scale with the file size (and object count) so I assume any reduction in the number of objects being created will affect write time as well. I imagine we could shave off a few seconds with some optimisations in the object conversion, but I don't expect any dramatic reduction there.

[apdavison]

Since we don't have ChannelIndex any more, I'm closing this. If the same problem exists with the updated object model, please open a new issue.