Skip to content

Add virtual concatenation of LHDataStores #36

@oschulz

Description

@oschulz

We want to concatenates multiple LH5 files in a virtual/lazy fashion, so that read operations see arrays and tables that are concatenated along all files. Reading, broadcasting, etc. should use Julia processes to work in parallel.

The implementation should open and close files dynamically to both limit the number of active file descriptors and prevent too frequent opening and closing of files.

User-facing, this could simply look like lh5open(vector_of_filenames).

Initially only reading needs to be supported, but longer term we'll probably want to support writing as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions