Skip to content

HDF5 Plugin Working Group

Jump to bottom
Dana Robinson edited this page Sep 19, 2024 · 17 revisions

The HDF5 Plugin Working Group meets monthly, normally on the second Thursday of the month at 10 am Central US time (the normal time for the HDF5 Working Group meeting).

This meeting is for the sustainability and governance of all HDF5 plugins - filter/compression, virtual file driver (VFD), and virtual object layer (VOL) connector. Anyone who is interested in this topic or wants to contribute their time and energy is welcome to attend. This meeting is NOT intended for providing technical support. Please use the forum for that.

Zoom link: https://us06web.zoom.us/j/89601195963

The agenda and any cancellations will be posted below and on the forum, usually on Monday. If you have any action items to discuss, please email derobins at hdfgroup dot org to get them added to the agenda. If there are no pressing issues by Monday, the meeting may be cancelled.

14 November 2024

10 October 2024

19 September 2024

Agenda

  • Are signed plugins working for the 1.14.5 release?

Notes

15 August 2024

Agenda

  • Who needs to be a CODEOWNER?
  • Create a to-do list for CI
    • Why is the existing CI failing and what is the timeline for resolution?
    • Do we do CI here or in the HDF5 repo? Both?
    • Which branches? develop? 1.14? 1.14.x releases?
    • Is the CI sufficient or should it be expanded?
  • Look over existing issues
  • Are all the non-supported filters in the filter list community filters?
    • We should create a directory and README for them
  • What are critical issues for the 1.14.5 release in September?
    • Roughly one month of development time to go, so we need to get any issues sorted ASAP

Notes

  • Don't move the supported filters to a common directory (at least not yet), as this will complicate the CMake code
  • We need to investigate the LZ4 filter to see if it adds extra bytes (issue #134)
  • Allen says the CI will be fixed today
  • CI does two things:
    • Check if it can open a dataset in a test file
    • Do a repack and make sure it can open it
  • This is pretty minimal and probably not enough
    • Do chunk extraction
    • Test a wider variety of data
    • Test group names?
  • Which branches?
    • HDF5 runs these tests as a part of its CI, so tests added here will run in that branch
    • develop and latest for-release branch (currently 1.14)
  • Add Elena and Quincey to CODEOWNERS, also HDFG devs as per HDF5
  • Any other outstanding issues?
    • No?

To-Do

Last week

  • Create basic documentation about purpose of repo, etc.
  • Separate existing filters into "supported" and "community" (community - DONE, supported - pending working CI)
  • Make a list of existing filters that should be added to the "community" section
  • Maintain filter ID list via this repo (list is already in docs, should probably be moved to the root)

This week

  • Need to document the filter parameters each filter uses
  • Create CI badges (Windows, MacOS, Linux, develop/1.14, etc.)

18 July 2024

Agenda

  • Governance (where put docs? - wiki?)
  • Who wants to contribute?
    • Allen, me
  • Repo reorganization
  • Supported platforms and compilers
  • CI - Do we have enough?
  • How do we manage compression libraries? (more challenging than managing filter code)
  • Do we need to add any new plugins for the 1.14.5 release?

Overall Goals

  • One-stop shopping for official filters
  • Maintain high quality filters
    • Alignment with h5py, netCDF, BLOSC, Conda, Debian, other installers
      • How do we deal with Matlab, IDL, older versions, etc.?
    • Uniform install locations, or at least best practices
    • How handle versioning?
  • Clearly indicate which filters are "highly supported"
  • Long-term HDF5 file maintenance (storing filter provenance)
  • Improve documentation

Random notes

  • Many versions of filters in the wild, which is a problem (Elena)
  • Need to support more filters (Elena)
  • Two parts to a filter - filter itself and compression library (Allen)
    • This repo is only about filters, not libraries
  • Need guidance for what constitutes a "proper filter"
    • Code quality, examples, etc.
    • Could go over this as a part of assigning a filter ID number
    • Though this shouldn't be mandatory (some filters may be private)
  • Need to move the filter ID list here
    • Turn it into a matrix w/ person responsible, supported compression library version(s), etc.
  • Cross-platform builds are difficult for some filters
    • Is this okay?
    • Need to document supported compilers and platforms
    • Do we support the Autotools?
      • I hope not (Dana)
  • Two levels of filters
    • Special filters that we carefully maintain and care about (h5py filters, etc.)
    • Community filters that are best-effort
  • What about external software like hsds? How do we help them deal with compressed HDF5 data?
  • Which filters are maintained/community?
    • Community: MAFISC, SZ
    • Maintained: Everything else

To-Do

  • Create basic documentation about purpose of repo, etc.
  • Separate existing filters into "highly maintained" and "community"
  • Make a list of existing filters that should be added to the "community" section
  • Maintain filter ID list via this repo
Clone this wiki locally