Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 79 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -499,10 +499,86 @@ jobs:
name: status-ubsan
path: badge-status/status-ubsan.json

tsan:
name: tsan / ubuntu-24.04 / clang-20
runs-on: ubuntu-24.04
timeout-minutes: 45

concurrency:
group: tsan-${{ github.ref }}
cancel-in-progress: true

steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 0

- name: Install dependencies
shell: bash
run: |
set -euxo pipefail
sudo apt-get update
sudo apt-get install -y cmake ninja-build wget gnupg lsb-release software-properties-common libhdf5-dev
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 20 all
echo "CC=$(command -v clang-20)" >> "$GITHUB_ENV"
echo "CXX=$(command -v clang++-20)" >> "$GITHUB_ENV"

- name: Configure
shell: bash
run: |
set -euxo pipefail
cmake -S . -B build -G Ninja \
-DCMAKE_C_COMPILER="$CC" \
-DCMAKE_CXX_COMPILER="$CXX" \
-DCMAKE_CXX_STANDARD=17 \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_C_FLAGS="-fsanitize=thread -fno-omit-frame-pointer -O1 -g" \
-DCMAKE_CXX_FLAGS="-fsanitize=thread -fno-omit-frame-pointer -O1 -g" \
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread" \
-DH5CPP_BUILD_TESTS=ON

- name: Build
shell: bash
run: cmake --build build --parallel

- name: Test
shell: bash
env:
# halt_on_error=1: TSAN exits on the first data race so CI fails loud.
# second_deadlock_stack=1: print full stack of the lock that completed
# the cycle (default prints only one stack for the offending pair).
TSAN_OPTIONS: "halt_on_error=1:second_deadlock_stack=1"
run: ctest --test-dir build --output-on-failure

- name: Record Badge Status
if: always()
shell: bash
run: |
set -euxo pipefail
mkdir -p badge-status
cat <<EOF > badge-status/status-tsan.json
{
"os": "ubuntu-24.04",
"compiler": "clang-20",
"label": "TSan",
"status": "${{ job.status }}"
}
EOF

- name: Upload Status Artifact
if: always()
uses: actions/upload-artifact@v7
with:
name: status-tsan
path: badge-status/status-tsan.json

badge:
name: Generate SVG Badges
if: always()
needs: [build, asan, ubsan]
needs: [build, asan, ubsan, tsan]
runs-on: ubuntu-24.04

steps:
Expand Down Expand Up @@ -648,8 +724,8 @@ jobs:
-DCMAKE_CXX_STANDARD=20 \
-DCMAKE_C_COMPILER=gcc-14 \
-DCMAKE_CXX_COMPILER=g++-14 \
-DCMAKE_C_FLAGS="--coverage -O0 -g" \
-DCMAKE_CXX_FLAGS="--coverage -O0 -g" \
-DCMAKE_C_FLAGS="--coverage -fprofile-update=atomic -O0 -g" \
-DCMAKE_CXX_FLAGS="--coverage -fprofile-update=atomic -O0 -g" \
-DH5CPP_BUILD_TESTS=ON \
-DH5CPP_BUILD_EXAMPLES=ON

Expand Down
87 changes: 84 additions & 3 deletions .github/workflows/package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,12 @@ jobs:

- name: Configure
run: |
HDF5_PREFIX="$(brew --prefix hdf5)"
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
-DHDF5_ROOT=$(brew --prefix hdf5) \
-DHDF5_ROOT="$HDF5_PREFIX" \
-DCMAKE_PREFIX_PATH="$HDF5_PREFIX" \
-DH5CPP_BUILD_EXAMPLES=OFF \
-DH5CPP_BUILD_TESTS=OFF \
-DH5CPP_BUILD_BENCH=OFF
Expand All @@ -107,21 +109,100 @@ jobs:
if-no-files-found: error

# ── Windows (NSIS .exe) ───────────────────────────────────────────────────────
# No Chocolatey 'hdf5' package exists; mirror the build-from-source approach
# from ci.yml. zlib is built fresh each run (~30s); HDF5 install prefix is
# cached between runs.
package-windows:
name: windows / x64 / NSIS
runs-on: windows-latest
env:
HDF5_VERSION: 1.12.2
HDF5_CACHE_VERSION: v3

steps:
- uses: actions/checkout@v4

- name: Install HDF5
run: choco install hdf5 -y --no-progress
- name: Build zlib from source
shell: powershell
run: |
$ErrorActionPreference = "Stop"
$zlib_version = "1.3.1"
$zlib_archive = "$env:RUNNER_TEMP\zlib-$zlib_version.tar.gz"
$zlib_source = "$env:RUNNER_TEMP\zlib-$zlib_version"
$zlib_build = "$env:RUNNER_TEMP\zlib-build"
$zlib_prefix = "$env:RUNNER_TEMP\zlib-install"

Invoke-WebRequest `
-Uri "https://github.com/madler/zlib/archive/refs/tags/v$zlib_version.tar.gz" `
-OutFile $zlib_archive
tar -xzf $zlib_archive -C $env:RUNNER_TEMP

cmake -S $zlib_source -B $zlib_build `
-G "Visual Studio 17 2022" -A x64 `
-DCMAKE_INSTALL_PREFIX="$zlib_prefix"
cmake --build $zlib_build --parallel --config Release
cmake --install $zlib_build --config Release

- name: Restore HDF5 cache
id: cache-hdf5
uses: actions/cache/restore@v5
with:
path: ${{ runner.temp }}\hdf5-${{ env.HDF5_VERSION }}-install
key: hdf5-windows-vs2022-${{ env.HDF5_VERSION }}-pkg-${{ env.HDF5_CACHE_VERSION }}

- name: Build HDF5 from source
if: steps.cache-hdf5.outputs.cache-hit != 'true'
shell: powershell
run: |
$ErrorActionPreference = "Stop"
$hdf5_version = "${{ env.HDF5_VERSION }}"
$hdf5_archive = "$env:RUNNER_TEMP\hdf5-$hdf5_version.tar.gz"
$hdf5_source = "$env:RUNNER_TEMP\hdf5-$hdf5_version"
$hdf5_build = "$env:RUNNER_TEMP\hdf5-$hdf5_version-build"
$hdf5_prefix = "$env:RUNNER_TEMP\hdf5-$hdf5_version-install"

Invoke-WebRequest `
-Uri "https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-$hdf5_version/src/hdf5-$hdf5_version.tar.gz" `
-OutFile $hdf5_archive
tar -xzf $hdf5_archive -C $env:RUNNER_TEMP

cmake -S $hdf5_source -B $hdf5_build `
-G "Visual Studio 17 2022" -A x64 `
-DCMAKE_INSTALL_PREFIX="$hdf5_prefix" `
-DHDF_CFG_NAME=Release `
-DBUILD_SHARED_LIBS=ON `
-DBUILD_TESTING=OFF `
-DHDF5_BUILD_TOOLS=OFF `
-DHDF5_BUILD_UTILS=OFF `
-DHDF5_BUILD_EXAMPLES=OFF `
-DHDF5_BUILD_CPP_LIB=OFF `
-DHDF5_BUILD_HL_LIB=OFF `
-DHDF5_BUILD_FORTRAN=OFF `
-DHDF5_ENABLE_Z_LIB_SUPPORT=ON `
-DHDF5_ENABLE_SZIP_SUPPORT=OFF `
-DZLIB_USE_EXTERNAL=OFF `
-DZLIB_INCLUDE_DIR="$env:RUNNER_TEMP/zlib-install/include" `
-DZLIB_LIBRARY="$env:RUNNER_TEMP/zlib-install/lib/zlib.lib"

cmake --build $hdf5_build --parallel --config Release
cmake --install $hdf5_build --config Release

- name: Save HDF5 cache
if: steps.cache-hdf5.outputs.cache-hit != 'true'
uses: actions/cache/save@v5
with:
path: ${{ runner.temp }}\hdf5-${{ env.HDF5_VERSION }}-install
key: ${{ steps.cache-hdf5.outputs.cache-primary-key }}

- name: Configure
shell: pwsh
run: |
$hdf5_prefix = "$env:RUNNER_TEMP/hdf5-${{ env.HDF5_VERSION }}-install"
$zlib_prefix = "$env:RUNNER_TEMP/zlib-install"
cmake -B build `
-DCMAKE_BUILD_TYPE=Release `
-DHDF5_ROOT="$hdf5_prefix" `
-DCMAKE_PREFIX_PATH="$hdf5_prefix;$zlib_prefix" `
-DH5CPP_BUILD_EXAMPLES=OFF `
-DH5CPP_BUILD_TESTS=OFF `
-DH5CPP_BUILD_BENCH=OFF `
Expand Down
44 changes: 41 additions & 3 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,24 @@ if(APPLE)
set(CMAKE_OSX_DEPLOYMENT_TARGET "14.4" CACHE STRING "Minimum macOS deployment version")
endif()

project(libh5cpp-dev VERSION 1.10.4.6 LANGUAGES CXX C)
# Derive package version from the latest git tag before project(), so CPack
# embeds it as PROJECT_VERSION rather than a stale hardcoded number. Falls
# back to a placeholder when building from a tarball without git history.
find_package(Git QUIET)
set(H5CPP_PROJECT_VERSION "1.12.0")
if(GIT_FOUND AND EXISTS "${CMAKE_SOURCE_DIR}/.git")
execute_process(
COMMAND ${GIT_EXECUTABLE} describe --tags --abbrev=0 --match "v[0-9]*"
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
OUTPUT_VARIABLE H5CPP_GIT_TAG
OUTPUT_STRIP_TRAILING_WHITESPACE
ERROR_QUIET)
if(H5CPP_GIT_TAG MATCHES "^v([0-9]+(\\.[0-9]+)*)$")
set(H5CPP_PROJECT_VERSION "${CMAKE_MATCH_1}")
endif()
endif()

project(libh5cpp-dev VERSION ${H5CPP_PROJECT_VERSION} LANGUAGES CXX C)

# ─── Standard Settings ────────────────────────────────────────────────────────────
set(CMAKE_CXX_STANDARD 20)
Expand Down Expand Up @@ -90,9 +107,22 @@ find_package(Threads REQUIRED QUIET)
# ─── HDF5 ─────────────────────────────────────────────────────────────────────────
find_package(HDF5 REQUIRED COMPONENTS C)

if(HDF5_VERSION VERSION_LESS ${H5CPP_BASE_VERSION})
# HDF5 floor is decoupled from h5cpp package version. Previously the check
# compared HDF5_VERSION against PROJECT_VERSION, which coupled package
# versioning to HDF5 minimums by coincidence. After #247 made
# PROJECT_VERSION track the git tag, the comparison stopped being meaningful.
# Pin the floor explicitly.
#
# 1.10.4 matches the prior implicit floor (the stale project(VERSION 1.10.4.6)
# line that #247 removed). The CI matrix runs Ubuntu 22.04 with system
# HDF5 1.10.7 (floor coverage restored by #235); raising the floor above
# 1.10.4 would break that matrix entry. When dropping 1.10.x coverage in
# a future cohort, bump this constant deliberately and remove the 22.04
# matrix entry in the same commit.
set(H5CPP_HDF5_FLOOR "1.10.4")
if(HDF5_VERSION VERSION_LESS ${H5CPP_HDF5_FLOOR})
message(FATAL_ERROR
"-- !!! H5CPP examples require HDF5 v${H5CPP_BASE_VERSION} or greater !!!"
"-- !!! H5CPP requires HDF5 v${H5CPP_HDF5_FLOOR} or greater (found ${HDF5_VERSION}) !!!"
)
else()
message(STATUS
Expand Down Expand Up @@ -382,6 +412,14 @@ set(CPACK_NSIS_ENABLE_UNINSTALL_BEFORE_INSTALL ON)
# productbuild (macOS .pkg)
set(CPACK_PRODUCTBUILD_IDENTIFIER "org.h5cpp.h5cpp")

# Override the default per-generator filename so that amd64 / arm64 / x86_64 /
# aarch64 / darwin-arm64 / windows-amd64 produce distinct names when collected
# into a single release upload directory. Without this, multi-arch CI matrix
# builds emit identical filenames and overwrite each other on the Release page.
set(CPACK_PACKAGE_FILE_NAME
"${CPACK_PACKAGE_NAME}-v${PROJECT_VERSION}-${CMAKE_SYSTEM_NAME}-${CMAKE_SYSTEM_PROCESSOR}")
string(TOLOWER "${CPACK_PACKAGE_FILE_NAME}" CPACK_PACKAGE_FILE_NAME)

include(CPack)

# ─── Developer convenience targets ───────────────────────────────────────────
Expand Down
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.20123216.svg)](https://doi.org/10.5281/zenodo.20123216)
[![GitHub release](https://img.shields.io/github/v/release/vargalabs/h5cpp.svg)](https://github.com/vargalabs/h5cpp/releases)
[![Documentation](https://img.shields.io/badge/docs-stable-blue)](https://vargalabs.github.io/h5cpp)
[![Downloads](https://img.shields.io/github/downloads/vargalabs/h5cpp/total)](https://github.com/vargalabs/h5cpp/releases)

# H5CPP — High-Performance [HDF5][hdf5] for Modern C++

Expand Down Expand Up @@ -92,10 +91,10 @@ cmake --install build
- **`std::float16_t`** (C++23 IEEE 754 half-precision)
- **Rank-7** array support
- **Expanded attribute** type coverage
- **Threaded I/O pipeline** for filter chains
- **FAPL-scoped worker pool** — `h5::create(..., h5::threads{N} | h5::backpressure{M})` opts the file into parallel filter compression; all chunked datasets opened on that file (and pt_t built from them) inherit the pool with async-pipelined dispatch
- **HDF5 1.12.2 ceiling** — tested and verified; `H5Dvlen_reclaim` / reference API compatibility
- **Windows MSVC** in the CI matrix
- **ASan + UBSan** clean on Clang 20
- **ASan + UBSan + TSan** clean on Clang 20

## Documentation

Expand Down
21 changes: 20 additions & 1 deletion docs/filtering-pipeline-rework-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ The current implementation is an experimental skeleton rather than a production
| Multi-filter read | Throws for more than one filter | Reverse-order decode through the complete filter plan |
| Buffer sizing | Uses chunk-sized scratch buffers | Encoded buffers must allow compression expansion |
| Filter mask | Partial handling | Preserve HDF5 chunk filter-mask semantics |
| Threading | `threaded_pipeline_t` is a placeholder | Worker-local state and bounded chunk scheduling |
| Threading | `threaded_pipeline_t` is a placeholder | Worker-local state and bounded chunk scheduling (delivered in #250 as FAPL-scoped `pool_pipeline_t`) |
| Portability | Linux path is the only recently verified path | Linux, macOS, and Windows allocation/build behavior |

Focused baseline probes confirmed two important failures:
Expand Down Expand Up @@ -175,3 +175,22 @@ Threading should initially use C++17 standard library primitives. Avoid platform
Start with correctness, not SIMD. The highest-value first milestone is a serial `filter_plan` that can round-trip standard HDF5 filters and reject unsupported filters explicitly. Once that foundation is correct, SIMD and multithreading become execution-policy improvements rather than a risky rewrite.

The strategic direction is to make H5CPP's filtering chain a modern CPU execution engine while preserving HDF5-compatible metadata and file interoperability.

## Status — Phase I (#250, FAPL worker pool)

Phase I of the threading workplan is delivered on PR #251. The design and trade-offs are summarised in `tasks/h5cpp-fapl-multithreading-workplan.md`; the user-visible surface is one line in the file's FAPL:

```cpp
h5::fd_t fd = h5::create(
"data.h5", H5F_ACC_TRUNC,
h5::default_fcpl,
h5::threads{N} | h5::backpressure{M}); // M default = 8 × N
```

When `h5::threads{N}` is installed, the FAPL allocates a `worker_pool_t` and parks a `shared_ptr<>` to it inside an `H5Pinsert2` slot. Every dataset created/opened on that file inherits the pool via `H5Fget_access_plist`. When a dataset's DAPL has `h5::high_throughput`, `h5::write` and `h5::read` construct a local `pool_pipeline_t` that submits per-chunk compression closures to the pool and drains in submission order; `H5Dwrite_chunk` still runs on the calling thread. `pt_t` resolves the same pool in `init()` and uses `pool_pipeline_t` as a variant alternative.

Back-pressure is bounded by `h5::backpressure{M}`: the producer blocks on the front future once the in-flight deque hits `M`. Default is 8 × worker count.

The legacy per-pt_t `h5::filter::threads{N}` constructor from #241 is removed in this cycle (see [Phase 1.4 commit message]). Two parallel threading paths in the pipeline invite contention bugs and confuse the surface; the FAPL pool fully subsumes it.

Phase II (compile-time C-API blocking on `async_fd_t`, full async mode) is tracked separately.
Loading
Loading