Vendored flatbuffers improvements new#92
Merged
oberstet merged 27 commits intocrossbario:masterfrom Dec 12, 2025
Merged
Conversation
…gly branched vendored-flatbuffers-improvements)
- Re-export vendored flatbuffers as `zlmdb.flatbuffers` - Add setup_flatbuffers_import() function for downstream packages - Allows cfxdb and other packages to explicitly set up flatbuffers imports - Add both to __all__ for public API This enables downstream packages (e.g. cfxdb) with generated flatbuffers code to reliably use zlmdb's vendored flatbuffers runtime. Note: This work was completed with AI assistance (Claude Code).
- Bump version to 25.12.2 - Require Python >= 3.11 (drop 3.9, 3.10 support) - Add "License :: OSI Approved :: MIT License" classifier - Update ruff target-version to py311 - Add check_autobahn_flatbuffers_version_in_sync() helper function to verify zlmdb and autobahn use the same flatbuffers version (important for data-at-rest/data-in-transit interoperability) Note: This work was completed with AI assistance (Claude Code).
- Add _git_version.py with __git_version__ (e.g. "v25.9.23-2-g95053e6a") - Add version() function to _flatbuffers_vendor/__init__.py - Update hatch_build.py to capture git describe from deps/flatbuffers at build time and write to _git_version.py - Fix _version.py to have correct version "25.9.23" (not "latest") - Update check_autobahn_flatbuffers_version_in_sync() to use version() This enables: import zlmdb print(zlmdb.flatbuffers.version()) # "v25.9.23-2-g95053e6a" The git version is captured at build time from the deps/flatbuffers submodule, providing the definitive version for data-at-rest/data-in-transit interoperability checks with autobahn. Note: This work was completed with AI assistance (Claude Code).
…y license expressions (see https://peps.python.org/pep-0639/)
- Rename src/zlmdb/_flatbuffers_vendor/ to src/zlmdb/flatbuffers/ - Update zlmdb/__init__.py imports from _flatbuffers_vendor to flatbuffers - Fix NameError: _lmdb_vendor not defined (use explicit import) - Update justfile update-flatbuffers recipe for new path - Replace old hand-patched reflection code with fresh upstream copy This aligns zlmdb's flatbuffers vendoring with autobahn-python's approach. Note: This work was completed with AI assistance (Claude Code).
- Add src/zlmdb/_flatc/ module with Python shim for bundled flatc - Build flatc from deps/flatbuffers during wheel build (hatch_build.py) - Generate reflection.bfbs during build for runtime schema introspection - Add console script: `flatc` command available after `pip install zlmdb` - Copy reflection.fbs from upstream for source schema access - Update pyproject.toml with flatc console script and sdist includes - Add .gitignore entries for build artifacts (flatc binary, reflection.bfbs) Users can now: - Run `flatc --version` after installing zlmdb - Use `from zlmdb._flatc import get_flatc_path, run_flatc` programmatically - Access reflection.fbs and reflection.bfbs at runtime for schema introspection Note: This work was completed with AI assistance (Claude Code).
- Tests flatc console script via `flatc --version` - Tests Python API: get_flatc_path() and run_flatc() - Verifies reflection.fbs is accessible at runtime - Verifies reflection.bfbs is accessible at runtime Note: This work was completed with AI assistance (Claude Code).
The hatchling `include` directive overrides default git-tracked file inclusion, causing the sdist to only contain flatbuffers sources while excluding: - src/zlmdb/ (Python sources) - lmdb-upstream/ (LMDB C sources) - lmdb-patches/ (patches) - build scripts Remove the `include` directive and rely on exclude-only approach. The exclude list removes build artifacts and large flatbuffers subdirectories (tests, samples, docs, language bindings) while keeping what's needed for building flatc from source. Note: This work was completed with AI assistance (Claude Code).
On Linux x86_64, pass -march=x86-64 -mtune=generic to cmake when building flatc. Without this, the compiler may use x86_64_v2+ instructions (SSE4.2, etc.) which auditwheel rejects as incompatible with manylinux. Fixes: auditwheel error "ELF file requires 'x86_64_v2' instruction set, not in 'x86_64'" during CI wheel builds. Note: This work was completed with AI assistance (Claude Code).
…inux compatibility
The previous approach using cmake -DCMAKE_C_FLAGS was not being applied reliably in CI. Changes: 1. Clean build directory before configuring to remove any cached CMakeCache.txt that might override our flags 2. Use CFLAGS/CXXFLAGS environment variables instead of cmake -D arguments - these take precedence and are more reliable 3. Pass the environment to both configure and build subprocess calls 4. Log the actual CFLAGS/CXXFLAGS being used for debugging This ensures flatc is built with baseline x86-64 instructions (-march=x86-64 -mtune=generic) so auditwheel doesn't reject it for depending on x86_64_v2 ISA extensions. Note: This work was completed with AI assistance (Claude Code).
The _INIT variants are applied BEFORE the project sets its own flags, and FlatBuffers appends rather than replaces. This ensures our baseline ISA flags (-march=x86-64 -mtune=generic) take effect. Previous approaches failed because: - CMAKE_*_FLAGS: FlatBuffers overwrites these - CFLAGS/CXXFLAGS env vars: Not always honored by cmake Also adds ISA verification logging after building flatc to confirm the correct instruction set was used (helps debug CI issues). Note: This work was completed with AI assistance (Claude Code).
The x86_64_v2 instructions were coming from dynamically linked libstdc++/libgcc, not from our compiled code. Even with baseline ISA flags, linking against system C++ runtime that contains v2 instructions causes auditwheel to reject the wheel. Solution: Use -static-libgcc -static-libstdc++ via CMAKE_EXE_LINKER_FLAGS_INIT to statically link these libraries, eliminating all x86_64_v2 leakage from the system toolchain. Note: This work was completed with AI assistance (Claude Code).
Complete the cmake flags for manylinux-compatible flatc builds: - CMAKE_C_FLAGS_INIT: baseline x86-64 ISA (no SSE4.2, AVX, etc.) - CMAKE_CXX_FLAGS_INIT: baseline x86-64 ISA for C++ code - FLATBUFFERS_CXX_FLAGS=: clear FlatBuffers' own flags - CMAKE_EXE_LINKER_FLAGS: static link libstdc++/libgcc Result: - Portable x86-64 baseline ISA only - No accidental v2/v3 instructions - Static C++ runtime (no x86_64_v2 from system libstdc++) - Static GCC runtime - Dynamic glibc (required for manylinux) - auditwheel-safe manylinux wheel Note: This work was completed with AI assistance (Claude Code).
Use belt-and-suspenders approach for baseline x86-64/arm64 flags: - Environment variables (CFLAGS, CXXFLAGS, LDFLAGS) via cmake_env - CMAKE_*_FLAGS_INIT (applied before project flags) - CMAKE_*_FLAGS (fallback) - FLATBUFFERS_CXX_FLAGS= (clear project additions) - CMAKE_EXE_LINKER_FLAGS_INIT and non-INIT for static linking Also add cmake command logging for debugging CI failures. Note: This work was completed with AI assistance (Claude Code).
Minimal change to fix auditwheel x86_64_v2 ISA compliance: wheels-docker.yml: - Switch from manylinux_2_34 to manylinux_2_28 (matches wheels-arm64.yml) - Container toolchain handles correct glibc/ISA requirements wheels.yml: - Remove Linux x86_64 wheel building (handled by wheels-docker.yml) - Keep Linux job for source distribution only - Keep macOS and Windows native wheel builds hatch_build.py: - Remove static linking hacks (no longer needed with container builds) - Add explanatory comment about manylinux container approach This is a less invasive fix than adding a container job to wheels.yml. Note: This work was completed with AI assistance (Claude Code).
release.yml: - Update artifact name from manylinux_2_34 to manylinux_2_28 - Remove artifact_linux_wheels (Linux wheels now from wheels-docker.yml) - Add comments explaining artifact sources .github/workflows/README.md: - Document all workflows and their purposes - Map artifact producers to consumers - Include ASCII diagram of artifact flow - Detail platform coverage and Python versions - Explain why manylinux containers are used Note: This work was completed with AI assistance (Claude Code).
Closed
4 tasks
CFFI compilation under QEMU ARM64 emulation is very slow. CPython builds were timing out at 15 minutes during the LMDB CFFI extension compilation step. Increased timeout from 15 to 30 minutes to accommodate the emulation overhead. Note: This work was completed with AI assistance (Claude Code).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this fixes #90
current WIP evaluation of this PR by different AI (ChatGPT):