Commit dfaac7b
authored
Vendored flatbuffers improvements new (#92)
* create new feature branch (this time from master tip! correcting wrongly branched vendored-flatbuffers-improvements)
* Add flatbuffers re-export and setup_flatbuffers_import() helper
- Re-export vendored flatbuffers as `zlmdb.flatbuffers`
- Add setup_flatbuffers_import() function for downstream packages
- Allows cfxdb and other packages to explicitly set up flatbuffers imports
- Add both to __all__ for public API
This enables downstream packages (e.g. cfxdb) with generated flatbuffers
code to reliably use zlmdb's vendored flatbuffers runtime.
Note: This work was completed with AI assistance (Claude Code).
* Bump to v25.12.2, require Python 3.11+, add version sync check
- Bump version to 25.12.2
- Require Python >= 3.11 (drop 3.9, 3.10 support)
- Add "License :: OSI Approved :: MIT License" classifier
- Update ruff target-version to py311
- Add check_autobahn_flatbuffers_version_in_sync() helper function
to verify zlmdb and autobahn use the same flatbuffers version
(important for data-at-rest/data-in-transit interoperability)
Note: This work was completed with AI assistance (Claude Code).
* Add flatbuffers.version() returning exact git describe version
- Add _git_version.py with __git_version__ (e.g. "v25.9.23-2-g95053e6a")
- Add version() function to _flatbuffers_vendor/__init__.py
- Update hatch_build.py to capture git describe from deps/flatbuffers
at build time and write to _git_version.py
- Fix _version.py to have correct version "25.9.23" (not "latest")
- Update check_autobahn_flatbuffers_version_in_sync() to use version()
This enables:
import zlmdb
print(zlmdb.flatbuffers.version()) # "v25.9.23-2-g95053e6a"
The git version is captured at build time from the deps/flatbuffers
submodule, providing the definitive version for data-at-rest/data-in-transit
interoperability checks with autobahn.
Note: This work was completed with AI assistance (Claude Code).
* avoid InvalidConfigError - license classifiers have been superseded by license expressions (see https://peps.python.org/pep-0639/)
* expand test-import justfile recipe to test zlmdb.flatbuffers.version()
* track FlatBuffers 25.9.23 release (not development)
* fix: F541 [*] f-string without any placeholders
* track FlatBuffers 25.9.23 release (not development)
* parse flatbuffers version into version components tuple
* expand just test-import recipe
* apply just autoformat ruff formatting
* Rename _flatbuffers_vendor to flatbuffers (match autobahn-python)
- Rename src/zlmdb/_flatbuffers_vendor/ to src/zlmdb/flatbuffers/
- Update zlmdb/__init__.py imports from _flatbuffers_vendor to flatbuffers
- Fix NameError: _lmdb_vendor not defined (use explicit import)
- Update justfile update-flatbuffers recipe for new path
- Replace old hand-patched reflection code with fresh upstream copy
This aligns zlmdb's flatbuffers vendoring with autobahn-python's approach.
Note: This work was completed with AI assistance (Claude Code).
* build: remove legacy readthedocs.yml to activate .readthedocs.yaml
* Bundle flatc compiler in zlmdb wheel (like grpcio-tools)
- Add src/zlmdb/_flatc/ module with Python shim for bundled flatc
- Build flatc from deps/flatbuffers during wheel build (hatch_build.py)
- Generate reflection.bfbs during build for runtime schema introspection
- Add console script: `flatc` command available after `pip install zlmdb`
- Copy reflection.fbs from upstream for source schema access
- Update pyproject.toml with flatc console script and sdist includes
- Add .gitignore entries for build artifacts (flatc binary, reflection.bfbs)
Users can now:
- Run `flatc --version` after installing zlmdb
- Use `from zlmdb._flatc import get_flatc_path, run_flatc` programmatically
- Access reflection.fbs and reflection.bfbs at runtime for schema introspection
Note: This work was completed with AI assistance (Claude Code).
* Add test-bundled-flatc justfile recipe
- Tests flatc console script via `flatc --version`
- Tests Python API: get_flatc_path() and run_flatc()
- Verifies reflection.fbs is accessible at runtime
- Verifies reflection.bfbs is accessible at runtime
Note: This work was completed with AI assistance (Claude Code).
* Fix sdist to include all source files
The hatchling `include` directive overrides default git-tracked
file inclusion, causing the sdist to only contain flatbuffers
sources while excluding:
- src/zlmdb/ (Python sources)
- lmdb-upstream/ (LMDB C sources)
- lmdb-patches/ (patches)
- build scripts
Remove the `include` directive and rely on exclude-only approach.
The exclude list removes build artifacts and large flatbuffers
subdirectories (tests, samples, docs, language bindings) while
keeping what's needed for building flatc from source.
Note: This work was completed with AI assistance (Claude Code).
* Use baseline x86-64 flags for flatc manylinux compatibility
On Linux x86_64, pass -march=x86-64 -mtune=generic to cmake when
building flatc. Without this, the compiler may use x86_64_v2+
instructions (SSE4.2, etc.) which auditwheel rejects as incompatible
with manylinux.
Fixes: auditwheel error "ELF file requires 'x86_64_v2' instruction
set, not in 'x86_64'" during CI wheel builds.
Note: This work was completed with AI assistance (Claude Code).
* improved logging & march support for baseline compile flags for manylinux compatibility
* Fix flatc baseline x86-64 flags via environment variables
The previous approach using cmake -DCMAKE_C_FLAGS was not being
applied reliably in CI. Changes:
1. Clean build directory before configuring to remove any cached
CMakeCache.txt that might override our flags
2. Use CFLAGS/CXXFLAGS environment variables instead of cmake -D
arguments - these take precedence and are more reliable
3. Pass the environment to both configure and build subprocess calls
4. Log the actual CFLAGS/CXXFLAGS being used for debugging
This ensures flatc is built with baseline x86-64 instructions
(-march=x86-64 -mtune=generic) so auditwheel doesn't reject it
for depending on x86_64_v2 ISA extensions.
Note: This work was completed with AI assistance (Claude Code).
* Use CMAKE_*_FLAGS_INIT for baseline x86-64 flags
The _INIT variants are applied BEFORE the project sets its own
flags, and FlatBuffers appends rather than replaces. This ensures
our baseline ISA flags (-march=x86-64 -mtune=generic) take effect.
Previous approaches failed because:
- CMAKE_*_FLAGS: FlatBuffers overwrites these
- CFLAGS/CXXFLAGS env vars: Not always honored by cmake
Also adds ISA verification logging after building flatc to confirm
the correct instruction set was used (helps debug CI issues).
Note: This work was completed with AI assistance (Claude Code).
* Add static linking for libstdc++/libgcc in flatc build
The x86_64_v2 instructions were coming from dynamically linked
libstdc++/libgcc, not from our compiled code. Even with baseline
ISA flags, linking against system C++ runtime that contains v2
instructions causes auditwheel to reject the wheel.
Solution: Use -static-libgcc -static-libstdc++ via
CMAKE_EXE_LINKER_FLAGS_INIT to statically link these libraries,
eliminating all x86_64_v2 leakage from the system toolchain.
Note: This work was completed with AI assistance (Claude Code).
* Add FLATBUFFERS_CXX_FLAGS= and detailed manylinux comments
Complete the cmake flags for manylinux-compatible flatc builds:
- CMAKE_C_FLAGS_INIT: baseline x86-64 ISA (no SSE4.2, AVX, etc.)
- CMAKE_CXX_FLAGS_INIT: baseline x86-64 ISA for C++ code
- FLATBUFFERS_CXX_FLAGS=: clear FlatBuffers' own flags
- CMAKE_EXE_LINKER_FLAGS: static link libstdc++/libgcc
Result:
- Portable x86-64 baseline ISA only
- No accidental v2/v3 instructions
- Static C++ runtime (no x86_64_v2 from system libstdc++)
- Static GCC runtime
- Dynamic glibc (required for manylinux)
- auditwheel-safe manylinux wheel
Note: This work was completed with AI assistance (Claude Code).
* Add env vars alongside cmake args for manylinux ISA compliance
Use belt-and-suspenders approach for baseline x86-64/arm64 flags:
- Environment variables (CFLAGS, CXXFLAGS, LDFLAGS) via cmake_env
- CMAKE_*_FLAGS_INIT (applied before project flags)
- CMAKE_*_FLAGS (fallback)
- FLATBUFFERS_CXX_FLAGS= (clear project additions)
- CMAKE_EXE_LINKER_FLAGS_INIT and non-INIT for static linking
Also add cmake command logging for debugging CI failures.
Note: This work was completed with AI assistance (Claude Code).
* Use manylinux_2_28 containers for Linux wheel builds
Minimal change to fix auditwheel x86_64_v2 ISA compliance:
wheels-docker.yml:
- Switch from manylinux_2_34 to manylinux_2_28 (matches wheels-arm64.yml)
- Container toolchain handles correct glibc/ISA requirements
wheels.yml:
- Remove Linux x86_64 wheel building (handled by wheels-docker.yml)
- Keep Linux job for source distribution only
- Keep macOS and Windows native wheel builds
hatch_build.py:
- Remove static linking hacks (no longer needed with container builds)
- Add explanatory comment about manylinux container approach
This is a less invasive fix than adding a container job to wheels.yml.
Note: This work was completed with AI assistance (Claude Code).
* Fix release.yml for manylinux_2_28 and add workflows README
release.yml:
- Update artifact name from manylinux_2_34 to manylinux_2_28
- Remove artifact_linux_wheels (Linux wheels now from wheels-docker.yml)
- Add comments explaining artifact sources
.github/workflows/README.md:
- Document all workflows and their purposes
- Map artifact producers to consumers
- Include ASCII diagram of artifact flow
- Detail platform coverage and Python versions
- Explain why manylinux containers are used
Note: This work was completed with AI assistance (Claude Code).
* Increase ARM64 build timeout to 30 minutes
CFFI compilation under QEMU ARM64 emulation is very slow.
CPython builds were timing out at 15 minutes during the
LMDB CFFI extension compilation step.
Increased timeout from 15 to 30 minutes to accommodate the
emulation overhead.
Note: This work was completed with AI assistance (Claude Code).1 parent fd287f4 commit dfaac7b
File tree
49 files changed
+2648
-3293
lines changed- .audit
- .github/workflows
- deps
- src/zlmdb
- _flatbuffers_vendor
- reflection
- _flatc
- flatbuffers
- reflection
- tests
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
49 files changed
+2648
-3293
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | | - | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
131 | 132 | | |
132 | 133 | | |
133 | 134 | | |
| 135 | + | |
| 136 | + | |
134 | 137 | | |
135 | 138 | | |
136 | 139 | | |
137 | | - | |
138 | 140 | | |
139 | 141 | | |
140 | | - | |
| 142 | + | |
141 | 143 | | |
142 | | - | |
| 144 | + | |
143 | 145 | | |
144 | 146 | | |
145 | 147 | | |
| |||
183 | 185 | | |
184 | 186 | | |
185 | 187 | | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
| 188 | + | |
| 189 | + | |
195 | 190 | | |
196 | 191 | | |
197 | 192 | | |
| |||
523 | 518 | | |
524 | 519 | | |
525 | 520 | | |
526 | | - | |
527 | | - | |
528 | | - | |
529 | | - | |
530 | | - | |
531 | | - | |
532 | | - | |
533 | | - | |
534 | | - | |
| 521 | + | |
| 522 | + | |
535 | 523 | | |
536 | 524 | | |
537 | 525 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
125 | | - | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
126 | 128 | | |
127 | 129 | | |
128 | 130 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
| 42 | + | |
| 43 | + | |
43 | 44 | | |
44 | | - | |
45 | | - | |
| 45 | + | |
| 46 | + | |
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
| |||
151 | 152 | | |
152 | 153 | | |
153 | 154 | | |
154 | | - | |
| 155 | + | |
155 | 156 | | |
156 | | - | |
| 157 | + | |
157 | 158 | | |
158 | 159 | | |
159 | 160 | | |
| |||
278 | 279 | | |
279 | 280 | | |
280 | 281 | | |
281 | | - | |
| 282 | + | |
282 | 283 | | |
283 | 284 | | |
284 | 285 | | |
| |||
0 commit comments