Skip to content

Commit 9082c4b

Browse files
oberstetoberstet
authored
Rel 25 12 3 (#104)
* add feature branch audit file * fix: use relative imports in vendored flatbuffers reflection module Fixes #102. The FlatBuffers compiler generates Python code with absolute imports like `from reflection.Type import Type` which fail when the module is vendored inside a package. Changed all 18 occurrences to use relative imports (`from .Type import Type`). Files modified: - Schema.py (4 imports) - Service.py (2 imports) - Object.py (2 imports) - Enum.py (3 imports) - EnumVal.py (1 import) - Field.py (2 imports) - RPCCall.py (3 imports) Note: This work was completed with AI assistance (Claude Code). * feat: add fix-flatbuffers-reflection-imports recipe and test Addresses #102 (permanent fix for vendored flatbuffers imports): 1. Added fix-flatbuffers-reflection-imports recipe - Automatically fixes absolute imports to relative imports - Now runs automatically after generate-flatbuffers-reflection - Uses sed to convert 'from reflection.X' to 'from .X' - Verifies fix was successful 2. Added test-reflection recipe and scripts/test_reflection.py - Tests importing Schema and all 10 reflection classes - Tests loading and parsing reflection.bfbs - This is the exact use case cfxdb, wamp-xbr, and crossbar need The fix survives regeneration because generate-flatbuffers-reflection now has fix-flatbuffers-reflection-imports as a dependency. Note: This work was completed with AI assistance (Claude Code). * ci: add test-reflection to CI and expand Python matrix - Added cpy312 and cpy313 to test matrix (was: cpy311, cpy314, pypy311) - Now tests: cpy311, cpy312, cpy313, cpy314, pypy311 - Added test-reflection step to verify FlatBuffers reflection imports - This ensures #102 fix is tested on all Python versions Note: This work was completed with AI assistance (Claude Code). * chore: bump submodules and fix GitHub templates Submodule updates: - .ai: bfb4804 -> d1647dc (polish project README) - .cicd: 024df86 -> 0954ce5 (add helper git version, fix PR template) GitHub template fixes: - Move PR template to root level (.github/pull_request_template.md) for GitHub auto-population (was in PULL_REQUEST_TEMPLATE/ subdirectory) - Remove obsolete .github/ISSUE_TEMPLATE.md (using ISSUE_TEMPLATE/ dir) Note: This work was completed with AI assistance (Claude Code). * docs: add FlatBuffers integration documentation Adds comprehensive documentation for FlatBuffers integration: - docs/flatbuffers/index.rst: Overview of FlatBuffers integration, explaining why FlatBuffers was chosen (zero-copy access, schema evolution, cross-language support, memory efficiency, reflection) - docs/flatbuffers/wamp-zerocopy-dataplane.rst (issue #98): Architecture and design of the zero-copy data plane for WAMP, covering data flow from LMDB through FlatBuffers to WAMP transport, design principles, and use cases - docs/flatbuffers/vendoring-design-and-technology.rst (issue #99): Technical details on native binaries, manylinux compliance, ISA compatibility, bundled flatc, and PyPy support Updates docs/index.rst to include the new flatbuffers section. Closes: #98, #99 Note: This work was completed with AI assistance (Claude Code).
1 parent 5e4d901 commit 9082c4b

File tree

19 files changed

+740
-37
lines changed

19 files changed

+740
-37
lines changed

‎.ai‎

Submodule .ai updated 1 file

‎.audit/oberstet_rel-25-12-3.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
- [ ] I did **not** use any AI-assistance tools to help create this pull request.
2+
- [x] I **did** use AI-assistance tools to *help* create this pull request.
3+
- [x] I have read, understood and followed the project's AI_POLICY.md when creating code, documentation etc. for this pull request.
4+
5+
Submitted by: @oberstet
6+
Date: 2025-12-16
7+
Related issue(s): #103
8+
Branch: oberstet:rel-25-12-3

‎.cicd‎

‎.github/ISSUE_TEMPLATE.md‎

Lines changed: 0 additions & 15 deletions
This file was deleted.

.github/PULL_REQUEST_TEMPLATE/pull_request_template.md renamed to .github/pull_request_template.md

File renamed without changes.

‎.github/workflows/main.yml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ jobs:
100100

101101
strategy:
102102
matrix:
103-
python-env: [cpy311, cpy314, pypy311]
103+
python-env: [cpy311, cpy312, cpy313, cpy314, pypy311]
104104

105105
steps:
106106
- name: Checkout code
@@ -158,6 +158,9 @@ jobs:
158158
- name: Run unit tests (PyTest) - zLMDB ORM
159159
run: just test-orm ${{ matrix.python-env }}
160160

161+
- name: Test FlatBuffers reflection imports
162+
run: just test-reflection ${{ matrix.python-env }}
163+
161164
documentation:
162165
name: Documentation Build
163166
needs: identifiers

‎docs/flatbuffers/index.rst‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
FlatBuffers Integration
2+
=======================
3+
4+
zLMDB integrates `FlatBuffers <https://flatbuffers.dev/>`__ as a core component
5+
for high-performance, zero-copy data serialization. This section documents
6+
the architecture, design decisions, and technical implementation details.
7+
8+
.. toctree::
9+
:maxdepth: 2
10+
11+
wamp-zerocopy-dataplane
12+
vendoring-design-and-technology
13+
14+
Overview
15+
--------
16+
17+
FlatBuffers is central to zLMDB's design, not incidental. The key insight is:
18+
19+
**FlatBuffers schemas serve as the single source of truth for data
20+
representation — across both storage (data-at-rest) and transport
21+
(data-in-transit).**
22+
23+
This enables a unified data model where:
24+
25+
* **zLMDB** handles persistent storage with memory-mapped access
26+
* **Autobahn|Python** handles WAMP messaging and transport
27+
* Both operate on the **same FlatBuffers data model**
28+
29+
The result is a schema-first, zero-copy, end-to-end data plane for WAMP
30+
applications.
31+
32+
Why FlatBuffers?
33+
----------------
34+
35+
FlatBuffers was chosen for several key properties:
36+
37+
1. **Zero-copy access** — Read structured data directly from memory without
38+
deserialization overhead
39+
40+
2. **Schema evolution** — Forward and backward compatibility without breaking
41+
existing data
42+
43+
3. **Cross-language support** — Same schema works in Python, C++, JavaScript,
44+
and other languages
45+
46+
4. **Memory efficiency** — No intermediate representation; data is accessed
47+
in-place
48+
49+
5. **Reflection support** — Runtime schema introspection via ``.bfbs`` files
50+
enables dynamic tooling
51+
52+
Documentation Pages
53+
-------------------
54+
55+
:doc:`wamp-zerocopy-dataplane`
56+
Architecture and design of the FlatBuffers-based zero-copy data plane
57+
for WAMP, explaining how data flows from storage through transport
58+
without serialization overhead.
59+
60+
:doc:`vendoring-design-and-technology`
61+
Technical details on native binaries, manylinux compliance, ISA
62+
compatibility, and PyPy support — explaining *how* and *why* FlatBuffers
63+
and flatc are bundled in Python wheels.

‎docs/flatbuffers/vendoring-design-and-technology.rst‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
Vendoring Design and Native Binaries
2+
====================================
3+
4+
.. note::
5+
6+
This document explains *how* and *why* native artifacts are built,
7+
bundled, verified, and distributed in zLMDB. It complements the
8+
higher-level :doc:`wamp-zerocopy-dataplane` architecture documentation.
9+
10+
Overview
11+
--------
12+
13+
Both **zLMDB** and **Autobahn|Python** ship **native code and native
14+
executables** as part of their Python distributions:
15+
16+
* CFFI-based native extensions (LMDB)
17+
* A bundled **FlatBuffers compiler** (``flatc``)
18+
* Native wheels for **x86-64 and ARM64**
19+
* First-class **PyPy** support
20+
* Fully **manylinux-compliant** wheels verified with ``auditwheel``
21+
22+
This is powerful — but also subtle and non-obvious.
23+
24+
What Users Get
25+
--------------
26+
27+
From a user's point of view, installation is intentionally simple:
28+
29+
.. code-block:: bash
30+
31+
pip install zlmdb
32+
# or
33+
pip install autobahn
34+
35+
After installation, users have:
36+
37+
* A working ``flatc`` executable
38+
* A native LMDB backend
39+
* No system-level dependencies required
40+
* Identical behavior on CPython and PyPy
41+
42+
Behind the scenes, careful engineering ensures that:
43+
44+
* Wheels install without compilation
45+
* Binaries work across distributions
46+
* PyPy works just as well as CPython
47+
* ``auditwheel repair`` succeeds
48+
* ISA incompatibilities (e.g. ``x86_64_v2``) are avoided
49+
* Users do not need system FlatBuffers, LMDB, or compilers
50+
51+
Native Components in the Wheel
52+
------------------------------
53+
54+
Each wheel contains:
55+
56+
**CFFI-based LMDB Extension**
57+
The ``_lmdb_cffi.*.so`` shared library provides high-performance
58+
access to LMDB without requiring compilation at install time.
59+
60+
**Native flatc Executable**
61+
The FlatBuffers compiler is shipped as a native executable inside
62+
the package, invoked via a Python wrapper.
63+
64+
**Vendored FlatBuffers Runtime**
65+
The FlatBuffers Python runtime library and reflection data
66+
(``reflection.bfbs``) are included.
67+
68+
All of these are installed transparently into the Python environment.
69+
70+
manylinux and ISA Compatibility
71+
-------------------------------
72+
73+
Key constraints for Linux wheels:
74+
75+
* Wheels must run on a wide range of Linux distributions
76+
* ``auditwheel`` enforces:
77+
78+
* ABI compatibility
79+
* Baseline CPU instruction sets
80+
81+
Compiling on a "modern" host can silently introduce:
82+
83+
* ``x86_64_v2`` instructions (SSE4.2, etc.)
84+
* Too-new symbol versions
85+
86+
Therefore:
87+
88+
* Builds are done in **manylinux containers**
89+
* Baseline architecture flags are enforced
90+
* Toolchains are chosen carefully
91+
* Wheels are verified with ``auditwheel repair``
92+
93+
This ensures wheels are installable on older systems, not just CI hosts.
94+
95+
Bundled flatc: Why and How
96+
--------------------------
97+
98+
Why Bundle flatc?
99+
^^^^^^^^^^^^^^^^^
100+
101+
* **Avoid system dependencies** — Users don't need to install FlatBuffers
102+
* **Schema/compiler compatibility** — Schema and compiler versions always match
103+
* **Reproducible builds** — Same compiler everywhere
104+
* **Hermetic CI** — No external dependencies during builds
105+
* **Schema-driven workflows** — Works out of the box
106+
107+
How flatc is Exposed
108+
^^^^^^^^^^^^^^^^^^^^
109+
110+
* ``flatc`` is shipped as a native executable inside the package
111+
* A Python console script (``flatc``) dispatches to it
112+
* Works identically on CPython and PyPy
113+
* No PATH manipulation required
114+
115+
.. code-block:: bash
116+
117+
# After pip install zlmdb
118+
flatc --version
119+
120+
This just works — even on PyPy.
121+
122+
PyPy as a First-Class Target
123+
----------------------------
124+
125+
PyPy support is not accidental — it is a design requirement.
126+
127+
Why PyPy Matters
128+
^^^^^^^^^^^^^^^^
129+
130+
Running on PyPy allows "almost C-like" performance, since PyPy is
131+
a *tracing JIT compiler* for Python with a *generational garbage
132+
collector*. Both features are crucial for:
133+
134+
* High throughput and bandwidth
135+
* Consistent low latency
136+
* Memory-efficient operation
137+
138+
How PyPy is Supported
139+
^^^^^^^^^^^^^^^^^^^^^
140+
141+
* LMDB is accessed via **CFFI**, not CPython C-API
142+
* Native wheels are built and published for PyPy
143+
* No fallback-to-sdist compilation required for users
144+
145+
The choice of CFFI over CPython's C-API is deliberate. The CPython
146+
C-API (CPyExt) is implementation-defined and performs poorly on PyPy.
147+
CFFI provides:
148+
149+
* Native performance on both CPython and PyPy
150+
* Clean separation between Python and native code
151+
* Consistent behavior across Python implementations
152+
153+
.. seealso::
154+
155+
`Inside CPyExt <https://pypy.org/posts/2018/09/inside-cpyext-why-emulating-cpython-c-8083064623681286567.html>`__
156+
— PyPy blog post explaining why CPyExt emulation is problematic.
157+
158+
Dynamic vs Static Linking
159+
-------------------------
160+
161+
The build process carefully manages linking:
162+
163+
**Dynamically Linked**
164+
* ``libc`` — Required for manylinux compatibility
165+
* System libraries that manylinux policy allows
166+
167+
**Statically Linked or Bundled**
168+
* LMDB library code
169+
* FlatBuffers runtime
170+
171+
Why This Matters
172+
^^^^^^^^^^^^^^^^
173+
174+
* ``auditwheel`` compatibility is more important than "fully static" binaries
175+
* Symbol versions must match manylinux policy
176+
* ISA levels must be compatible with target platforms
177+
178+
Build Infrastructure
179+
--------------------
180+
181+
Wheels are built using:
182+
183+
* **GitHub Actions** for CI/CD
184+
* **manylinux containers** (``manylinux_2_28``) for Linux builds
185+
* **cibuildwheel** for cross-platform wheel building
186+
* **auditwheel** for verification
187+
188+
The build matrix includes:
189+
190+
* CPython 3.11, 3.12, 3.13, 3.14
191+
* PyPy 3.11
192+
* Linux x86-64 and ARM64
193+
194+
Version Synchronization
195+
-----------------------
196+
197+
FlatBuffers is vendored to ensure version consistency:
198+
199+
* The ``flatc`` binary version matches the Python runtime
200+
* The reflection schema (``.bfbs``) matches both
201+
* zLMDB and Autobahn|Python can verify compatibility at runtime
202+
203+
This prevents subtle bugs from version mismatches between:
204+
205+
* Schema compilation (``flatc``)
206+
* Runtime deserialization (Python library)
207+
* Binary schema files (``.bfbs``)
208+
209+
Related Documentation
210+
---------------------
211+
212+
* :doc:`wamp-zerocopy-dataplane` — Architecture overview of the
213+
FlatBuffers-based zero-copy data plane
214+
* `FlatBuffers Documentation <https://flatbuffers.dev/>`__
215+
* `LMDB Documentation <http://www.lmdb.tech/doc/>`__
216+
* `PyPy Documentation <https://doc.pypy.org/>`__
217+
* `manylinux Policy <https://github.com/pypa/manylinux>`__

0 commit comments

Comments
 (0)