[BUG] flatc must be _built_ & bundled, but NOT _used_ when building / packaging (otherwise it'll bite back in cross-compilation scenarios)

[jameshilliard]

Bug Description

Summary:
We can't run target binaries on cross compilation build hosts.

Expected Behavior:
Have a way to specify host compatible versions of binaries.

Actual Behavior:

ERROR: flatc failed:
aarch64-binfmt-P: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory

Reproduction Steps

Attempt to build with buildroot.

################################################################################
#
# python-zlmdb
#
################################################################################

PYTHON_ZLMDB_VERSION = 25.12.3
PYTHON_ZLMDB_SOURCE = zlmdb-$(PYTHON_ZLMDB_VERSION).tar.gz
PYTHON_ZLMDB_SITE = https://files.pythonhosted.org/packages/40/69/d403e26bc059a70578d0406b29cb239a9aa018e28d79f92b8106a1a7d8e5
PYTHON_ZLMDB_SETUP_TYPE = hatch
PYTHON_ZLMDB_LICENSE = MIT
PYTHON_ZLMDB_LICENSE_FILES = LICENSE
PYTHON_ZLMDB_DEPENDENCIES = \
	host-python-cffi \
	host-python-setuptools

$(eval $(python-package))

Additional Context

Error Messages:

======================================================================
Building FlatBuffers compiler (flatc)
======================================================================
  -> Configuring with cmake...
  -> Building flatc...
  -> Built flatc: /home/buildroot/buildroot/output/build/python-zlmdb-25.12.3/src/zlmdb/_flatc/bin/flatc
  -> Verifying ISA level...
     /home/buildroot/buildroot/output/build/python-zlmdb-25.12.3/src/zlmdb/_flatc/bin/flatc: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped
  -> Added to wheel: zlmdb/_flatc/bin/flatc

======================================================================
Generating reflection.bfbs
======================================================================
ERROR: flatc failed:
aarch64-binfmt-P: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory

[oberstet]

Thanks for the detailed bug report! This is a valid cross-compilation issue.

Root Cause Analysis

The problem is that hatch_build.py tries to run the freshly-built flatc during the build to generate reflection.bfbs. When cross-compiling (x86_64 host → ARM64 target), the built flatc is an ARM64 binary that cannot execute on the x86_64 host.

Looking at the build output:

-> Built flatc: .../src/zlmdb/_flatc/bin/flatc
-> Verifying ISA level...
   .../flatc: ELF 64-bit LSB pie executable, ARM aarch64, ...
...
ERROR: flatc failed:
aarch64-binfmt-P: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory

The build correctly compiles flatc for the target architecture (ARM64), but then incorrectly tries to execute it on the host to generate reflection.bfbs.

Current State

Currently:

• reflection.bfbs is listed in .gitignore and regenerated during every build
• The reflection Python wrappers (src/zlmdb/flatbuffers/reflection/*.py) ARE committed to git
• This inconsistency means reflection.bfbs generation requires running flatc, which fails during cross-compilation

Proposed Fix

The fix follows the same pattern we already use for the reflection Python wrappers: pre-generate and commit to git.

Changes Required

1. Remove reflection.bfbs from .gitignore and commit it to the source tree

2. Add version tracking for generated reflection files via src/zlmdb/flatbuffers/reflection/_version.py:

   # Auto-generated - do not edit manually
   # flatc version used to generate reflection.bfbs and Python wrappers
   __flatc_version__ = "25.9.23"

3. Add/update just recipe (e.g., just build-reflection) that:

   • Builds flatc from deps/flatbuffers submodule
   • Uses that flatc to regenerate reflection.bfbs + Python wrappers
   • Updates reflection/_version.py with the flatc version used
   • This recipe is run by the developer, not during packaging

4. Modify hatch_build.py to simply include the pre-committed reflection.bfbs in the wheel, without trying to regenerate it

Developer Workflow (after fix)

When bumping the vendored flatbuffers version:

1. Update deps/flatbuffers submodule to new version
2. Run just build-reflection to regenerate all reflection files
3. git add and commit the regenerated files
4. The committed files are then used for all packaging (sdist, wheels, cross-compilation)

Benefits

• Cross-compilation works: No flatc execution needed during packaging
• Reproducible builds: Generated files are version-controlled
• Version tracking: reflection/_version.py ensures the developer doesn't forget to regenerate after bumping flatbuffers
• Consistent with existing pattern: Matches how reflection Python wrappers are already handled

---

we'll need a fix for the next release. In the meantime, as a workaround, you could try building from a pre-built wheel instead of sdist (wheels already contain the compiled flatc and reflection.bfbs).

[oberstet]

btw, autobahn-python has the same issue crossbario/autobahn-python#1832

note: both autobahn-python and zlmdb each vendor & bundle flatbuffers! this is deliberate / by design. eg a user might want to access Crossbar.io database files (which are using LMDB with Flatbuffers), and zlmdb (or specifically, cfxdb) is enough for that use - data-at-rest. a user might want to do WAMP-Flatbuffers (and that's what Crossbar.io supports), and autobahn-python is enough for that use - data-in-transit. finally, we have facilities in both libraries to allow users to verify the bundled flatbuffers is exactly at the same version! which is important, since generated Python wrappers and binary schemata depend on flatbuffers version (change over time).

[jameshilliard]

4. Modify hatch_build.py to simply include the pre-committed reflection.bfbs in the wheel, without trying to regenerate it

I guess the packaging stage would include the pre-generate reflection.bfbs in the sdist as well(buildroot doesn't really support pre-built wheels and generally uses sdists)?

[oberstet]

yes exactly, reflection.bfbs will be in the source tree, in the sdist, and in the wheels, and no flatc (neither system nor built/bundled) is needed, so it should work for buildroot cross-compile perfectly! and I renamed this issue to make that goal explicit, and I want to continue support that.

Debian and Fedora use (I guess) the native binary wheels in packaging, Gentoo uses the sdist but build on the target host without cross-compile (which already worked), and buildroot can use sdist (or directly the source tree) and cross-compile (which will work once this issue is fixed).

Users just using pip or uv will also use the native wheels.

The native wheels are published not only for x86-64, but also ARM64, and adding RISC-V should be easy and I want to do that at some point. The GitHub workflows here in this repo use Qemu and PyPA Docker images.

and all native binary wheels are also available for PyPy - which I personally care a lot about! that includes PyPy on ARM64!

[jameshilliard]

no flatc (neither system nor built/bundled) is needed, so it should work for buildroot cross-compile perfectly!

Yeah, that should make building much easier when cross compiling. Python tends to be one of the most complex languages for cross compilation in general since projects often assume the build host can run target binaries so we have a lot of workarounds to make things work properly.

The native wheels are published not only for x86-64, but also ARM64, and adding RISC-V should be easy and I want to do that at some point. The GitHub workflows here in this repo use Qemu and PyPA Docker images.

For buildroot at least we don't really care about/support native prebuilt wheels since we have far more target architectures/options than upstream projects could ever build native wheels for. We wrap a pep517 compatible build frontend and build wheels ourselves from sdists internally and then install the internally built wheels into the target rootfs since python packaging requires wheels as an intermediary stage these days.

[oberstet]

For buildroot at least we don't really care about/support native prebuilt wheels since we have far more target architectures/options than upstream projects could ever build native wheels for.

yes, I understand! and that's a perfectly valid approach. similar to Gentoo - which has a policy of always build from source - but even more complex, because "cross compile".

the most complex scenario would be: cross-compile on x86-64 for PyPy on ARM64.

curious: does buildroot support PyPy, first class, these days?

---

fwiw, I have invested a lot of work into: make AutobahnPython and zLMDB both first class "modern 2025" Python build & packaging, and first class PyPy support, e.g. the vendored LMDB is using CFFI (which makes it fast & nice on PyPy) rather than CPyExt.

also, following PEPs, and also tracking https://wheelnext.dev/, and I want to publish to https://astral.sh/pyx once that goes live (in addition to PyPI).

zlmdb contains vendored LMDB and vendored Flatbuffers

autobahn-python contains vendored Flatbuffers, and our own native code: NVX, which is SIMD accelerated WebSocket

https://github.com/crossbario/autobahn-python/tree/master/src/autobahn/nvx

ah, and autobahn-python is fully tested using autobahn-testsuite both without and with NVX, eg release notes has summary, and docs include full test results

https://autobahn.readthedocs.io/en/latest/release-notes.html
https://autobahn.readthedocs.io/en/latest/websocket/conformance.html
https://zlmdb.readthedocs.io/en/latest/releases.html

finally, to make native binary wheels work on "old libc" and "old CPUs", NVX by default uses pretty conservative CPU flags: https://github.com/crossbario/autobahn-python/blob/master/src/autobahn/nvx/_compile_args.py

for Gentoo you can of course use -march native, and for buildroot you can of course cross-compile using "latest CPU arch" - though for NVX code, I don't think it would make a big difference compare to our defaults:

Architecture Baselines
----------------------

x86-64 (portable baseline):
    -march=x86-64-v2 (microarchitecture level 2, 2009+)
    Includes: SSE4.2, POPCNT, SSSE3, SSE4.1
    Compatible with: Intel Nehalem+, AMD Bulldozer+ (2009+)
    Coverage: ~99% of x86-64 CPUs from 2010 onwards

ARM64 (portable baseline):
    -march=armv8-a (baseline ARMv8-A architecture)
    Compatible with: Raspberry Pi 4/5, AWS Graviton, Apple M1/M2, etc.
    Coverage: All 64-bit ARM CPUs

[jameshilliard]

which has a policy of always build from source

Yeah, we generally have a similar policy(although we do allow for prebuilt gcc toolchains to some degree), we have a few exceptions for parts of the rust cross toolchain as well due to bootstrapping entirely from source being especially tricky. Rust also requires lots of funny env variables to cross compile properly in general due to some latent bugs in cargo similar to this issue here where host tools are built with target options and then fail to run on the build host.

curious: does buildroot support PyPy, first class, these days?

AFAIU buildroot has no PyPy support whatsoever. I think supporting PyPy would be quite complex for buildroot in general as language feature support tends to be out of sync with cpython(python is on version 3.14 while PyPy only supports up to version 3.11 last I checked) and bootstrapping PyPy from source looks complicated. A while back we used to support selecting either python2 or python3 but that was very annoying and introduced a lot of unwanted complexity as we had to manage two interpreters and their associated packages on the host toolchain side at the same time as well.

for Gentoo you can of course use -march native, and for buildroot you can of course cross-compile using "latest CPU arch" - though for NVX code, I don't think it would make a big difference compare to our defaults:

Hmm, I did notice an issue that probably results in the optimized c modules not building properly, where presumably we are falling back to the pure python variant. This platform.machine detection doesn't really work when cross compiling since it picks up the host rather than the target architecture info. I'm thinking maybe using sysconfig.get_platform() is probably what you want to use there instead of platform.machine() since platform.machine() gives info for the build host while sysconfig.get_platform() gives the target AFAIU.

Build error when compiling for aarch64 from x86_64:

>>> python-autobahn 25.12.2 Building
(cd /home/buildroot/buildroot/output/build/python-autobahn-25.12.2//; _PYTHON_HOST_PLATFORM="linux-aarch64" _PYTHON_PROJECT_BASE="/home/buildroot/buildroot/output/build/python3-3.13.11" _PYTHON_SYSCONFIGDATA_NAME="`{ [ -e /home/buildroot/buildroot/output/per-package/python-autobahn/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/python3.13//_sysconfigdata__linux_*.py ] && basename /home/buildroot/buildroot/output/per-package/python-autobahn/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/python3.13//_sysconfigdata__linux_*.py .py; } || true`" PATH="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin:/home/buildroot/buildroot/output/per-package/python-autobahn/host/sbin:/home/buildroot/.cargo/bin:/home/buildroot/bin:/home/buildroot/.local/bin:/home/buildroot/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" GIT_DIR=. PATH="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin:/home/buildroot/buildroot/output/per-package/python-autobahn/host/sbin:/home/buildroot/.cargo/bin:/home/buildroot/bin:/home/buildroot/.local/bin:/home/buildroot/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" AR="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gcc-ar" AS="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-as" LD="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-ld" NM="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gcc-nm" CC="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gcc" GCC="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gcc" CPP="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-cpp" CXX="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-g++" FC="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gfortran" F77="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gfortran" RANLIB="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gcc-ranlib" READELF="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-readelf" STRIP="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-strip" OBJCOPY="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-objcopy" OBJDUMP="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-objdump" AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as" CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc" CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld" CPPFLAGS_FOR_BUILD="-I/home/buildroot/buildroot/output/per-package/python-autobahn/host/include" CFLAGS_FOR_BUILD="-O2 -I/home/buildroot/buildroot/output/per-package/python-autobahn/host/include" CXXFLAGS_FOR_BUILD="-O2 -I/home/buildroot/buildroot/output/per-package/python-autobahn/host/include" LDFLAGS_FOR_BUILD="-L/home/buildroot/buildroot/output/per-package/python-autobahn/host/lib -Wl,-rpath,/home/buildroot/buildroot/output/per-package/python-autobahn/host/lib" FCFLAGS_FOR_BUILD="" DEFAULT_ASSEMBLER="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-as" DEFAULT_LINKER="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-ld" CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0 -D_FORTIFY_SOURCE=1" CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0 -D_FORTIFY_SOURCE=1" LDFLAGS="" FCFLAGS=" -O2 -g0" FFLAGS=" -O2 -g0" PKG_CONFIG="/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/pkg-config" STAGING_DIR="/home/buildroot/buildroot/output/per-package/python-autobahn/host/aarch64-buildroot-linux-gnu/sysroot" INTLTOOL_PERL=/usr/bin/perl PYTHONPATH="/home/buildroot/buildroot/output/per-package/python-autobahn/host/aarch64-buildroot-linux-gnu/sysroot/usr/lib/python3.13/" PYTHONNOUSERSITE=1  /home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/python3 -m build -n -w )
* Getting build dependencies for wheel...
* Building wheel...
======================================================================
Capturing FlatBuffers git version from deps/flatbuffers
======================================================================
  -> deps/flatbuffers not found or not a git repo
  -> Using existing version in _git_version.py
Building for Python with extension suffix: .cpython-313-aarch64-linux-gnu.so
Building CFFI module: /home/buildroot/buildroot/output/build/python-autobahn-25.12.2/src/autobahn/nvx/_utf8validator.py
generating ./_nvx_utf8validator.c
the current directory is '/home/buildroot/buildroot/output/build/python-autobahn-25.12.2/src/autobahn/nvx'
cc1: error: unknown value ‘native’ for ‘-march’
cc1: note: valid arguments are: armv8-a armv8.1-a armv8.2-a armv8.3-a armv8.4-a armv8.5-a armv8.6-a armv8.7-a armv8.8-a armv8.9-a armv8-r armv9-a armv9.1-a armv9.2-a armv9.3-a armv9.4-a armv9.5-a
warning: build_ext: building extension "_nvx_utf8validator" failed: command '/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gcc' failed with exit code 1

Building CFFI module: /home/buildroot/buildroot/output/build/python-autobahn-25.12.2/src/autobahn/nvx/_xormasker.py
generating ./_nvx_xormasker.c
the current directory is '/home/buildroot/buildroot/output/build/python-autobahn-25.12.2/src/autobahn/nvx'
cc1: error: unknown value ‘native’ for ‘-march’
cc1: note: valid arguments are: armv8-a armv8.1-a armv8.2-a armv8.3-a armv8.4-a armv8.5-a armv8.6-a armv8.7-a armv8.8-a armv8.9-a armv8-r armv9-a armv9.1-a armv9.2-a armv9.3-a armv9.4-a armv9.5-a
warning: build_ext: building extension "_nvx_xormasker" failed: command '/home/buildroot/buildroot/output/per-package/python-autobahn/host/bin/aarch64-linux-gcc' failed with exit code 1


======================================================================
Building FlatBuffers compiler (flatc)
======================================================================
WARNING: /home/buildroot/buildroot/output/build/python-autobahn-25.12.2/deps/flatbuffers not found
  -> Initialize git submodule: git submodule update --init
Successfully built autobahn-25.12.2-py3-none-any.whl

[oberstet]

AFAIU buildroot has no PyPy support whatsoever. I think supporting PyPy would be quite complex for buildroot in general as language feature support tends to be out of sync with cpython(python is on version 3.14 while PyPy only supports up to version 3.11 last I checked) and bootstrapping PyPy from source looks complicated

yes, PyPy latest stable is at 3.11, which is one reason this is the minimum version we test & announce in pyproject.toml - the other reason is that 3.11 is the latest stable available from official package repos for RHEL 9 (which a customer of mine is using).

yes, cross building PyPy from source (starting with RPython) is something that would be a significant challenge!

on the other hand: PyPy is pretty awesome (performance wise), practically relevant (I have measured it), and the JIT of PyPy is just one reason, the other is the incremental GC (!) included, which is of pretty big relevance for performance in network servers (like Crossbar.io) - which is the reason it is a "top prio" for me personally to aim for first class PyPy support in all my packages related to Crossbar.io. I have even taken care to ensure Crossbar.io deps (~160 transitive closure) has only deps which a) use CFFI, and b) publish native binary wheels, c) including for ARM64

I aim to be able to do pip install crossbar on a Raspberry Pi running PyPy - and "it just works" =)

[jameshilliard]

on the other hand: PyPy is pretty awesome (performance wise), practically relevant (I have measured it), and the JIT of PyPy is just one reason, the other is the incremental GC (!) included, which is of pretty big relevance for performance in network servers (like Crossbar.io) - which is the reason it is a "top prio" for me personally to aim for first class PyPy support in all my packages related to Crossbar.io. I have even taken care to ensure Crossbar.io deps (~160 transitive closure) has only deps which a) use CFFI, and b) publish native binary wheels, c) including for ARM64

Yeah, I've used it for performance dependent services in the past(back in the python 2 days), for buildroot there's probably less advantages and more disadvantages to using PyPy however, such as potentially higher memory usage and difficulty in integration. A lot of packages in buildroot provide python c extensions for integrations and they often don't test against PyPy either AFAIU. For embedded systems python c extensions are less used for performance optimization reasons and more just used as easy to integrate glue code for c libraries I think. We used to package crossbar itself in buildroot but had to remove it due to the dependency tree being a bit too large to maintain.

[oberstet]

We used to package crossbar itself in buildroot but had to remove it due to the dependency tree being a bit too large to maintain.

yeah, I understand.

fwiw, the deps tree of Crossbar.io also had been still expanding historically, as we have completed features (from the WAMP Advanced Profile) and such.

but we have been doing quite some cleanup "recently" making sure CFFI supporting deps, and ARM64 official wheels for all deps are available.

if you are interested, here is a recent status part of our modernization efforts: https://github.com/crossbario/crossbar/blob/master/MODERNIZATION.md#appendix-crossbario-dependencies-analysis

for example, we have now vendored & integrated LMDB right into zlmdb, with proper CFFI and PyPy and binary wheels for ARM64

we have vendored Flatbuffers - another historic pain, but required by Crossbar.io

we have removed dependencies like wsaccel, replaced by our own NVX, which is more capable anyways

and so on.

---

now, regarding the general question: "is PyPy worth" for embedded, specifically Crossbar.io, my own view is: yes, very much! here is a summary of the argument:

---

Summary: PyPy on ARM64 for Crossbar.io — Platform Support and Practical Relevance

This issue combines two closely related topics:

1. Status of PyPy support on Buildroot vs Yocto vs Debian (ARM64)
2. Why PyPy is practically and significantly relevant for Crossbar.io–class network / messaging servers on ARM64

---

1. PyPy platform support on ARM64

Buildroot

• Buildroot includes PyPy / PyPy3 as packaged, cross-compiled targets
• Works reliably on AArch64
• Integrated into the standard Buildroot package flow
• Clearly not the default Python runtime, but usable in production
• Fewer architectures and options than CPython, but sufficient for server-style workloads

Summary:
Buildroot currently offers the best out-of-the-box PyPy story for embedded / edge ARM64 systems.

---

Yocto / OpenEmbedded

• PyPy is not part of the core Yocto layers

• Available only via external / community layers (e.g. meta-pypy)

• Recipes are often:

  • Lagging behind upstream PyPy
  • Fragile across Yocto releases
  • Inconsistently maintained

• Cross-compiling PyPy in Yocto remains complex due to PyPy’s bootstrap and JIT build model

Summary:
PyPy support in Yocto is experimental and unofficial, which is a real limitation for Python-heavy, long-running services.

---

Debian (ARM64)

• PyPy is a first-class, distro-supported package
• Very robust on AArch64
• Easiest platform from a packaging and maintenance perspective
• Heavier footprint and less control than Buildroot / Yocto

Summary:
Debian provides the most mature PyPy experience, but at the cost of larger system size and less tailoring.

---

2. Why PyPy is hugely relevant for Crossbar.io on ARM64

Real-world measurements with Crossbar.io on ARM64 show PyPy significantly outperforming CPython, often by a wide margin. This is not accidental and not limited to JIT effects.

Key reasons PyPy excels for Crossbar.io

a) Incremental / generational GC (not just JIT)

• Crossbar.io is:

  • Long-running
  • Highly async
  • Extremely allocation-heavy (messages, frames, protocol objects)

• PyPy’s incremental GC:

  • Eliminates refcount overhead
  • Avoids stop-the-world GC spikes
  • Produces much smoother latency under load

For message routers and brokers, GC behavior matters as much as raw throughput.

b) Long-lived, stable hot paths

• Crossbar.io runs as a daemon
• JIT warm-up costs are amortized
• Hot loops stabilize quickly
• PyPy’s optimizer becomes highly effective

This is exactly the workload PyPy is designed for.

c) ARM64 benefits disproportionately

• Modern AArch64 PyPy JIT backend is mature and stable
• Removing refcount traffic helps ARM cores more than x86 in many cases
• The performance delta vs CPython is often larger on ARM64 than on x86

---

3. Why PyPy is often underestimated in “embedded Linux” discussions

PyPy’s reputation suffers because it is often evaluated using:

• Short-lived scripts
• CLI tools
• Memory-starved systems
• C-extension-heavy workloads

These are not representative of systems like Crossbar.io.

For edge / embedded network servers with sufficient RAM, PyPy can deliver:

• Higher throughput
• Lower and more predictable latency
• Better tail behavior under load

---

Conclusion

• PyPy on ARM64 is not niche for Crossbar.io-like workloads — it is technically well-aligned
• Buildroot currently offers the most practical embedded PyPy option
• Yocto’s lack of first-class PyPy support is a real gap for async-heavy Python services
• Debian offers the most mature PyPy packaging, at the cost of system size

For ARM64 network and messaging servers, PyPy is not just viable — it is often the better runtime choice.

---