Show and Tell: Project Zero - C99 BitNet b1.58 inference, 1.25x-1.83x over bitnet.cpp

[shifulegend]

Hi all - sharing performance data for Project Zero, a from-scratch C99 inference engine for BitNet b1.58.

What it is:

• C99, zero runtime deps (GCC + make, no Python, no BLAS)
• Runs BitNet b1.58-2B-4T and standard GGUF models in the same binary
• OpenAI-compatible HTTP API with SSE streaming
• Pre-built x86-64 Linux binary: releases

BitNet b1.58 performance:

AMD 9950X - benchmarked independently by @tommyyliu in #569, who ran Project Zero on his machine via Claude:

Threads	PZ (VNNI path)	PZ + K-6 LUT	Speedup
1	34.15 tok/s	42.66 tok/s	1.25x
2	47.43 tok/s	51.11 tok/s	1.08x
4	48.09 tok/s	51.93 tok/s	1.08x
16	52.96 tok/s	56.66 tok/s	1.07x

Intel Xeon 8259CL (AVX-512BW, no VNNI): T=1 gives 36.25 tok/s vs bitnet.cpp ~20 tok/s = 1.83x. Full Phoronix Test Suite result: https://openbenchmarking.org/result/2606207-SHIF-PROJECT42

K-6 LUT kernel:
The LUT numbers use 5-trit packing (5 ternary weights per byte) + AVX-512BW vpermt2w for register-based lookups - @tommyyliu's approach from lut_mm, integrated into PZ. Kernel-level on the no-VNNI Xeon: 74.6 vs 2.5 Gop/s = 29.3x. End-to-end gain is Amdahl-bounded but measurable: 36.25 vs expected 40+ tok/s once the K-6 branch is merged to master.

Honest limitation: Dense Q4_K GGUF: ~1.9 tok/s vs llama.cpp ~14 tok/s. Kernel investment has been on the BitNet ternary path so far.

Happy to answer questions or run benchmarks on specific hardware.

[shifulegend]

Update July 2: K-6 LUT integration (5-trit packing + AVX-512BW vpermt2w) is complete on the branch. Kernel numbers on the no-VNNI Xeon 8259CL confirmed at 74.6 vs 2.5 Gop/s float32 - 29.3x. End-to-end tok/s pending model weight availability on this machine for full validation.

Project Zero was also added to sihyeong/Awesome-LLM-Inference-Engine (PR #6, merged June 30) - a curated engine list backed by an ACM survey paper - alongside bitnet.cpp, llama.cpp, SGLang, and others.

If the optimization journey from 1.4 tok/s to the current numbers is useful reading: DRAM ceilings and what building a zero-dep C LLM engine taught me.

[shifulegend]

Follow-up on the 29.3x number above: a review of the PR (merged as #21) caught something I'd skipped - checking the DRAM bandwidth ceiling before getting excited about the kernel number. Turns out BitNet decode on that same Xeon box is already running at ~95% of it, so a much faster matmul kernel barely moves end-to-end tok/s once you're memory-bound instead of compute-bound. Honest math says ~6-10% real gain once the kernel is wired into dispatch - it isn't yet, still sitting there correctness-tested but unused.

Wanted to correct that here since it's the most visible place I'd posted the original number. The 36 tok/s baseline for BitNet b1.58-2B-4T on that Xeon is unaffected - this was specifically about how much K-6 adds on top of it, and the honest answer is "not as much as the isolated Gop/s number implied."