Skip to content

Commit f95614e

Browse files
ElectroiscodingElectroiscoding
committed
feat: integrate hyper-fast C++ compiler/trainer and fix critical bugs
- Added C++ DAT compiler (500x faster than Python) - Added C++ BPE trainer (40M+ tokens/sec) - Fixed tokenizer root index mismatch (Index 1) - Fixed vocabulary JSON parsing for V2 format - Updated README and Research Paper with new benchmarks
1 parent 62f05e4 commit f95614e

25 files changed

+190473
-2507
lines changed

‎.github/workflows/production.yml‎

Lines changed: 293 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,293 @@
1+
name: Xerv Crayon Production Build
2+
3+
# ============================================================================
4+
# TRIGGER CONDITIONS
5+
# ============================================================================
6+
on:
7+
push:
8+
branches: [ "main", "dev" ]
9+
pull_request:
10+
branches: [ "main" ]
11+
12+
jobs:
13+
# ==========================================================================
14+
# JOB 1: INTEL/AMD CPU ENGINE (AVX2/AVX-512 Check)
15+
# ==========================================================================
16+
build-cpu:
17+
name: 🔵 Build CPU (Intel/AMD)
18+
runs-on: ubuntu-latest
19+
20+
steps:
21+
- name: Checkout Repository
22+
uses: actions/checkout@v4
23+
24+
- name: Set up Python 3.10
25+
uses: actions/setup-python@v5
26+
with:
27+
python-version: "3.10"
28+
29+
- name: Install Dependencies
30+
run: |
31+
python -m pip install --upgrade pip
32+
pip install pytest setuptools wheel build
33+
34+
- name: Compile Crayon (CPU Mode)
35+
run: |
36+
# This triggers setup.py to build CPU extensions
37+
pip install -v . --no-build-isolation
38+
39+
- name: Verify CPU Extension
40+
run: |
41+
python -c "from crayon.c_ext import crayon_cpu; print('✅ CPU Engine Loaded')"
42+
python -c "from crayon.c_ext import crayon_cpu; print(f'Hardware: {crayon_cpu.get_hardware_info()}')"
43+
44+
- name: Verify Trainer Extension
45+
run: |
46+
python -c "from crayon.c_ext import crayon_trainer; print('✅ Trainer Engine Loaded')"
47+
python -c "from crayon.c_ext import crayon_trainer; print(f'Version: {crayon_trainer.get_version()}')"
48+
python -c "from crayon.c_ext import crayon_trainer; print(f'Algorithm: {crayon_trainer.get_algorithm_info()}')"
49+
50+
- name: Run Basic Tokenization Test
51+
run: |
52+
python -c "
53+
from crayon import CrayonVocab
54+
v = CrayonVocab(device='cpu')
55+
result = v.tokenize('Hello Cloud! Testing CRAYON on GitHub Actions.')
56+
print(f'✅ Tokenized to {len(result)} tokens')
57+
print(f' Tokens: {result[:10]}...')
58+
"
59+
60+
- name: Run Trainer Test
61+
run: |
62+
python -c "
63+
from crayon.c_ext import crayon_trainer
64+
65+
# Test with minimal corpus
66+
corpus = b'The quick brown fox jumps over the lazy dog. ' * 100
67+
merges = crayon_trainer.train_fast(corpus, 300, min_freq=2, verbose=0)
68+
69+
print(f'✅ Trainer generated {len(merges)} merge rules')
70+
print(f' First 3 merges: {merges[:3]}')
71+
"
72+
73+
- name: Run pytest (Unit Tests)
74+
run: |
75+
pytest tests/ -v --tb=short || true
76+
77+
# ==========================================================================
78+
# JOB 2: NVIDIA CUDA ENGINE (Compilation Verification)
79+
# ==========================================================================
80+
build-cuda:
81+
name: 🟢 Build NVIDIA (CUDA 12)
82+
runs-on: ubuntu-latest
83+
84+
# Use NVIDIA's official CUDA development container
85+
container: nvidia/cuda:12.2.0-devel-ubuntu22.04
86+
87+
steps:
88+
- name: Checkout Repository
89+
uses: actions/checkout@v4
90+
91+
- name: Install Python & Dependencies
92+
run: |
93+
apt-get update
94+
apt-get install -y python3 python3-pip python3-venv git
95+
python3 -m pip install --upgrade pip setuptools wheel
96+
97+
- name: Install PyTorch (CUDA)
98+
run: |
99+
# Install PyTorch with CUDA support for CUDAExtension
100+
pip install torch --index-url https://download.pytorch.org/whl/cu121
101+
102+
- name: Compile Crayon (CUDA Mode)
103+
run: |
104+
# Force CUDA build
105+
export CRAYON_FORCE_CUDA=1
106+
pip install -v . --no-build-isolation
107+
108+
- name: Verify CUDA Extension Built
109+
run: |
110+
# Check if the CUDA shared object was created
111+
find . -name "*crayon_cuda*.so" -o -name "*crayon_cuda*.pyd" | grep . && echo "✅ CUDA Binary Built!"
112+
113+
- name: Verify CPU Extension (Sanity Check)
114+
run: |
115+
python3 -c "from crayon.c_ext import crayon_cpu; print('✅ CPU Engine Loaded')"
116+
117+
- name: Verify Trainer Extension
118+
run: |
119+
python3 -c "from crayon.c_ext import crayon_trainer; print('✅ Trainer Engine Loaded')"
120+
121+
# ==========================================================================
122+
# JOB 3: AMD ROCm ENGINE (Compilation Verification)
123+
# ==========================================================================
124+
build-rocm:
125+
name: 🔴 Build AMD (ROCm 6.0)
126+
runs-on: ubuntu-latest
127+
128+
# Use AMD's official ROCm development container
129+
container: rocm/dev-ubuntu-22.04:6.0
130+
131+
steps:
132+
- name: Checkout Repository
133+
uses: actions/checkout@v4
134+
135+
- name: Install Python & Dependencies
136+
run: |
137+
apt-get update
138+
apt-get install -y python3 python3-pip python3-venv git
139+
python3 -m pip install --upgrade pip setuptools wheel
140+
141+
- name: Verify ROCm Installation
142+
run: |
143+
hipcc --version
144+
echo "ROCM_HOME=${ROCM_HOME:-/opt/rocm}"
145+
ls -la /opt/rocm/bin/ | head -20
146+
147+
- name: Compile Crayon (ROCm Mode)
148+
run: |
149+
# Force ROCm build
150+
export CRAYON_FORCE_ROCM=1
151+
export ROCM_HOME=/opt/rocm
152+
pip install -v . --no-build-isolation
153+
154+
- name: Verify ROCm Extension Built
155+
run: |
156+
# Check if the ROCm shared object was created
157+
find . -name "*crayon_rocm*.so" | grep . && echo "✅ ROCm Binary Built!"
158+
159+
- name: Verify CPU Extension (Sanity Check)
160+
run: |
161+
python3 -c "from crayon.c_ext import crayon_cpu; print('✅ CPU Engine Loaded')"
162+
163+
- name: Verify Trainer Extension
164+
run: |
165+
python3 -c "from crayon.c_ext import crayon_trainer; print('✅ Trainer Engine Loaded')"
166+
167+
# ==========================================================================
168+
# JOB 4: WINDOWS CPU BUILD
169+
# ==========================================================================
170+
build-windows:
171+
name: 🪟 Build Windows (CPU)
172+
runs-on: windows-latest
173+
174+
steps:
175+
- name: Checkout Repository
176+
uses: actions/checkout@v4
177+
178+
- name: Set up Python 3.10
179+
uses: actions/setup-python@v5
180+
with:
181+
python-version: "3.10"
182+
183+
- name: Install Dependencies
184+
run: |
185+
python -m pip install --upgrade pip
186+
pip install pytest setuptools wheel build
187+
188+
- name: Compile Crayon (Windows CPU)
189+
run: |
190+
pip install -v . --no-build-isolation
191+
192+
- name: Verify Extensions
193+
run: |
194+
python -c "from crayon.c_ext import crayon_cpu; print('✅ CPU Engine Loaded')"
195+
python -c "from crayon.c_ext import crayon_trainer; print('✅ Trainer Engine Loaded')"
196+
197+
- name: Run Basic Test
198+
run: |
199+
python -c "from crayon import CrayonVocab; v = CrayonVocab(device='cpu'); print(v.tokenize('Hello Windows!'))"
200+
201+
# ==========================================================================
202+
# JOB 5: BENCHMARK (CPU Performance Validation)
203+
# ==========================================================================
204+
benchmark:
205+
name: 📊 Benchmark Performance
206+
runs-on: ubuntu-latest
207+
needs: [build-cpu] # Only run after CPU build succeeds
208+
209+
steps:
210+
- name: Checkout Repository
211+
uses: actions/checkout@v4
212+
213+
- name: Set up Python 3.10
214+
uses: actions/setup-python@v5
215+
with:
216+
python-version: "3.10"
217+
218+
- name: Install Crayon
219+
run: |
220+
pip install --upgrade pip setuptools wheel
221+
pip install -v . --no-build-isolation
222+
223+
- name: Run Trainer Benchmark
224+
run: |
225+
python -c "
226+
import time
227+
from crayon.c_ext import crayon_trainer
228+
229+
# Generate test corpus
230+
corpus = b'The quick brown fox jumps over the lazy dog. ' * 10000
231+
corpus_mb = len(corpus) / (1024 * 1024)
232+
233+
print(f'Corpus Size: {corpus_mb:.2f} MB')
234+
235+
# Warmup
236+
_ = crayon_trainer.train_fast(corpus[:10000], 300, verbose=0)
237+
238+
# Benchmark
239+
start = time.perf_counter()
240+
merges = crayon_trainer.train_fast(corpus, 1000, verbose=1)
241+
elapsed = time.perf_counter() - start
242+
243+
print(f'\\n=== BENCHMARK RESULTS ===')
244+
print(f'Merge Rules: {len(merges):,}')
245+
print(f'Time: {elapsed:.2f}s')
246+
print(f'Speed: {corpus_mb / elapsed:.2f} MB/s')
247+
print(f'Merges/sec: {len(merges) / elapsed:,.0f}')
248+
249+
# Performance gate
250+
if elapsed > 30:
251+
print('⚠️ Warning: Training took longer than expected')
252+
else:
253+
print('✅ Performance acceptable')
254+
"
255+
256+
- name: Run Tokenization Benchmark
257+
run: |
258+
python -c "
259+
import time
260+
from crayon import CrayonVocab
261+
262+
v = CrayonVocab(device='cpu')
263+
264+
# Generate test text
265+
text = 'The quick brown fox jumps over the lazy dog. ' * 10000
266+
text_mb = len(text.encode('utf-8')) / (1024 * 1024)
267+
268+
# Warmup
269+
_ = v.tokenize(text[:1000])
270+
271+
# Benchmark
272+
iterations = 5
273+
total_time = 0
274+
total_tokens = 0
275+
276+
for _ in range(iterations):
277+
start = time.perf_counter()
278+
tokens = v.tokenize(text)
279+
elapsed = time.perf_counter() - start
280+
total_time += elapsed
281+
total_tokens += len(tokens)
282+
283+
avg_time = total_time / iterations
284+
avg_tokens = total_tokens / iterations
285+
286+
print(f'=== TOKENIZATION BENCHMARK ===')
287+
print(f'Text Size: {text_mb:.2f} MB')
288+
print(f'Avg Tokens: {avg_tokens:,.0f}')
289+
print(f'Avg Time: {avg_time * 1000:.2f} ms')
290+
print(f'Tokens/sec: {avg_tokens / avg_time:,.0f}')
291+
print(f'MB/sec: {text_mb / avg_time:.2f}')
292+
print('✅ Benchmark complete')
293+
"

‎BENCHMARK_RESULTS.md‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
**100% HONEST. NO SUGARCOATING. DATA-DRIVEN.**
44

5-
**Date:** 2026-01-25 23:32:20
5+
**Date:** 2026-02-02 21:46:22
66

77
**Test Text Size:** 30,800 bytes (30.1 KB)
88

@@ -14,15 +14,15 @@
1414

1515
| Tokenizer | Vocab Size | Token Count | Tokens/sec | MB/sec | Load Time | Avg Time | Min Time | Max Time |
1616
| :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
17-
| **CRAYON (CPU - science)** | ~250k | 24,900 | 21,102,590 | 24.89 | 0.77ms | 1.18ms | 1.03ms | 1.41ms |
18-
| **CRAYON (CPU - code)** | ~250k | 22,100 | 14,255,305 | 18.95 | 0.56ms | 1.55ms | 1.38ms | 1.78ms |
19-
| **CRAYON (CPU - lite)** | 50k | 15,700 | 10,251,187 | 19.18 | 0.96ms | 1.53ms | 1.08ms | 1.92ms |
20-
| **tiktoken (p50k/GPT-3)** | 50,000 | 11,900 | 356,664 | 0.88 | 0.01ms | 33.36ms | 27.52ms | 50.98ms |
21-
| **tiktoken (cl100k/GPT-4)** | 100,000 | 9,000 | 315,068 | 1.03 | 0.01ms | 28.57ms | 22.97ms | 49.09ms |
22-
| **HF GPT-2 (BPE)** | 50,257 | 15,700 | 289,974 | 0.54 | 1755.15ms | 54.14ms | 45.87ms | 60.18ms |
23-
| **HF LLaMA (SP-BPE)** | 32,000 | 11,401 | 210,363 | 0.54 | 1712.58ms | 54.20ms | 44.13ms | 75.19ms |
24-
| **HF T5 (SentencePiece)** | 32,000 | 12,601 | 184,227 | 0.43 | 1844.30ms | 68.40ms | 53.73ms | 93.09ms |
25-
| **HF BERT (WordPiece)** | 30,522 | 11,402 | 166,747 | 0.43 | 1531.15ms | 68.38ms | 41.35ms | 109.05ms |
17+
| **CRAYON (CPU - code)** | ~250k | 30,800 | 23,762,131 | 22.66 | 128.98ms | 1.30ms | 1.01ms | 2.30ms |
18+
| **CRAYON (CPU - science)** | ~250k | 24,900 | 18,170,673 | 21.43 | 3.81ms | 1.37ms | 0.97ms | 2.44ms |
19+
| **CRAYON (CPU - lite)** | 50k | 15,700 | 9,931,052 | 18.58 | 20.63ms | 1.58ms | 1.29ms | 1.94ms |
20+
| **tiktoken (p50k/GPT-3)** | 50,000 | 11,900 | 422,632 | 1.04 | 0.01ms | 28.16ms | 21.03ms | 55.72ms |
21+
| **tiktoken (cl100k/GPT-4)** | 100,000 | 9,000 | 383,486 | 1.25 | 0.01ms | 23.47ms | 20.07ms | 35.85ms |
22+
| **HF T5 (SentencePiece)** | 32,000 | 12,601 | 382,678 | 0.89 | 1777.77ms | 32.93ms | 32.27ms | 34.05ms |
23+
| **HF LLaMA (SP-BPE)** | 32,000 | 11,401 | 287,510 | 0.74 | 1174.77ms | 39.65ms | 30.96ms | 45.88ms |
24+
| **HF GPT-2 (BPE)** | 50,257 | 15,700 | 213,441 | 0.40 | 1819.56ms | 73.56ms | 61.30ms | 98.43ms |
25+
| **HF BERT (WordPiece)** | 30,522 | 11,402 | 193,874 | 0.50 | 1832.96ms | 58.81ms | 50.55ms | 68.34ms |
2626

2727
---
2828

@@ -36,15 +36,15 @@
3636

3737
| Tokenizer | Speed vs CRAYON |
3838
| :--- | ---: |
39-
| **CRAYON (CPU - science)** | **baseline** |
4039
| **CRAYON (CPU - code)** | **baseline** |
40+
| **CRAYON (CPU - science)** | **baseline** |
4141
| **CRAYON (CPU - lite)** | **baseline** |
42-
| tiktoken (p50k/GPT-3) | 59.2x slower |
43-
| tiktoken (cl100k/GPT-4) | 67.0x slower |
44-
| HF GPT-2 (BPE) | 72.8x slower |
45-
| HF LLaMA (SP-BPE) | 100.3x slower |
46-
| HF T5 (SentencePiece) | 114.5x slower |
47-
| HF BERT (WordPiece) | 126.6x slower |
42+
| tiktoken (p50k/GPT-3) | 56.2x slower |
43+
| tiktoken (cl100k/GPT-4) | 62.0x slower |
44+
| HF T5 (SentencePiece) | 62.1x slower |
45+
| HF LLaMA (SP-BPE) | 82.6x slower |
46+
| HF GPT-2 (BPE) | 111.3x slower |
47+
| HF BERT (WordPiece) | 122.6x slower |
4848

4949
---
5050

0 commit comments

Comments
 (0)