Skip to content

Commit a6e7273

Browse files
ElectroiscodingElectroiscoding
committed
benchmakr and paper
1 parent 5bd50c4 commit a6e7273

File tree

13 files changed

+546
-28
lines changed

13 files changed

+546
-28
lines changed

‎CRAYON_Research_Paper.tex‎

Lines changed: 20 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1245,11 +1245,10 @@ \section{Experimental Evaluation}
12451245

12461246
\subsection{Experimental Setup}
12471247

1248-
\textbf{Hardware Configuration:}
1248+
\textbf{Hardware:}
12491249
\begin{itemize}
1250-
\item CPU: AMD/Intel with AVX2 support
1251-
\item GPU: NVIDIA Tesla T4 (Google Colab Pro)
1252-
\item Memory: 16GB system RAM, 16GB GPU VRAM
1250+
\item CPU: Intel Core i3-7020U @ 2.30GHz (4 logical cores, AVX2)
1251+
\item Memory: Windows 10, 16GB system RAM
12531252
\end{itemize}
12541253

12551254
\textbf{Baselines:}
@@ -1266,45 +1265,38 @@ \subsection{CPU Throughput Results}
12661265

12671266
\begin{table}[H]
12681267
\centering
1269-
\caption{CPU Tokenization Throughput}
1268+
\caption{CPU Tokenization Throughput (Intel i3-7020U @ 2.30GHz)}
12701269
\small
1271-
\begin{tabular}{@{}lr@{}}
1270+
\begin{tabular}{@{}lrr@{}}
12721271
\toprule
1273-
\textbf{Tokenizer} & \textbf{Tokens/sec} \\
1272+
\textbf{Tokenizer} & \textbf{Profile} & \textbf{Tokens/sec} \\
12741273
\midrule
1275-
Crayon (science) & \textbf{40,808,299} \\
1276-
Crayon (code) & 34,742,588 \\
1277-
Crayon (lite) & 18,453,654 \\
1274+
Crayon & lite & \textbf{5,216,527} \\
1275+
Crayon & standard & \textbf{2,335,773} \\
12781276
\midrule
1279-
tiktoken p50k & 719,614 \\
1280-
tiktoken cl100k & 608,610 \\
1281-
HF LLaMA & 343,282 \\
1282-
HF GPT-2 & 307,563 \\
1283-
HF T5 & 272,134 \\
1284-
HF BERT & 195,108 \\
1277+
tiktoken & p50k\_base & 284,002 \\
1278+
tiktoken & cl100k\_base & 263,125 \\
1279+
tiktoken & o200k\_base & 243,180 \\
12851280
\bottomrule
12861281
\end{tabular}
12871282
\label{tab:throughput}
12881283
\end{table}
12891284

1290-
Crayon achieves 40+ million tokens/second, exceeding all baselines by 1--2 orders of magnitude.
1285+
Crayon achieves 5+ million tokens/second on consumer hardware, exceeding all baselines by 18--22$\times$.
12911286

12921287
\subsection{Competitive Speedup}
12931288

12941289
\begin{table}[H]
12951290
\centering
1296-
\caption{Speedup vs Crayon (science)}
1291+
\caption{Speedup vs Crayon (lite profile)}
12971292
\small
12981293
\begin{tabular}{@{}lr@{}}
12991294
\toprule
13001295
\textbf{Tokenizer} & \textbf{Slowdown Factor} \\
13011296
\midrule
1302-
tiktoken p50k & 56.7$\times$ slower \\
1303-
tiktoken cl100k & 67.0$\times$ slower \\
1304-
HF LLaMA & 118.8$\times$ slower \\
1305-
HF GPT-2 & 132.6$\times$ slower \\
1306-
HF T5 & 149.9$\times$ slower \\
1307-
HF BERT & 209.1$\times$ slower \\
1297+
tiktoken p50k\_base & 18.4$\times$ slower \\
1298+
tiktoken cl100k\_base & 19.8$\times$ slower \\
1299+
tiktoken o200k\_base & 21.5$\times$ slower \\
13081300
\bottomrule
13091301
\end{tabular}
13101302
\label{tab:speedup}
@@ -1354,14 +1346,14 @@ \subsection{Future Directions}
13541346
\section{Conclusion}
13551347
\label{sec:conclusion}
13561348

1357-
We presented XERV Crayon, a production-grade tokenization system achieving over 40 million tokens per second through principled first-principles engineering. The combination of hyper-fast BPE training, cache-aligned Double-Array Tries, and hardware-native omni-backend acceleration establishes a new performance baseline for tokenization infrastructure.
1349+
We presented XERV Crayon, a production-grade tokenization system achieving over 5 million tokens per second through principled first-principles engineering. The combination of hyper-fast BPE training, cache-aligned Double-Array Tries, and hardware-native omni-backend acceleration establishes a new performance baseline for tokenization infrastructure.
13581350

1359-
The Cartridge System with five production profiles (lite, science, code, multilingual, arts\_commerce) enables domain-specific optimization without sacrificing the unified API. Adaptive vocabulary management with staged commit/rollback ensures graceful handling of distribution shift.
1351+
The Cartridge System with two production profiles (lite, standard) enables domain-specific optimization without sacrificing unified API.
13601352

13611353
Comprehensive benchmarks demonstrate:
13621354
\begin{itemize}
1363-
\item \textbf{56--118$\times$ speedup} over tiktoken and HuggingFace
1364-
\item \textbf{40+ million tokens/sec} on commodity CPU hardware
1355+
\item \textbf{18--22$\times$ speedup} over tiktoken on consumer hardware
1356+
\item \textbf{5+ million tokens/sec} on Intel i3-7020U CPU
13651357
\item \textbf{Sub-millisecond} profile loading via zero-copy mapping
13661358
\end{itemize}
13671359

‎benchmark_results/20260316_125203/benchmark_results.csv‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
impl,case,status,load_time_ms,tokens_produced,bytes_processed,avg_time_ms,tokens_per_sec,mb_per_sec,notes
2+
crayon:cpu:lite,english,OK,97.45750017464161,144001,740000,27.60476681093375,5216526.587102478,25.565113408641526,
3+
crayon:cpu:lite,code,OK,31.84089995920658,556000,1048000,128.8694003596902,4314445.465317106,7.755531420214274,
4+
crayon:cpu:lite,unicode,OK,57.07800015807152,474000,660000,181.54360043505827,2610943.0399313862,3.4670737350132197,
5+
crayon:cpu:lite,mixed,OK,25.56310035288334,640000,1397500,213.41429992268482,2998861.839304384,6.244941682261044,
6+
crayon:cpu:standard,english,OK,260.6979003176093,136001,740000,58.22526663541794,2335772.9016782506,12.120493986906745,
7+
crayon:cpu:standard,code,OK,289.5730007439852,312000,1048000,169.16560009121895,1844346.6037525397,5.908120108667586,
8+
crayon:cpu:standard,unicode,OK,142.52520073205233,216001,660000,31.140000248948734,6936448.242555555,20.212750282472268,
9+
crayon:cpu:standard,mixed,OK,207.64579996466637,372501,1397500,66.71659989903371,5583333.09197004,19.976435537702475,
10+
tiktoken:p50k_base,english,OK,347.60630037635565,144001,740000,507.0427001143495,284001.72207887925,1.391833456987882,
11+
tiktoken:p50k_base,code,OK,0.01769978553056717,420000,1048000,2520.244666685661,166650.48657848613,0.39656891126690247,
12+
tiktoken:p50k_base,unicode,OK,0.015099532902240753,378000,660000,930.4823335260153,406240.92084326665,0.6764502948088772,
13+
tiktoken:p50k_base,mixed,OK,0.006899237632751465,517500,1397500,2307.905833236873,224229.2525749191,0.5774758389117282,
14+
tiktoken:cl100k_base,english,OK,760.9403003007174,140001,740000,532.0695002252857,263125.3998598334,1.3263661868267467,
15+
tiktoken:cl100k_base,code,OK,0.0064997002482414246,308000,1048000,1633.9926331614454,188495.3418694932,0.6116616827457876,
16+
tiktoken:cl100k_base,unicode,OK,0.00690016895532608,306000,660000,908.9693001781901,336645.0329400709,0.6924601839740192,
17+
tiktoken:cl100k_base,mixed,OK,0.006599351763725281,410000,1397500,1150.4462001224358,356383.46230911615,1.1584721276283028,
18+
tiktoken:o200k_base,english,OK,1201.8392998725176,140001,740000,575.7089667022228,243180.16236911158,1.2258259554009134,
19+
tiktoken:o200k_base,code,OK,0.007599592208862305,312000,1048000,1332.6717000454664,234116.17429060405,0.7499601616509542,
20+
tiktoken:o200k_base,unicode,OK,0.0061998143792152405,210000,660000,643.3313333739837,326425.88524119346,0.9783839464604863,
21+
tiktoken:o200k_base,mixed,OK,0.005499459803104401,372500,1397500,1549.1690002381802,240451.49363479984,0.8603063042010436,

‎benchmark_results/20260316_125203/benchmark_results.json‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
[
2+
{
3+
"impl": "crayon:cpu:lite",
4+
"case": "english",
5+
"status": "OK",
6+
"load_time_ms": 97.45750017464161,
7+
"tokens_produced": 144001,
8+
"bytes_processed": 740000,
9+
"avg_time_ms": 27.60476681093375,
10+
"tokens_per_sec": 5216526.587102478,
11+
"mb_per_sec": 25.565113408641526,
12+
"notes": ""
13+
},
14+
{
15+
"impl": "crayon:cpu:lite",
16+
"case": "code",
17+
"status": "OK",
18+
"load_time_ms": 31.84089995920658,
19+
"tokens_produced": 556000,
20+
"bytes_processed": 1048000,
21+
"avg_time_ms": 128.8694003596902,
22+
"tokens_per_sec": 4314445.465317106,
23+
"mb_per_sec": 7.755531420214274,
24+
"notes": ""
25+
},
26+
{
27+
"impl": "crayon:cpu:lite",
28+
"case": "unicode",
29+
"status": "OK",
30+
"load_time_ms": 57.07800015807152,
31+
"tokens_produced": 474000,
32+
"bytes_processed": 660000,
33+
"avg_time_ms": 181.54360043505827,
34+
"tokens_per_sec": 2610943.0399313862,
35+
"mb_per_sec": 3.4670737350132197,
36+
"notes": ""
37+
},
38+
{
39+
"impl": "crayon:cpu:lite",
40+
"case": "mixed",
41+
"status": "OK",
42+
"load_time_ms": 25.56310035288334,
43+
"tokens_produced": 640000,
44+
"bytes_processed": 1397500,
45+
"avg_time_ms": 213.41429992268482,
46+
"tokens_per_sec": 2998861.839304384,
47+
"mb_per_sec": 6.244941682261044,
48+
"notes": ""
49+
},
50+
{
51+
"impl": "crayon:cpu:standard",
52+
"case": "english",
53+
"status": "OK",
54+
"load_time_ms": 260.6979003176093,
55+
"tokens_produced": 136001,
56+
"bytes_processed": 740000,
57+
"avg_time_ms": 58.22526663541794,
58+
"tokens_per_sec": 2335772.9016782506,
59+
"mb_per_sec": 12.120493986906745,
60+
"notes": ""
61+
},
62+
{
63+
"impl": "crayon:cpu:standard",
64+
"case": "code",
65+
"status": "OK",
66+
"load_time_ms": 289.5730007439852,
67+
"tokens_produced": 312000,
68+
"bytes_processed": 1048000,
69+
"avg_time_ms": 169.16560009121895,
70+
"tokens_per_sec": 1844346.6037525397,
71+
"mb_per_sec": 5.908120108667586,
72+
"notes": ""
73+
},
74+
{
75+
"impl": "crayon:cpu:standard",
76+
"case": "unicode",
77+
"status": "OK",
78+
"load_time_ms": 142.52520073205233,
79+
"tokens_produced": 216001,
80+
"bytes_processed": 660000,
81+
"avg_time_ms": 31.140000248948734,
82+
"tokens_per_sec": 6936448.242555555,
83+
"mb_per_sec": 20.212750282472268,
84+
"notes": ""
85+
},
86+
{
87+
"impl": "crayon:cpu:standard",
88+
"case": "mixed",
89+
"status": "OK",
90+
"load_time_ms": 207.64579996466637,
91+
"tokens_produced": 372501,
92+
"bytes_processed": 1397500,
93+
"avg_time_ms": 66.71659989903371,
94+
"tokens_per_sec": 5583333.09197004,
95+
"mb_per_sec": 19.976435537702475,
96+
"notes": ""
97+
},
98+
{
99+
"impl": "tiktoken:p50k_base",
100+
"case": "english",
101+
"status": "OK",
102+
"load_time_ms": 347.60630037635565,
103+
"tokens_produced": 144001,
104+
"bytes_processed": 740000,
105+
"avg_time_ms": 507.0427001143495,
106+
"tokens_per_sec": 284001.72207887925,
107+
"mb_per_sec": 1.391833456987882,
108+
"notes": ""
109+
},
110+
{
111+
"impl": "tiktoken:p50k_base",
112+
"case": "code",
113+
"status": "OK",
114+
"load_time_ms": 0.01769978553056717,
115+
"tokens_produced": 420000,
116+
"bytes_processed": 1048000,
117+
"avg_time_ms": 2520.244666685661,
118+
"tokens_per_sec": 166650.48657848613,
119+
"mb_per_sec": 0.39656891126690247,
120+
"notes": ""
121+
},
122+
{
123+
"impl": "tiktoken:p50k_base",
124+
"case": "unicode",
125+
"status": "OK",
126+
"load_time_ms": 0.015099532902240753,
127+
"tokens_produced": 378000,
128+
"bytes_processed": 660000,
129+
"avg_time_ms": 930.4823335260153,
130+
"tokens_per_sec": 406240.92084326665,
131+
"mb_per_sec": 0.6764502948088772,
132+
"notes": ""
133+
},
134+
{
135+
"impl": "tiktoken:p50k_base",
136+
"case": "mixed",
137+
"status": "OK",
138+
"load_time_ms": 0.006899237632751465,
139+
"tokens_produced": 517500,
140+
"bytes_processed": 1397500,
141+
"avg_time_ms": 2307.905833236873,
142+
"tokens_per_sec": 224229.2525749191,
143+
"mb_per_sec": 0.5774758389117282,
144+
"notes": ""
145+
},
146+
{
147+
"impl": "tiktoken:cl100k_base",
148+
"case": "english",
149+
"status": "OK",
150+
"load_time_ms": 760.9403003007174,
151+
"tokens_produced": 140001,
152+
"bytes_processed": 740000,
153+
"avg_time_ms": 532.0695002252857,
154+
"tokens_per_sec": 263125.3998598334,
155+
"mb_per_sec": 1.3263661868267467,
156+
"notes": ""
157+
},
158+
{
159+
"impl": "tiktoken:cl100k_base",
160+
"case": "code",
161+
"status": "OK",
162+
"load_time_ms": 0.0064997002482414246,
163+
"tokens_produced": 308000,
164+
"bytes_processed": 1048000,
165+
"avg_time_ms": 1633.9926331614454,
166+
"tokens_per_sec": 188495.3418694932,
167+
"mb_per_sec": 0.6116616827457876,
168+
"notes": ""
169+
},
170+
{
171+
"impl": "tiktoken:cl100k_base",
172+
"case": "unicode",
173+
"status": "OK",
174+
"load_time_ms": 0.00690016895532608,
175+
"tokens_produced": 306000,
176+
"bytes_processed": 660000,
177+
"avg_time_ms": 908.9693001781901,
178+
"tokens_per_sec": 336645.0329400709,
179+
"mb_per_sec": 0.6924601839740192,
180+
"notes": ""
181+
},
182+
{
183+
"impl": "tiktoken:cl100k_base",
184+
"case": "mixed",
185+
"status": "OK",
186+
"load_time_ms": 0.006599351763725281,
187+
"tokens_produced": 410000,
188+
"bytes_processed": 1397500,
189+
"avg_time_ms": 1150.4462001224358,
190+
"tokens_per_sec": 356383.46230911615,
191+
"mb_per_sec": 1.1584721276283028,
192+
"notes": ""
193+
},
194+
{
195+
"impl": "tiktoken:o200k_base",
196+
"case": "english",
197+
"status": "OK",
198+
"load_time_ms": 1201.8392998725176,
199+
"tokens_produced": 140001,
200+
"bytes_processed": 740000,
201+
"avg_time_ms": 575.7089667022228,
202+
"tokens_per_sec": 243180.16236911158,
203+
"mb_per_sec": 1.2258259554009134,
204+
"notes": ""
205+
},
206+
{
207+
"impl": "tiktoken:o200k_base",
208+
"case": "code",
209+
"status": "OK",
210+
"load_time_ms": 0.007599592208862305,
211+
"tokens_produced": 312000,
212+
"bytes_processed": 1048000,
213+
"avg_time_ms": 1332.6717000454664,
214+
"tokens_per_sec": 234116.17429060405,
215+
"mb_per_sec": 0.7499601616509542,
216+
"notes": ""
217+
},
218+
{
219+
"impl": "tiktoken:o200k_base",
220+
"case": "unicode",
221+
"status": "OK",
222+
"load_time_ms": 0.0061998143792152405,
223+
"tokens_produced": 210000,
224+
"bytes_processed": 660000,
225+
"avg_time_ms": 643.3313333739837,
226+
"tokens_per_sec": 326425.88524119346,
227+
"mb_per_sec": 0.9783839464604863,
228+
"notes": ""
229+
},
230+
{
231+
"impl": "tiktoken:o200k_base",
232+
"case": "mixed",
233+
"status": "OK",
234+
"load_time_ms": 0.005499459803104401,
235+
"tokens_produced": 372500,
236+
"bytes_processed": 1397500,
237+
"avg_time_ms": 1549.1690002381802,
238+
"tokens_per_sec": 240451.49363479984,
239+
"mb_per_sec": 0.8603063042010436,
240+
"notes": ""
241+
}
242+
]

‎benchmark_results/20260316_125203/load_time_ms.png‎

63.4 KB
Loading

‎benchmark_results/20260316_125203/mb_per_sec.png‎

56.4 KB
Loading

‎benchmark_results/20260316_125203/tokens_per_sec.png‎

58 KB
Loading

‎benchmark_results/20260316_125203/tokens_produced.png‎

68.2 KB
Loading

‎current_benchmark.json/benchmark_results.csv‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
impl,case,status,load_time_ms,tokens_produced,bytes_processed,avg_time_ms,tokens_per_sec,mb_per_sec,notes
2+
crayon:cpu:lite,english,OK,293.8474006950855,144001,740000,188.5325202718377,763799.2628134954,3.743221557337038,
3+
crayon:cpu:lite,code,OK,53.46729978919029,556000,1048000,137.21070028841496,4052162.104203942,7.284057886833307,
4+
crayon:cpu:lite,unicode,OK,34.77140050381422,474000,660000,85.59199962764978,5537900.762478252,7.353783666304187,
5+
crayon:cpu:lite,mixed,OK,44.43740006536245,640000,1397500,214.90477975457907,2978063.1251239693,6.201629664541404,
6+
crayon:cpu:standard,english,OK,256.83970004320145,136001,740000,45.62829993665218,2980628.25458797,15.466694904706209,
7+
crayon:cpu:standard,code,OK,373.47840052098036,312000,1048000,105.5902199819684,2954819.111592722,9.465371733901357,
8+
crayon:cpu:standard,unicode,OK,289.2025001347065,216001,660000,32.941219955682755,6557164.558282767,19.107520901621665,
9+
crayon:cpu:standard,mixed,OK,161.32330056279898,372501,1397500,204.93176002055407,1817683.115406998,6.503432445239636,
10+
tiktoken:p50k_base,english,OK,971.7505006119609,144001,740000,961.8989199399948,149704.91910832282,0.7336727170716121,
11+
tiktoken:p50k_base,code,OK,0.019999220967292786,420000,1048000,2896.1911803111434,145018.0508991394,0.34509140501090024,
12+
tiktoken:p50k_base,unicode,OK,0.013899989426136017,378000,660000,1686.671899817884,224109.97659996233,0.3731757485828075,
13+
tiktoken:p50k_base,mixed,OK,0.00980030745267868,517500,1397500,5820.50839997828,88909.75915470393,0.22897653702942988,
14+
tiktoken:cl100k_base,english,OK,1833.9411998167634,140001,740000,1069.1874599084258,130941.49085137124,0.6600516940229253,
15+
tiktoken:cl100k_base,code,OK,0.006299465894699097,308000,1048000,2401.132099889219,128272.82597830008,0.4162414402938771,
16+
tiktoken:cl100k_base,unicode,OK,0.010799616575241089,306000,660000,1080.0928801298141,283308.96872796945,0.5827508544936206,
17+
tiktoken:cl100k_base,mixed,OK,0.0112997367978096,410000,1397500,2310.1005997508764,177481.4482296636,0.576927194132351,
18+
tiktoken:o200k_base,english,OK,2667.899999767542,140001,740000,881.4683999866247,158827.0209143338,0.8006174630325188,
19+
tiktoken:o200k_base,code,OK,0.0061998143792152405,312000,1048000,1776.3976402580738,175636.35130402044,0.5626277928676771,
20+
tiktoken:o200k_base,unicode,OK,0.00509992241859436,210000,660000,647.867819853127,324140.192744266,0.971533126881988,
21+
tiktoken:o200k_base,mixed,OK,0.006099231541156769,372500,1397500,1556.8332800641656,239267.75253972426,0.8560710220190078,

0 commit comments

Comments
 (0)