add tokenizer model and tokenizer tests

Author: csabakecskemeti

models/ggml-vocab-mellum.gguf

@@ -0,0 +1,112 @@
+ied 4 ½ months
+__ggml_vocab_test__
+Äpfel
+__ggml_vocab_test__
+
+__ggml_vocab_test__
+ 
+__ggml_vocab_test__
+  
+__ggml_vocab_test__
+   
+__ggml_vocab_test__
+	
+__ggml_vocab_test__
+
+
+__ggml_vocab_test__
+
+
+
+__ggml_vocab_test__
+
+
+
+
+__ggml_vocab_test__
+	
+
+__ggml_vocab_test__
+Hello world
+__ggml_vocab_test__
+ Hello world
+__ggml_vocab_test__
+Hello World
+__ggml_vocab_test__
+ Hello World
+__ggml_vocab_test__
+ Hello World!
+__ggml_vocab_test__
+Hello, world!
+__ggml_vocab_test__
+ Hello, world!
+__ggml_vocab_test__
+ this is 🦙.cpp
+__ggml_vocab_test__
+w048 7tuijk dsdfhu
+__ggml_vocab_test__
+нещо на Български
+__ggml_vocab_test__
+កាន់តែពិសេសអាចខលចេញ
+__ggml_vocab_test__
+🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token)
+__ggml_vocab_test__
+Hello
+__ggml_vocab_test__
+ Hello
+__ggml_vocab_test__
+  Hello
+__ggml_vocab_test__
+   Hello
+__ggml_vocab_test__
+    Hello
+__ggml_vocab_test__
+    Hello
+    Hello
+__ggml_vocab_test__
+ (
+__ggml_vocab_test__
+
+ =
+__ggml_vocab_test__
+' era
+__ggml_vocab_test__
+Hello, y'all! How are you 😁 ?我想在apple工作1314151天～
+__ggml_vocab_test__
+!!!!!!
+__ggml_vocab_test__
+3
+__ggml_vocab_test__
+33
+__ggml_vocab_test__
+333
+__ggml_vocab_test__
+3333
+__ggml_vocab_test__
+33333
+__ggml_vocab_test__
+333333
+__ggml_vocab_test__
+3333333
+__ggml_vocab_test__
+33333333
+__ggml_vocab_test__
+333333333
+__ggml_vocab_test__
+Cửa Việt
+__ggml_vocab_test__
+ discards
+__ggml_vocab_test__
+
+ 
+
+ 
+
+
+ 	 		 	
+  
+   
+    
+     
+🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български ''''''```````""""......!!!!!!?????? I've been 'told he's there, 'RE you sure? 'M not sure I'll make it, 'D you like some tea? We'Ve a'lL
+__ggml_vocab_test__

models/ggml-vocab-mellum.gguf.inp

@@ -0,0 +1,46 @@
+ 1743 775 4588 148 9225
+ 49822 10221 332
+
+ 247
+ 247 247
+ 247 247 247
+ 224
+ 225
+ 225 225
+ 225 225 225
+ 2572
+ 10626 3822
+ 19762 3822
+ 10626 8098
+ 19762 8098
+ 19762 8098 27
+ 10626 38 3822 27
+ 19762 38 3822 27
+ 520 432 8892 126 274 40 2354
+ 113 18619 1345 42961 17444 6729 1714 4590
+ 30813 7682 834 8413 33028 13043 1719 6391 9791 29827
+ 33595 25149 31055 51151 43497 78271 66809 58451 37263 66436 37263 65068 25149 65088 5367 250 47929 65088 66436 90924
+ 61661 249 341 4111 35 26591 141 19340 68052 131 20565 341 15611 350 69572 72291 35 62765 341 5935 39662 569 1202 2027 4174 3961 35
+ 10626
+ 19762
+ 247 19762
+ 247 247 19762
+ 247 247 247 19762
+ 247 247 247 19762 225 247 247 247 19762
+ 341
+ 225 306
+ 33 24747
+ 10626 38 437 33 474 27 6863 748 691 26591 250 1364 91890 3038 10123 18854 45990 10408 10742 41579
+ 12525 3740
+ 45
+ 592
+ 9476
+ 1882
+ 34333
+ 3586
+ 33390
+ 17892
+ 17892 45
+ 61 40968 91 10788 79508
+ 936 16228
+ 225 493 225 493 225 225 34362 3787 2572 722 948 984 247 247 247 247 247 225 61661 249 341 4111 35 26591 141 19340 68052 131 20565 341 15611 350 69572 72291 35 62765 8892 126 274 4641 126 274 690 4875 49633 247 1882 247 34333 247 3586 247 33390 247 17892 690 40 45 690 502 45 690 1366 45 247 33595 25149 31055 51151 43497 78271 66809 58451 37263 66436 37263 65068 25149 65088 40837 250 1364 91890 3038 10123 18854 45990 10408 10742 41579 31166 21059 11927 7682 834 8413 33028 13043 1719 6391 9791 29827 3638 32208 21001 1305 17155 29835 12525 3740 22922 8061 391 6070 1962 382 298 1061 633 1151 1828 38 382 736 691 4076 57 382 71 654 4076 391 5906 1773 650 38 382 62 691 1893 1475 7939 57 2168 33 4314 283 33 102 70