add compatibility to huggingface interface

Author: jararap

.gitattributes

@@ -6,4 +6,6 @@ pcatt/slow_bpe.py export-ignore
 pcatt/greedy_cache.cpp export-ignore
 pcatt/max_cover.cpp export-ignore
 eval_notebook.ipynb export-ignore
-eval_tokenizer_example.ipynb export-ignore
+eval_tokenizer_example.ipynb export-ignore
+pcatt/hf/examples-ignore
+eval_hf.ipynb-ignore

README.md

@@ -2,7 +2,29 @@
 In this work, we formulate tokenization as an optimization objective, show that it is NP-hard via a simple reduction from vertex cover, and propose a polynomial-time greedy algorithm **GreedTok**.
 Our formulation naturally relaxes to the well-studied weighted maximum coverage problem which has a simple $(1 - 1/e)$-approximation greedy algorithm.
 
-To do: Huggingface AutoTokenizer interface
+### Beta: Huggingface AutoTokenizer interface
+
+For "training" either:
+```
+from pcatt.hf.greedtok import GreedTok
+greedtok = GreedTok().train_new_from_iterator(word_iterator, 100, max_token_length=5, min_word_count=1)
+```
+or
+```
+from pcatt.hf.greedtok import GreedTok
+greedtok = GreedTok().train_new_from_counts(word_count_dict, 100, max_token_length=5, min_word_count=1)
+```
+To use either:
+```
+from pcatt.hf.greedtok import GreedTok
+greedtok = GreedTok.from_pretrained(greedtok_file_directory)
+```
+or
+```
+import pcatt.hf
+greedtok = AutoTokenizer.from_pretrained("greedtok_file_directory")
+```
+Refer to [eval_hf.ipynb](https://github.com/PreferredAI/aoatt/blob/main/eval_hf.ipynb)
 
 ### GreedTok 
 1. If using python wrapper

eval_hf.ipynb

@@ -0,0 +1,13 @@
+from pcatt.hf.greedtok import GreedTok
+from transformers import PretrainedConfig, AutoConfig, AutoTokenizer
+
+
+class GreedTokConfig(PretrainedConfig):
+    model_type = "greedtok"
+
+    def __init__():
+        pass
+
+
+AutoConfig.register("greedtok", GreedTokConfig)
+AutoTokenizer.register(GreedTokConfig, GreedTok)

pcatt/__init__.py

@@ -0,0 +1,265 @@
+6161
+6262
+616263
+6263
+3132
+313233
+3334
+3c7061643e
+3c656f733e
+00
+01
+02
+03
+04
+05
+06
+07
+08
+09
+0a
+0b
+0c
+0d
+0e
+0f
+10
+11
+12
+13
+14
+15
+16
+17
+18
+19
+1a
+1b
+1c
+1d
+1e
+1f
+20
+21
+22
+23
+24
+25
+26
+27
+28
+29
+2a
+2b
+2c
+2d
+2e
+2f
+30
+31
+32
+33
+34
+35
+36
+37
+38
+39
+3a
+3b
+3c
+3d
+3e
+3f
+40
+41
+42
+43
+44
+45
+46
+47
+48
+49
+4a
+4b
+4c
+4d
+4e
+4f
+50
+51
+52
+53
+54
+55
+56
+57
+58
+59
+5a
+5b
+5c
+5d
+5e
+5f
+60
+61
+62
+63
+64
+65
+66
+67
+68
+69
+6a
+6b
+6c
+6d
+6e
+6f
+70
+71
+72
+73
+74
+75
+76
+77
+78
+79
+7a
+7b
+7c
+7d
+7e
+7f
+80
+81
+82
+83
+84
+85
+86
+87
+88
+89
+8a
+8b
+8c
+8d
+8e
+8f
+90
+91
+92
+93
+94
+95
+96
+97
+98
+99
+9a
+9b
+9c
+9d
+9e
+9f
+a0
+a1
+a2
+a3
+a4
+a5
+a6
+a7
+a8
+a9
+aa
+ab
+ac
+ad
+ae
+af
+b0
+b1
+b2
+b3
+b4
+b5
+b6
+b7
+b8
+b9
+ba
+bb
+bc
+bd
+be
+bf
+c0
+c1
+c2
+c3
+c4
+c5
+c6
+c7
+c8
+c9
+ca
+cb
+cc
+cd
+ce
+cf
+d0
+d1
+d2
+d3
+d4
+d5
+d6
+d7
+d8
+d9
+da
+db
+dc
+dd
+de
+df
+e0
+e1
+e2
+e3
+e4
+e5
+e6
+e7
+e8
+e9
+ea
+eb
+ec
+ed
+ee
+ef
+f0
+f1
+f2
+f3
+f4
+f5
+f6
+f7
+f8
+f9
+fa
+fb
+fc
+fd
+fe
+ff

pcatt/hf/__init__.py

@@ -0,0 +1,4 @@
+{
+  "eos_token": "<eos>",
+  "pad_token": "<pad>"
+}

pcatt/hf/examples/greedtok_test1/added_tokens.txt

@@ -0,0 +1,5 @@
+{
+  "clean_up_tokenization_spaces": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GreedTok"
+}