When I finish lora training and run 'python chat.py',
Traceback (most recent call last):
File "/root/data1/ym/RWKV-LM/RWKV-LM-LoRA/RWKV-v4neo/chat.py", line 28, in
tokenizer = TOKENIZER(WORD_NAME, UNKNOWN_CHAR=UNKNOWN_CHAR)
File "/root/data1/ym/RWKV-LM/RWKV-LM-LoRA/RWKV-v4neo/src/utils.py", line 22, in init
self.tokenizer = PreTrainedTokenizerFast(tokenizer_file=WORD_NAME[0])
File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py", line 111, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: invalid type: integer 1, expected struct Tokenizer at line 1 column 1
When I finish lora training and run 'python chat.py',
Traceback (most recent call last):
File "/root/data1/ym/RWKV-LM/RWKV-LM-LoRA/RWKV-v4neo/chat.py", line 28, in
tokenizer = TOKENIZER(WORD_NAME, UNKNOWN_CHAR=UNKNOWN_CHAR)
File "/root/data1/ym/RWKV-LM/RWKV-LM-LoRA/RWKV-v4neo/src/utils.py", line 22, in init
self.tokenizer = PreTrainedTokenizerFast(tokenizer_file=WORD_NAME[0])
File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py", line 111, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: invalid type: integer
1, expected struct Tokenizer at line 1 column 1