Skip to content

New model World run error, is it support new vocab 'rwkv_vocab_v20230424.txt'? #43

Description

@Helloworld2345567

When I finish lora training and run 'python chat.py',
Traceback (most recent call last):
File "/root/data1/ym/RWKV-LM/RWKV-LM-LoRA/RWKV-v4neo/chat.py", line 28, in
tokenizer = TOKENIZER(WORD_NAME, UNKNOWN_CHAR=UNKNOWN_CHAR)
File "/root/data1/ym/RWKV-LM/RWKV-LM-LoRA/RWKV-v4neo/src/utils.py", line 22, in init
self.tokenizer = PreTrainedTokenizerFast(tokenizer_file=WORD_NAME[0])
File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py", line 111, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: invalid type: integer 1, expected struct Tokenizer at line 1 column 1

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions