如何添加新的special token，诸如<user>,<assistant>这种token #1230
-
|
|
Beta Was this translation helpful? Give feedback.
Answered by
zRzRzRzRzRzRzR
May 24, 2024
Replies: 1 comment 1 reply
-
|
在这改 |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
这个config文件应该不是手动编写的吧，应该是生成的吧，我猜测是不是应该在训练的时候添加新token，训练完毕后，生成的这个配置文件里就有这个token了。但是具体的不知如何操作 |
Beta Was this translation helpful? Give feedback.
Answer selected by
zRzRzRzRzRzRzR
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
在这改
https://huggingface.co/THUDM/chatglm3-6b/blob/main/tokenizer_config.json
但是这没办法增加一个全新的 6开头的 比如 64790这种token，因为这设计到重新训练词表