Replies: 14 comments 2 replies
-
|
补充一些内容， transformers 的一些相关代码 https://github.com/huggingface/transformers/blob/main/src/transformers/convert_slow_tokenizer.py 以下是 candle 支持 marian-mt 修改的 convert_slow_tokenizer.py 的代码 |
Beta Was this translation helpful? Give feedback.
-
|
@zRzRzRzRzRzRzR 这个 issue 是受理了还是？ |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
已经被标注，处理成了关闭，是我这边的问题，已重新打开，明天要做一下适配尝试 |
Beta Was this translation helpful? Give feedback.
-
感谢，正巧 Yi-6B 那边已经给了测试的 tokenizer.json 我测试过 ok, 技术上应该走的通 01-ai/Yi#24 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
收到请求，正在处理，有消息会第一时间回复 |
Beta Was this translation helpful? Give feedback.
-
|
不确定是否支持，可能要改 tokenizer.json。我们参照您最新提供的这份链接让算法同学在看看，回复较慢，望理解 |
Beta Was this translation helpful? Give feedback.
-
https://huggingface.co/THUDM/chatglm3-6b/discussions/12 |
Beta Was this translation helpful? Give feedback.
-
|
不是，我们看一下 |
Beta Was this translation helpful? Give feedback.
-
这个PR可能能够解决你提到的这个问题，算法工程师同事正在审核并验证是否能用，能用会合并 |
Beta Was this translation helpful? Give feedback.
-
恩，如果你们验证了，直接生成 tokenizer.json 提交在 hf 和 modelscope 呗，这样其他库就可以直接调用了 |
Beta Was this translation helpful? Give feedback.
-
对，目前该问题被记录为Bad Case，我们的开发人员正在做这个任务，该问题移交到讨论区 |
Beta Was this translation helpful? Give feedback.
-
|
不知道这块有新的进展么 |
Beta Was this translation helpful? Give feedback.
-
|
这份PR我们没有合并，但是可以自行尝试，由于人手不足，目前应该没有相关计划了 |
Beta Was this translation helpful? Give feedback.
-
|
@Liangdi Hi，这个 PR 我更新了一下，现在的 fast tokenizer 导出的 |
Beta Was this translation helpful? Give feedback.
-
|
感谢，我去试试 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
最近使用 candle , 在提交 chatglm 需求的时候遇到一个问题，candle 使用 https://github.com/huggingface/tokenizers 这个库， 使用时候需要一个 tokenizer.json , 在 chatglm 中没有这个文件，一些其他模型如：https://huggingface.co/bert-base-chinese ,https://huggingface.co/Salesforce/blip-image-captioning-large 等有相关支持。
看了一下 transformer 文档， 似乎是 fast-tokenziers 这个模块 https://huggingface.co/docs/transformers/fast_tokenizers
不知道 chatglm 是否能支持这个模块。
candle issue:
huggingface/candle#1177 (comment)
Beta Was this translation helpful? Give feedback.
All reactions