fix: convert_hf_to_gguf - change Jamba non-sentencepiece mode (tokeni… #16470
+59
−6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…zer.json) vocab construction
CISC
reviewed
Oct 8, 2025
Comment on lines
5915
to
5919
| def get_vocab_base_pre(self, tokenizer) -> str: | ||
| del tokenizer # unused | ||
|
|
||
| return "gpt-2" | ||
| return "default" | ||
|
|
There was a problem hiding this comment.
Remove this method, only for BPE.
Comment on lines
+5934
to
+5936
| assert max(vocab.values()) < vocab_size | ||
|
|
||
| tokpre = self.get_vocab_base_pre(tokenizer) |
There was a problem hiding this comment.
Suggested change
| assert max(vocab.values()) < vocab_size | |
| tokpre = self.get_vocab_base_pre(tokenizer) | |
| assert max(vocab.values()) < vocab_size |
| tokens.append(token) | ||
|
|
||
| self.gguf_writer.add_tokenizer_model("llama") | ||
| self.gguf_writer.add_tokenizer_pre(tokpre) |
There was a problem hiding this comment.
Suggested change
| self.gguf_writer.add_tokenizer_pre(tokpre) | |
| self.gguf_writer.add_tokenizer_pre("default") |
Comment on lines
+5975
to
+5976
| self.gguf_writer.add_token_list(tokens) | ||
| self.gguf_writer.add_token_types(toktypes) |
| self.gguf_writer.add_token_list(tokens) | ||
| self.gguf_writer.add_token_types(toktypes) | ||
|
|
||
| special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=True) |
There was a problem hiding this comment.
Suggested change
| special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=True) | |
| special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=False) |
Comment on lines
+5953
to
+5955
| token = tokenizer.decode( | ||
| tokenizer.encode(token, add_special_tokens=False) | ||
| ) |
There was a problem hiding this comment.
Suggested change
| token = tokenizer.decode( | |
| tokenizer.encode(token, add_special_tokens=False) | |
| ) | |
| token = tokenizer.decode(tokenizer.encode(token, add_special_tokens=False)) |
Comment on lines
+5957
to
+5959
| logger.info( | ||
| f"{repr(previous_token)} is encoded and decoded back to {repr(token)} using AutoTokenizer" | ||
| ) |
There was a problem hiding this comment.
Suggested change
| logger.info( | |
| f"{repr(previous_token)} is encoded and decoded back to {repr(token)} using AutoTokenizer" | |
| ) | |
| logger.info(f"{repr(previous_token)} is encoded and decoded back to {repr(token)} using AutoTokenizer") |
Comment on lines
+5961
to
+5963
| if added_tokens_decoder[i].special or self.does_token_look_special( | ||
| token | ||
| ): |
There was a problem hiding this comment.
Suggested change
| if added_tokens_decoder[i].special or self.does_token_look_special( | |
| token | |
| ): | |
| if added_tokens_decoder[i].special or self.does_token_look_special(token): |
Comment on lines
+5929
to
+5931
| tokenizer = AutoTokenizer.from_pretrained( | ||
| self.dir_model, trust_remote_code=True | ||
| ) |
There was a problem hiding this comment.
Suggested change
| tokenizer = AutoTokenizer.from_pretrained( | |
| self.dir_model, trust_remote_code=True | |
| ) | |
| tokenizer = AutoTokenizer.from_pretrained(self.dir_model) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The JambaModel implementation at
convert_hf_to_gguf.pywas incorrectly constructing its vocab using the gpt-2 tokenizer logic when no SentencePiece model was present (i.e., tokenizer.json path). Jamba actually uses a llama tokenizer, not gpt-2.This change updates the vocab build path to use the correct llama tokenizer for non-SentencePiece Jamba models. Also includes several small adjustments within the Jamba llama based tokenizer construction.
No changes are expected for other model types.
Testing
Verified with local conversion of Jamba GGUF model (tokenizer.json mode) and confirmed generated vocab matches the llama tokenizer layout. SentencePiece mode GGUF was also verified and it remains unaffected.