Skip to content

Commit 3f3c746

Browse files
CopilotericstjCopilot
authored
Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json (#7625)
* Initial plan * Add public SentencePieceTokenizer.Create(vocab) and CreateFromTokenizerJson APIs Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com> * Fix BOS/EOS positional fallback, normalizer type validation, and prepend_scheme handling Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com> * Fix null normalizer guard, recursive Sequence support, BOS/EOS validation, and add tests Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com> * Generalize SentencePiece special tokens to prefix/suffix lists with post-processor support Support Hugging Face tokenizer.json post-processor templates that wrap a sequence with multiple special tokens (XLNet/mBART/NLLB styles) by generalizing the single BOS/EOS handling into ordered prefix/suffix special-token lists, applied consistently across the encode/count/index paths. CreateFromTokenizerJson now: - parses post_processor (TemplateProcessing, RobertaProcessing, BertProcessing, Sequence) to resolve the prefix/suffix wrapping, resolving ids via special_tokens -> added_tokens -> vocab. - reads model.byte_fallback and exposes byteFallback on Create. - treats added_tokens (special) as the authoritative special-token source. - tolerates unmodeled sibling steps inside a Sequence normalizer. - deduces remove_extra_whitespaces from the normalizer Strip/Replace(" {2,}") steps instead of hardcoding it, matching the Hugging Face fast-tokenizer runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support JSON-only Unigram normalizer chains and broaden tokenizer.json coverage Apply the Hugging Face tokenizer.json normalizer chain when loading JSON-only SentencePiece Unigram tokenizers whose normalizer has content-modifying steps (per-character Replace, Lowercase, StripAccents, NFC/NFD/NFKC/NFKD, Nmt, Prepend), running it before the Metaspace pass to match the reference order. Bare-charsmap models keep their existing fast path; only chain-mode models run the chain, which is a simple string pipeline (the boundary decode/encode use pooled buffers). Also broadens coverage of real models: - Infer Unigram when model.type is absent (older xlm-roberta-base/albert files) unless BPE merges are present. - Treat a WhitespaceSplit pre-tokenizer as remove_extra_whitespaces. Validated against a Hugging Face oracle: 41 JSON-only Unigram models load with no failures and tokenize identically to the reference except two documented cases (per-character vs merged unk for out-of-vocabulary runs, and trailing/leading whitespace exactness on whitespace-collapsing Replace models). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review: byte_fallback init, Metaspace replacement, doc - Set MaxByteId in the vocab-based Unigram constructor (and validate a contiguous <0x00>..<0xFF> block) so byte_fallback decode works instead of being inert, and fix a Debug.Assert that wrongly forbade unknown tokens on the byte-fallback path. - Reject a Metaspace "replacement" other than U+2581 with NotSupportedException rather than silently disabling whitespace escaping. - Correct the CreateFromTokenizerJson remarks: remove_extra_whitespaces is deduced (defaulting false), not assumed true. Adds byte-fallback round-trip and validation tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 2: Nmt, processor affix validation, cleanup - Apply Nmt even when it is the only normalizer step (HasRichSteps no longer treats Nmt as non-rich), so format/control characters are normalized and the behavior matches the documented step list. - Validate the RobertaProcessing/BertProcessing cls/sep [token, id] pairs against the vocabulary and added tokens, rejecting inconsistent files. - Drop the unused ref treatWhitespaceAsSuffix parameter from ExtractMetaspaceSettings. Adds standalone-Nmt and inconsistent-processor-affix tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 3: clarify charsmap comment, label missing normalizer type - Reword the ExtractPrecompiledCharsMap Sequence comment: it is reached only for non-rich Sequences (rich steps such as Nmt route through the managed chain), so it no longer implies those steps are silently ignored. - Render a missing/null normalizer type as <missing> in the NotSupportedException message instead of an empty string. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 4: byte-offset gating, base64 + unk_id validation - Only initialize the byte-piece offsets (ByteCodeToIdOffset and the derived OneByteUtf8EncodingMaxId/MaxIdByteFallbackId) when byte_fallback is enabled, so a vocab that merely contains <0xNN> pieces with byte_fallback:false decodes normal low ids instead of dropping them as byte-fallback pieces. - Decode a malformed base64 precompiled_charsmap into InvalidDataException via a shared DecodePrecompiledCharsMap helper used by both the chain and charsmap paths. - Validate model.unk_id against the parsed vocab length and throw InvalidDataException for an out-of-range value. Adds byte-fallback-disabled, malformed-charsmap, and out-of-range-unk_id tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 5: validate unk_id and vocab entry value kinds - Require model.unk_id to be a JSON number, throwing InvalidDataException with a targeted message instead of letting GetInt32 raise InvalidOperationException. - Require each model.vocab entry to be a [string piece, number score] pair so a missing/non-numeric score fails diagnostically rather than via a cast error. Adds non-numeric unk_id and non-numeric vocab-score tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 6: validate JSON value kinds across the loader Harden the tokenizer.json parser so malformed inputs fail with a diagnostic InvalidDataException instead of an InvalidOperationException/ArgumentException escaping from CreateFromTokenizerJson: - added_tokens: require string content and numeric id. - post_processor TemplateProcessing: require a string SpecialToken.id and a numeric special_tokens ids[0]. - RobertaProcessing/BertProcessing cls/sep: check the [token, id] value kinds before reading. - Metaspace pre_tokenizer: require a boolean add_prefix_space and a string replacement. - Replace normalizer: require an object pattern with a string String/Regex, and wrap invalid Regex construction as InvalidDataException. - Read all normalizer/pre_tokenizer/post_processor 'type' fields through a shared GetStringOrNull helper so a non-string type is handled in a controlled way rather than throwing. Adds tests covering the new validation paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 7: validate template special-token id consistency ResolveTemplateTokenId now verifies that a post_processor special_tokens id maps back to the referenced token (via added tokens or the vocabulary), mirroring the RobertaProcessing/BertProcessing affix validation, so an inconsistent tokenizer.json fails with InvalidDataException instead of emitting an id whose decoded token differs from the template's token name. Adds an inconsistent-template-special-token test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 8: remove redundant casts in tests CreateFromTokenizerJson returns SentencePieceTokenizer, so the explicit casts in the new tests were unnecessary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 9: guard normalizer Build value kinds - Require a normalizer entry to be a JSON object at the start of Build, so a non-object entry in a Sequence array fails with InvalidDataException instead of InvalidOperationException. - Validate the Precompiled 'precompiled_charsmap' is a string before reading it. - Validate the Prepend 'prepend' is a string before reading it. Adds tests for the non-object Sequence entry, non-string Precompiled charsmap, and non-string Prepend cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Tarek review: support null unk_id and reject unknown pre-tokenizers Two gaps flagged in review, since a full tokenizer.json is broader than the Unigram model block: - unk_id: accept a null unk_id (a Unigram model with no unknown token), which HF permits and the loader previously rejected. It is represented internally as -1, and the model constructor tolerates that (guards the unk re-insertion and GetPieceAtIndex). Because out-of-vocabulary input can then only be represented by byte fallback, a null unk_id without byte_fallback throws NotSupportedException rather than emitting an invalid id at encode time. A numeric out-of-range unk_id is still rejected as InvalidDataException. - pre_tokenizer: reject pre-tokenizer types the loader does not model (Punctuation, Split, Digits, ByteLevel, BertPreTokenizer, ...) with NotSupportedException, recursing into a Sequence, instead of silently ignoring and mis-tokenizing, mirroring the normalizer chain. The decoder section is still not read; documented and tracked separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ericstj <8918108+ericstj@users.noreply.github.com> Co-authored-by: Eric StJohn <ericstj@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 11ed3fe commit 3f3c746

6 files changed

Lines changed: 3085 additions & 25 deletions

File tree

‎src/Microsoft.ML.Tokenizers/Model/SentencePieceBaseModel.cs‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,50 @@ internal SentencePieceBaseModel(ModelProto modelProto, bool addBos = false, bool
5959
specialTokens);
6060
}
6161

62+
internal SentencePieceBaseModel(
63+
bool addBos, bool addEos,
64+
string bosToken, int bosId,
65+
string eosToken, int eosId,
66+
string unkToken, int unkId,
67+
bool addDummyPrefix, bool escapeWhiteSpaces,
68+
bool treatWhitespaceAsSuffix, bool byteFallback,
69+
ReadOnlySpan<byte> precompiledCharsmap, bool removeExtraWhitespaces,
70+
IReadOnlyDictionary<string, int>? specialTokens)
71+
{
72+
AddBeginningOfSentence = addBos;
73+
AddEndOfSentence = addEos;
74+
BeginningOfSentenceToken = bosToken;
75+
BeginningOfSentenceId = bosId;
76+
EndOfSentenceToken = eosToken;
77+
EndOfSentenceId = eosId;
78+
UnknownToken = unkToken;
79+
UnknownId = unkId;
80+
AddDummyPrefix = addDummyPrefix;
81+
EscapeWhiteSpaces = escapeWhiteSpaces;
82+
TreatWhitespaceAsSuffix = treatWhitespaceAsSuffix;
83+
ByteFallback = byteFallback;
84+
SpecialTokens = specialTokens;
85+
86+
if (specialTokens is not null && specialTokens.Count > 0)
87+
{
88+
InternalSpecialTokens = new Dictionary<StringSpanOrdinalKey, int>();
89+
SpecialTokensReverse = new Dictionary<int, string>();
90+
91+
foreach (var item in specialTokens)
92+
{
93+
InternalSpecialTokens.Add(new StringSpanOrdinalKey(item.Key), item.Value);
94+
SpecialTokensReverse.Add(item.Value, item.Key);
95+
}
96+
97+
SpecialTokensRegex = new Regex(string.Join("|", specialTokens.Keys.Select(s => Regex.Escape(s))), RegexOptions.Compiled);
98+
}
99+
100+
Normalizer = new SentencePieceNormalizer(
101+
precompiledCharsmap, removeExtraWhitespaces,
102+
addDummyPrefix, escapeWhiteSpaces,
103+
treatWhitespaceAsSuffix, specialTokens);
104+
}
105+
62106
internal Regex? SpecialTokensRegex { get; }
63107

64108
internal Dictionary<StringSpanOrdinalKey, int>? InternalSpecialTokens { get; }

0 commit comments

Comments
 (0)