How do the nlp.tokenizer and subword tokenizers interact?

[BramVanroy]

Looking at the config of en_core_web_trf, I was wondering how the set spaCy tokenizer works together with the subword tokenizer used for the language model.

In the case of en_core_web_trf uses RoBERTa, which uses BPE under the hood if I remember correctly. But the default spaCy tokenizer does not work with subword units AFAIK. Yet, there is this line in the [nlp] section

tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

So how does this work in practice? My guess is the LM outputs logits for each BPE token, they are then converted back into (word) token-based representation as per components.*.model.tok2vec.pooling = {"@layers":"reduce_mean.v1"}, so that the components can simply work with the (word) tokens.

If this is correct, how does spaCy know these alignments? (I know that HF Fast Tokenizers output this mapping, but I don't think the slow ones do.)

Can you confirm/correct? Thank you.

[polm]

Your understanding of how embeddings is converted is correct. spaCy generates the alignment from two sequences of tokens; see the aligner section in the docs for notes on that. The alignments are generated using the spacy-alignments package.

[BramVanroy]

It seems that the aligner is agnostic to which type of tokenization is done, that is pretty cool. I did some tests and it seems quite robust. E.g. it works well with WordPieces Goo ##d morn ##ing and inconsistent casing, too!

Thanks!

[polm]

The author wrote in more detail about it here:

https://gist.github.com/tamuhey/af6cbb44a703423556c32798e1e1b704