English Sentenciser - Acronyms

[SpyriP]

I was running some examples through tokenisation, such as:

nlp.tokenizer.explain("U.S.")
[('TOKEN', 'U.S.')]

I understand that "U.S." should be one token as shown above. Since this is the case, why isn't it inside special cases in
tokenizer exceptions: https://github.com/explosion/spaCy/blame/master/spacy/lang/en/tokenizer_exceptions.py. Is there any specific exception for this?

Another similar case is the single-letter followed by dot, like: "B."
In the special cases, we have lowercase single letters, like: "a." "b." etc but there are no uppercases. Since the latter are not included in the special cases in the tokeniser, then why do we get the following output ? Am I missing something?

nlp.tokenizer.explain("He is working on the plan B.")
[('TOKEN', 'He'), ('TOKEN', 'is'), ('TOKEN', 'working'), ('TOKEN', 'on'), ('TOKEN', 'the'), ('TOKEN', 'plan'), ('TOKEN', 'B.')]

[adrianeboyd]

TOKEN means that it's not from an exception (those say SPECIAL). There are suffix rules that don't split off . after certain patterns (single capital letters like middle initials are one of the exceptions), so the form is left as is as comes out as a token because nothing else was split off. TOKEN is what is left after all the patterns/rules are applied.