How to properly mock a spacy Vocab?

[DavidNemeskey]

Following this question on SO, I would like to create spacy Docs manually with a mock Vocab. However, when I create the following document:

doc = Doc(vocab, words=['This', 'is', 'a', 'sentence', '.'],
          spaces=[True, True, True, False, True],
          tags=['DT', 'VBZ', 'DT', 'NN', '.'],
          pos=['DET', 'AUX', 'DET', 'NOUN', 'PUNCT'],
          lemmas=['this', 'be', 'a', 'sentence', '.'],
          sent_starts=[True, False, False, False, False],
          heads=[1, 1, 3, 1, 1],
          deps=['nsubj', 'ROOT', 'det', 'attr', 'punct'])

, displacy visualizes the dependency graph in two different ways. If vocab is a mock object (e.g. vocab = Vocab(['This', 'is', 'a', 'sentence', '.']) or even vocab = Vocab()), the period is displayed as a separate node and it has a PUNCT edge leading to it:

However, if I use the English nlp's vocab, the punctuation mark is concatenated with the previous node and the edge is missing:

Having looked at the Lexemes in both, I see that the is_punct method of the "." lexeme in nlp.vocab returns True, but it returns False in the mock one. Could this be the reason why the two vocabularies behave differently? If so, is there a way I could set up my mock vocabulary in such a way that punctuation tokens are properly marked so? And if not, what may be the cause of the difference in behavior?

Thanks!

[polm]

The different behavior in displaCy is because it has a collapse_punct option, which is True by default, that merges punctuation with preceding tokens. So if the period is flagged as punctuation it will be merged.

The reason your periods are not getting flagged is that by calling the object constructor directly you are not getting any of the default functions for setting lex attributes. You can get them by using spacy.vocab.create_vocab instead, though I would recommend just stealing the Vocab from a blank English pipeline (spacy.blank("en")).

[DavidNemeskey]

Thank you, this was just what I needed. I was looking for a solution that works for any language, and doesn't require a model; create_vocab ticks both checkboxes. Too bad the documentation doesn't even mention the function, let alone explain how it should be called. So for others like me, who aren't privy to spacy internals, this is what I came up with:

from spacy.language import Language
from spacy.vocab import create_vocab

vocab = create_vocab('hu', Language.Defaults)

Then use this vocab to create Docs; tokens will be added automatically and the lexeme attributes (including is_punct) will work as expected.

[adrianeboyd]

This is close, but is missing the language-specific defaults, since most languages have some level of customizations beyond Language.Defaults. Like this you'd want:

from spacy.lang.hu import Hungarian
from spacy.vocab import create_vocab

vocab = create_vocab('hu', Hungarian.Defaults)

Like Paul suggested, spacy.blank() is easier given just the language code:

import spacy
nlp = spacy.blank("hu")
vocab = nlp.vocab

This does take a little bit of unneeded time to load the tokenizer settings, too, but is a lot simpler.

[DavidNemeskey]

Yeah, I should have tried spacy.blank :) I had a quick glance at the code, saw that it raises an ImportError, so I figured it would do that if a model of the language is not available. Apparently not. Thanks for the update!

BTW: is there a way to go from 'hu' to Hungarian? I know that Hungarian.lang == 'hu', but what about the other direction?

[adrianeboyd]

There's no mapping within spacy, no, so it is a hassle, which is part of why I always use spacy.blank("en") instead of English() (they do the exact same thing). With the en shortcut for en_core_web_sm in spacy v2, we tried to use English() in the docs to make the distinction between the pretrained pipeline and a blank pipeline clear, but now there should be less confusion.

There's a library that can handle a lot of these mappings (the author just recently joined the Explosion team, so maybe some of the internal language code handling can be improved in the future, too): https://github.com/rspeer/langcodes