What is the best way to replace one word with one/more words

[antonpibm]

Hi

I wish to use SpaCy to analyze a bunch of sentences, the things is I want to explicitly transform abbreviations and common phrases into their full meaning, suppose I have the following dictionary:

{"nyc": "new york city", "philly": "philadelphia"}

I want to be able to transform words according to the dictionary when the language is applied on a sentence (sentences are deliberately lower case). e.g:

nlp("I want to move to nyc") would be equal to nlp("I want to move to new york city")
nlp("I went to philly for a vacation") would be equal to nlp("I went to philadelphia for a vacation")

Changing the Token's text is not possible (attribute 'text' of 'spacy.tokens.token.Token' objects is not writable). I guess I can simply use string replacement before creating the Doc, but I would like a method which is embedded into the language object. Any ideas how this might be achieved?

[polm]

You could a custom extension or even just your dict for this. Something like:

abbreviations = {"nyc": "New York City", ...}

out  = []
for tok in doc:
    out.append(abbreviations.get(tok.text, tok.text))
    out.append(tok.whitespace_)
print("".join(out))

If you use a custom extension you could set an underscore property, like tok._.expanded.

[antonpibm]

Thx for the reply.

Custom extension won't exactly do what I want due to 3 reasons:

1. It's not embedded into the language model, thus would have to be redefined when saving/loading
2. It won't correctly process the new text, first you would have to check if the extension is set for every text usage even for unextended words, which is cumbersome. Also the new representation won't be processed and won't have token attributes (pos as an example)

Ideally, I wish there was a possibility to add a Language component before tokenization

[polm]

Ah, OK. Yeah, you can't run components before the tokenizer, so your best option here is to just do string replacement on the input text. It's not as clean as having it in a component but it's very simple so it doesn't seem like it should be an issue?

In recent versions of spaCy you can pass Docs in to pipelines (see #9069), which would let you bypass the tokenizer, so you could use two pipelines here if you wanted. I don't think that would simplify things for you though.