Is it possible to merge a list of spacy tokens into a doc?

[vrunm]

I have a document which I've tokenized using Spacy tokenizer. I want to apply ner on a sequence of tokens(a section of this document).

Currently I'm creating a doc first and then applying ner

nlp = spacy.load("en_core_web_sm")
# tokens_list is a list of Spacy tokens
words = [tok.text for tok in tokens_list]

spaces = [True if tok.whitespace_ else False for tok in tokens_list]

doc = spacy.tokens.doc.Doc(blackstone_nlp.vocab,
      words=words, spaces=spaces)
doc = nlp.get_pipe("ner")(doc)

But this is not ideal because I loose their original ids within the document, which is important.

[svlandeg]

Hi!

Could you clarify a bit more what you're trying to achieve? As I understand it, you already have a Doc object with tokens, so it's not clear to me why you need to create a new one. Do you want to have a subsection of it only? In this case, perhaps working through a Span object is more beneficial. You can call span.as_doc on it if you really need a new document, but you could also just run the NER on your full document and only look at the entities in a given subsection (Span).

FYI - if you're interested in only keeping the ner component of "en_core_web_sm" , you could do

nlp = spacy.load("en_core_web_sm", enable=["ner"])

or, alternatively, you could load it as is and do

    with nlp.select_pipes(enable="ner"):
        doc = nlp(text)

which uses a context manager to enable/disable pipeline components on the fly.

[vrunm]

Is there a way to merge tokens into a doc and still maintain their ids?

To merge a list of tokens back into Doc this what I tried:

import spacy
nlp = spacy.load("en_core_web_sm")
txt = "This is some text"
doc = nlp(txt)
words = [tok.text for tok in doc]

spaces = [True if tok.whitespace_ else False for tok in doc]
doc2 = spacy.tokens.doc.Doc(nlp.vocab, words=words, spaces=spaces)

[vrunm]

I want my model to apply NER to the whole document but would like the model to ignore the unique id's and then apply the NER on the text of the text. I would like to mask that part of the document and apply the NER for the rest of the document.

Another solution which could work was to remove the id's and train the custom NER model on the remaining text and then using the model for inference ( but the model gets confused sometimes and identifies the unique id's as DATE sometimes).

[svlandeg]

Ok, that's more clear. In such a case, you should do the preprocessing before you send the text to spaCy. Basically, remove the identifiers from the text, then create your Doc by running your pipeline and doing NER on the cleaned text. You can store the identifiers that you removed as a custom attribute.

[vrunm]

Sure I will try that but at inference the unique id's sometimes confuse the model and return wrong NER tags like DATE, ORG etc. Will the custom attribute deal with that problem also?

[svlandeg]

Well, at inference time you should preprocess your texts in exactly the same way - i.e. remove the integers before running the text through spaCy.

[vrunm]

That won't be the case because I am using it in a another application which is fetching those documents in real time and cannot remove those unique id's as that is a unique identifier required to query each document.