How to save doc to disk with nlp.pipe(as.tuple = True)?

[fatihbozdag]

Greetings,

I am looking for a way to save doc file with context included.

for task doc = nlp(text), dumbing with pickle works fine. However, I want to include context data with the tagged file.

So the question is, is there a way to save both context and doc in a container? for doc, context in nlp.pipe(text, as_tuple = True)?
It really takes so much to process a large corpus with metadata(context) again and again for various tasks.

[fatihbozdag]

Also I've faced with a strange behaviour (it might be also due to my lack of knowledge)

for instance,

import spacy
import pandas as pd

nlp = spacy.load('en_core_web_lg')

meta  = pd.read_csv('.../metadata.csv')
text = pd.read_csv('.../text.csv')
meta_x = meta.to_dict('records')
text_only = text['text_field'].values.tolist()
corpus = list(zip(text_only, meta_x))
verbs = []
meta1= []
meta2 = []

annotated_text = nlp.pipe(corpus, as_tuples = True:)

for doc, context in annotated_text:
             for token in doc:
                   if token.pos_ == 'VERB':
                       verbs.append(token.lemma_)
                       meta1.append(context['META1'])

This works like a charm! However, if I run the loop (with the very same code) for the second time, I got no results at all. Indeed, it returns blank.

For instance;

for doc, context in annotated_text:
             for token in doc:
                   if token.pos_ == 'NOUN':
                       verbs.append(token.lemma_)
                       meta1.append(context['META1'])

or

for doc, context in annotated_text:
             for token in doc:
                   if token.pos_ == 'VERB':
                       verbs.append(token.lemma_)
                       meta1.append(context['META2'])

Provides no results. So I have to rerun annotated_text = nlp.pipe(corpus, as_tuples = True:) so that I can run the loop again. Why is the case?

[polm]

nlp.pipe is what is known as a "generator" - it is like a list that is created only when you ask for the next element. This makes it take less memory, and it doesn't run computation until you need it, but a side effect is that it can only be iterated once. See the Python docs for more info.

[polm]

The right way to do this depends on your access pattern.

A simple thing you can do is, if the data can be serialized easily, you can put it in a custom extension attribute (_ attribute) and use a DocBin to serialize things. However DocBins aren't designed for random access.

If you need random access, you'll want to serialize Docs individually, so you could just create a tuple like (doc, stuff) and serialize it, with some kind of separate index structure. However that will be very inefficient due to the way Vocabs work, so if you need to do that you should copy just the parts of the Doc you need into another data structure.

If you give more details about the type of data you have in "context" and what your access needs are we may be able to give better advice.

[polm]

In that case it sounds like an extension attribute on the Doc will do what you need then.

[fatihbozdag]

I have failed to apply custom extension on doc object. I couldn't figure out how to import 'docid_field' column from pandas to getter function.

TypeError: Argument 'other' has incorrect type (expected spacy.tokens.token.Token, got str)

So should the attribute be included in the text to be parsed? What I am trying to do is to set Doc._. from df['docid_field'] while parsing the text included in df['text_field']. From the error above, it seems like the custom extension should be a token object from doc, is it true?

[kinghuang]

We also deal with a large amount of metadata in our pipeline. It looks like the documentation hasn't been updated to reflect this, but the Language.__call__ method was modified to accept a Doc object in addition to str. You can leverage this to create a Doc object for your text, add metadata to extensions on the doc, then run the doc through the pipeline.

Register extensions first:

from spacy.tokens import Doc

# Register extensions to hold metadata.
Doc.set_extension("docid", default=None)
Doc.set_extension("language", default=None)

Set extension values on doc before running through the pipeline:

text = "Some text to process"
docid_field = "BGSU1001"
native_language = "Bulgarian"

# Prepare the doc with its metadata.
doc = nlp.make_doc(text)
doc._.docid = docid_field
doc._.language = native_language

# Run the doc through the language pipeline.
doc = nlp(doc)

[fatihbozdag]

Awesome. How about iterating pandas list or dataframe? Csv file consists of 9529 obs, i.e. documents

[pmbaumgartner]

@fatihbozdag Usually the best way is through df.itertuples(), for example:

for row in df.itertuples():
   doc = nlp.make_doc(row.text)
   # custom attribute logic here