Can't retrieve unregistered extension attribute 'is_doid_term'. Did you forget to call the set_extension method?

[ali3assi]

We define and add the following extratcor to spacy pipeline:

from spacy.matcher import PhraseMatcher
from spacy.util import filter_spans
from spacy.language import Language

@Language.factory("doid_extractor") 
class DOIDExtractorComponent(object):
    # name of the component
   

    def __init__(self, nlp, label="DOID",  name = "doid_extractor"):
        # label that is applied to the matches
        self.label = label

        # load ontology
        print("Loading DOID ontology")
        doid = Ontology("http://purl.obolibrary.org/obo/doid.obo")
        
        # init terms and patterns
        self.terms = {}
        patterns = []

        i = 0
        nr_terms = len(doid.terms())
        # init progress bar as loading terms takes long
        print("Importing terms")
        bar = progressbar.ProgressBar(maxval=nr_terms, 
                                      widgets=[progressbar.Bar('=', '[', ']'), ' ', progressbar.Percentage()])
        bar.start()

        # iterate over terms in ontology
        for term in doid.terms():
          # if term has a name
          if term.name is not None:
            self.terms[term.name.lower()] = {'id': term.id}
            patterns.append(nlp(term.name))
          i += 1
          bar.update(i)
            
        bar.finish()
        
        # initialize matcher and add patterns
        self.matcher = PhraseMatcher(nlp.vocab, attr='LOWER')
        self.matcher.add(label, None, *patterns)
        
        # set extensions to tokens, spans and docs
        Token.set_extension("is_doid_term", default=False, force=True)
        Token.set_extension("doid_id", default=False, force=True)
        Token.set_extension("merged_concept", default=False, force=True)
        Doc.set_extension("has_doids", getter=self.has_doids, force=True)
        Doc.set_extension("doids", default=[], force=True)
        Span.set_extension("has_doids", getter=self.has_doids, force=True)
        
    def __call__(self, doc):
        matches = self.matcher(doc)
        spans = [Span(doc, match[1], match[2], label=self.label) for match in matches]
        for i, span in enumerate(spans):
          span._.set("has_doids", True)
          for token in span:
               token._.set("is_doid_term", True)
               token._.set("doid_id", self.terms[span.text.lower()]["id"])

        with doc.retokenize() as retokenizer:
            for span in filter_spans(spans):
                retokenizer.merge(span, attrs={"_": {"merged_concept": True}})
                doc._.doids = list(doc._.doids) + [span]

        return doc
    # setter function for doc level
    def has_doids(self, tokens):
        return any([t._.get("is_doid_term") for t in tokens])

and then in the

if __name__ == "__main__":

 # # python -m spacy download en_core_web_sm
    nlp = spacy.load("en_core_web_sm")

nlp.add_pipe("doid_extractor", after = "ner")

It works good as I want. Now I have a dataframe where each row include a column called text including some text.

So I write I the following function:

def row_df(df, i):
    text = {} 
    text = str(df.loc[i, "text"]) 
    doc = nlp(text)
    for token in doc:                
          if token._.is_doid_term:              
              root = token._.icco_id.replace(":","_")

and I want to parrallelise the use of this functio. So, in the main I add the following:

Parallel(n_jobs = 4)(delayed(row_df)(df, nlp, idx) for idx, _ in df.iterrows())

When run the main I get the following error:

Traceback (most recent call last):
  File "C:\Users\Admin\Documents\onto.py", line 108, in <module>
    Parallel(n_jobs = 4)(delayed(row_df)(df, idx) for idx, _ in df.iterrows())
  File "C:\Users\Admin\miniconda3\envs\projet1\lib\site-packages\joblib\parallel.py", line 1056, in __call__
    self.retrieve()
  File "C:\Users\Admin\miniconda3\envs\projet1\lib\site-packages\joblib\parallel.py", line 935, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\Users\Admin\miniconda3\envs\projet1\lib\site-packages\joblib\_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File "C:\Users\Admin\miniconda3\envs\projet1\lib\concurrent\futures\_base.py", line 445, in result
    return self.__get_result()
  File "C:\Users\Admin\miniconda3\envs\projet1\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
AttributeError: [E046] Can't retrieve unregistered extension attribute 'is_doid_term'. Did you forget to call the `set_extension` method?

I dont know how to solve this issue. any suggestion. Thanks

Noting that if I set the number of jobs to 1, the code works perfectly?

[polm]

It looks like your set_extension methods should be called before any multiprocessing happens, but that seems not to be working. Can you move your set_extension calls out of __init__ and call them at the top of your code? I believe that should resolve it or clarify that joblib is doing something that is interfering with the global state.

Also see #9184 for a similar issue.

[ali3assi]

@polm Sorry I dont understand your answer can you be more precise?

[polm]

Put your set_extension calls at the top of your code, like this:

Token.set_extension("is_doid_term", default=False, force=True)
Token.set_extension("doid_id", default=False, force=True)
Token.set_extension("merged_concept", default=False, force=True)
Doc.set_extension("has_doids", getter=self.has_doids, force=True)
Doc.set_extension("doids", default=[], force=True)
Span.set_extension("has_doids", getter=self.has_doids, force=True)

if __name__ == "__main__":
    ...

And delete them from where they currently are.

You'll have to change the ones that have getters, since you don't have self.

[ali3assi]

@polm this solution is not working. I still get the same error

[ali3assi]

@adrianeboyd can I get your help to solve this issue please?

[polm]

Don't @ maintainers to get their attention.

[polm]

I didn't realize it in my previous reply, but this is a general issue with extensions and multiprocessing. See #8352 (comment) for background and a fix.

[ali3assi]

The mentionned comment doesnt solve the issue

[polm]

To clarify, the linked comment explains that on Windows the default multiprocess strategy is spawn, and in order for extensions to work with that you need to declare them both at the top of your code and in the method where you set or access the extension value. You should try that if you haven't.

Another option is to use nlp.pipe(texts, n_process=...), which will handle all of this for you while making use of parallelization.

[ali3assi]

@polm Thanks for your suggestion. I solved this issue by reduplicate the set_extensions inside the call :)

[polm]

Glad you figured it out!