Cython access to lemma string

[Gargonslipfisk]

Hey there!

I am trying to rewrite some text preprocessing pieces in Cython. Regarding with that, I have this snippet to filter stopwords:

from spacy.structs cimport TokenC
from spacy.attrs cimport IS_STOP, LEMMA
from spacy.lexeme cimport Lexeme

cdef struct Tok:
    TokenC* c
    int length
    
cdef filter_stop(Tok* docs, int n_docs):
    cdef list words
    words = []
    for doc in docs[:n_docs]:
        for c in doc.c[:doc.length]:
            if Lexeme.get_struct_attr(c.lex, IS_STOP):
                pass
            else:
                # words.append(Lexeme.get_struct_attr(c.lex, LEMMA))
                words.append(c.lemma)
    return words

What I expected was that Lexeme.get_struct_attr(c.lex, LEMMA) return a string instead of an integer (in fact, it always return 0, seems it can't find any entry).

Currently, filter_stop return the StringStore hash number, but what I am trying to retrieve is the lemma for those words whose aren't be flagged as stop. What would be the right way to do this in Cython?

[adrianeboyd]

Token/Lexeme.get_struct_attr (and basically all the cython methods) work with the string store hashes internally rather than strings, so it's expected to get an integer (attr_t which is uint64_t).

The underlying problem above is that LEMMA is only a Token attribute, not a Lexeme attribute. The only attribute that's stored on both underneath is NORM, but Token.get_struct_attr backs off to Lexeme.get_struct_attr for any unknown attributes. And then Lexeme.get_struct_attr returns 0 for any unknown attributes.

[Gargonslipfisk]

Thanks for the answer Adriane, now I have a better understanding of the problem.

Then, let me rephrase my question: is there any suitable method to bring the UTF8 string from Cython?

I cant use StringStore.as_string method because I have an uint64_t instead of a key.

[adrianeboyd]

A key can be uint64_t. I think it should be fine to call either StringStore.as_string() or just skip straight to StringStore.__getitem__():

cpdef my_method(StringStore string_store):
    cdef hash_t key = 5
    cdef str str_for_key = string_store[key]
    print(str_for_key)

[Gargonslipfisk]

Perfect. That is very convenient.

However, now I have two questions regarding with this implementation.

First, I suppose that you decide to use cpdef because we need to establish the bindings with the Python StringStore. There is no other way to retrieve strings, right?

I asking you because, my second question, I am trying to rewrite filter_stop function exclusively in C to take advantage of multi-threading capacities. According with Spacy documentation set functions as nogil is a good practice. But, I don't know if the advice applies to an stopword filter.

[adrianeboyd]

This method also works with cdef, but not with nogil. I used cpdef here to make it easier to call directly from python for testing purposes.

In general it seems like you might be trying to optimize/parallelize this at the wrong point. For easier parallelization, I'd recommend implementing a pipeline component that sets the filtered hash values or words as a custom attribute. Then you can parallelize with nlp.pipe(n_process=).

A base class is shown in Pipe, at a minimum you need to implement __call__ to process the doc and set the annotation you want on the doc.

https://github.com/explosion/spaCy/blob/2d89dd9db898e66058bf965e1b483b0019ce1b35/spacy/pipeline/pipe.pyx

Guidelines for custom components: https://spacy.io/usage/processing-pipelines#custom-components

I'd recommend initially implementing the pipeline component in python, testing the speed for your task with nlp.pipe (also see tips in #8402) and then only switching to cython if it seems necessary. You might not need to.

We don't have a minimal example for this in cython, but one example you could reference is the arc_labeler in spacy-experimental. See the entry point in setup.cfg and make_arc_labeler in the component. (Just note the trainable components implement predict+set_annotations rather than __call__, so the component itself isn't an example to follow.)

https://github.com/explosion/spacy-experimental/blob/7ca435d01aa87b2e61760a266fe8cdb748043ee1/spacy_experimental/biaffine_parser/arc_labeler.pyx