Cython compile with MacOS 14, Spacy 3.7.2 and Cython 3.0.4

[kpdowney]

Hi all...

I have used Spacy for many years now but am very new to Cython and attempting to look at speeding up parts of our pipleline with Cython. However, I am stuck at the most basic of questions. I can't find a solution to the Cython extension build to find the spacy components on my system. I always get a bund of comiple errors starting with:
"venv/lib/python3.11/site-packages/spacy/tokens/Doc.pxd:4:0: 'spacy/attrs.pxd' not found"
and it just goes on from there. I'm guessing that my paths are not set correctly but I have tried to correct this in my setup.py file as follows:

from setuptools import setup, Extension

module = Extension ('ds_class_fast', sources=['async_classify/cython_file.pyx'],
include_dirs=['/Users/kevinmac/Documents/Companies/Sentinel/Codebase/Python/2023_Code/venv/lib/python3.11/site-packages/spacy'])

setup(
name='ds_test',
ext_modules=[module]
)

I have tried many different iterations of the path but nothing seems to work. The cython file is the most basic at this point just to point to something. Which is:

#cython: infer_types=True
cimport cython

from cymem.cymem cimport Pool

from spacy.tokens cimport Doc
from spacy.vocab cimport Vocab

doc = Doc(Vocab())

I realize this does almost nothing but it was just to see if I can get the compile to work.

Any assistance to help me in the right direction would be appreciated.

[adrianeboyd]

This is definitely frustrating. If it helps, I made a demo package that shows how to create a package with a custom extended spacy Tokenizer, which is 5% tokenizer and 95% packaging:

https://github.com/adrianeboyd/custom-cython-tokenizer

You shouldn't need to specify the spacy path in include_dirs and you shouldn't need to cimport cython in general in the .pyx file.

If you're using --no-build-isolation and you've installed spacy as an editable install with pip install --editable I would recommend switching back to a standard non-editable install. There are problems with header files being found with the newer types of editable installs.

[kpdowney]

Thanks for this Adriane.. It did help but just for your info the issue seems to be the version of Cython. There seems to be an issue with the 3.0.4 version I was using linking to relative repository references. I downgraded and it worked fine. Thanks again for your help!

[adrianeboyd]

Glad you figured it out, thanks for reporting back!