Type checking with Pylance

[ezorita]

How to reproduce the behaviour

Not really a bug, but rather an issue with type checking using vscode. Not sure if this is an issue of spacy or the vscode Pylance extension...

import spacy
from spacy.tokens import Doc, Span, Token
from spacy.language import Language
from spacy.lang.en import English

Importing Doc, Span, or Token triggers an unknown import symbol.

Your Environment

• Operating System: Ubuntu 20.04
• Python Version Used: 3.8.5
• spaCy Version Used: 3.0.6
• Environment Information: vscode 1.57.0 with Pylance v2021.6.1

[polm]

I believe this is because those types are defined in Cython and therefore don't have normal type annotations.

[ezorita]

I can try to manually create all these stub files, taking type info from the docs.

[polm]

I think we would really want a method to be able to build these automatically - it would be pretty bad to have updates to internal types break annotations for people.

[ezorita]

Not sure whether there is any good solution for automatic stub file generation.. I tried stubgen and produces poor generic .pyi files.

In any case, isn't the Python class interface supposed to be backward compatible? If so, manual updates would only be required whenever some class/method interface is modified.

[hixan]

my understanding is that if internal types change, the stub files will have to change as well. A way to mitigate the problem of broken types would be to update test cases to enforce correct annotations in addition to functionality. Unfortunately, this is even more work then just generating or writing type stubs.

If this were done, changes in types in a way that would break an existing test case will get flagged as having incorrect annotations as well, forcing the relevant stub to be updated at the same time.

Not sure whether there is any good solution for automatic stub file generation.. I tried stubgen and produces poor generic .pyi files.

How did you get stubgen to work on the cython code at all?

[ezorita]

I agree @hixan, I think changes in interface types will be rare and these would break or at least raise a type incompatibility in test cases.
I did run stubgen out of the box on the entire spacy package stubgen -p path_to_spacy

[ezorita]

fyi; In case anyone is interested in contributing -- I created a first version of the stub files that works fine in my vscode environment and opened a PR #8427

[NixBiks]

Great work @ezorita

It doesn't seem to be working for me though. I've installed v3.1.3 where your PR is merged into but no autocomplete and pyright yells at me too. I've looked in .venv/lib/python3.9/site-packages/spacy/tokens and I don't see any .pyi files - only .py, .pxd and pyx. Is that a bug or do I need additional configuration to make it work?

[svlandeg]

Hi @mr-bjerre , thanks for the note! I had tested the functionality from #8427 on a branch compiled from source, but it looks like we forgot to package the actual .pyi files into the release. We'll get that fixed!

[svlandeg]

Update: should be fixed by #9500