Python 3.14 compatibility

[frenzymadness]

In Python 3.14, some implementation details of typing have been changed, for example: python/cpython#105499

This is causing failures of some of the tests:

=================================== FAILURES ===================================
________________ test_string_annotation[annotations8-result8-] _________________

annotations = {'return': 'typing.Union[str, int]'}, result = ['int', 'str']
code = ''

    @pytest.mark.parametrize(
        'annotations, result, code', [
            ({}, [], ''),
            (None, [], ''),
            ({'asdf': 'str'}, [], ''),
    
            ({'return': 'str'}, ['str'], ''),
            ({'return': 'None'}, ['NoneType'], ''),
            ({'return': 'str().upper'}, [], ''),
            ({'return': 'foo()'}, [], ''),
            ({'return': 'bar()'}, ['float'], ''),
    
            # typing is available via globals.
            ({'return': 'typing.Union[str, int]'}, ['int', 'str'], ''),
            ({'return': 'typing.Union["str", int]'},
             ['int', 'str'] if sys.version_info >= (3, 9) else ['int'], ''),
            ({'return': 'typing.Union["str", 1]'},
             ['str'] if sys.version_info >= (3, 11) else [], ''),
            ({'return': 'typing.Optional[str]'}, ['NoneType', 'str'], ''),
            ({'return': 'typing.Optional[str, int]'}, [], ''),  # Takes only one arg
            ({'return': 'typing.Any'},
             ['_AnyMeta'] if sys.version_info >= (3, 11) else [], ''),
    
            ({'return': 'typing.Tuple[int, str]'},
             ['Tuple' if sys.version_info[:2] == (3, 6) else 'tuple'], ''),
            ({'return': 'typing.Tuple[int, str]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.Tuple[int, str]'}, ['str'], 'x()[1]'),
            ({'return': 'typing.Tuple[int, str]'}, [], 'x()[2]'),
    
            ({'return': 'typing.List'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.List[int, str]'}, [], 'x()[0]'),
    
            ({'return': 'typing.Iterator[int]'}, [], 'x()[0]'),
            ({'return': 'typing.Iterator[int]'}, ['int'], 'next(x())'),
            ({'return': 'typing.Iterable[float]'}, ['float'], 'list(x())[0]'),
    
            ({'return': 'decimal.Decimal'}, [], ''),
            ({'return': 'lalalalallalaa'}, [], ''),
            ({'return': 'lalalalallalaa.lala'}, [], ''),
        ]
    )
    def test_string_annotation(annotations, result, code):
        x = lambda foo: 1  # noqa: E731
        x.__annotations__ = annotations
        defs = jedi.Interpreter(code or 'x()', [locals()]).infer()
>       assert [d.name for d in defs] == result
E       AssertionError: assert [] == ['int', 'str']
E         
E         Right contains 2 more items, first extra item: 'int'
E         Use -v to get more diff

test/test_api/test_interpreter.py:697: AssertionError
________________ test_string_annotation[annotations9-result9-] _________________

annotations = {'return': 'typing.Union["str", int]'}, result = ['int', 'str']
code = ''

    @pytest.mark.parametrize(
        'annotations, result, code', [
            ({}, [], ''),
            (None, [], ''),
            ({'asdf': 'str'}, [], ''),
    
            ({'return': 'str'}, ['str'], ''),
            ({'return': 'None'}, ['NoneType'], ''),
            ({'return': 'str().upper'}, [], ''),
            ({'return': 'foo()'}, [], ''),
            ({'return': 'bar()'}, ['float'], ''),
    
            # typing is available via globals.
            ({'return': 'typing.Union[str, int]'}, ['int', 'str'], ''),
            ({'return': 'typing.Union["str", int]'},
             ['int', 'str'] if sys.version_info >= (3, 9) else ['int'], ''),
            ({'return': 'typing.Union["str", 1]'},
             ['str'] if sys.version_info >= (3, 11) else [], ''),
            ({'return': 'typing.Optional[str]'}, ['NoneType', 'str'], ''),
            ({'return': 'typing.Optional[str, int]'}, [], ''),  # Takes only one arg
            ({'return': 'typing.Any'},
             ['_AnyMeta'] if sys.version_info >= (3, 11) else [], ''),
    
            ({'return': 'typing.Tuple[int, str]'},
             ['Tuple' if sys.version_info[:2] == (3, 6) else 'tuple'], ''),
            ({'return': 'typing.Tuple[int, str]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.Tuple[int, str]'}, ['str'], 'x()[1]'),
            ({'return': 'typing.Tuple[int, str]'}, [], 'x()[2]'),
    
            ({'return': 'typing.List'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.List[int, str]'}, [], 'x()[0]'),
    
            ({'return': 'typing.Iterator[int]'}, [], 'x()[0]'),
            ({'return': 'typing.Iterator[int]'}, ['int'], 'next(x())'),
            ({'return': 'typing.Iterable[float]'}, ['float'], 'list(x())[0]'),
    
            ({'return': 'decimal.Decimal'}, [], ''),
            ({'return': 'lalalalallalaa'}, [], ''),
            ({'return': 'lalalalallalaa.lala'}, [], ''),
        ]
    )
    def test_string_annotation(annotations, result, code):
        x = lambda foo: 1  # noqa: E731
        x.__annotations__ = annotations
        defs = jedi.Interpreter(code or 'x()', [locals()]).infer()
>       assert [d.name for d in defs] == result
E       AssertionError: assert [] == ['int', 'str']
E         
E         Right contains 2 more items, first extra item: 'int'
E         Use -v to get more diff

test/test_api/test_interpreter.py:697: AssertionError
_______________ test_string_annotation[annotations10-result10-] ________________

annotations = {'return': 'typing.Union["str", 1]'}, result = ['str'], code = ''

    @pytest.mark.parametrize(
        'annotations, result, code', [
            ({}, [], ''),
            (None, [], ''),
            ({'asdf': 'str'}, [], ''),
    
            ({'return': 'str'}, ['str'], ''),
            ({'return': 'None'}, ['NoneType'], ''),
            ({'return': 'str().upper'}, [], ''),
            ({'return': 'foo()'}, [], ''),
            ({'return': 'bar()'}, ['float'], ''),
    
            # typing is available via globals.
            ({'return': 'typing.Union[str, int]'}, ['int', 'str'], ''),
            ({'return': 'typing.Union["str", int]'},
             ['int', 'str'] if sys.version_info >= (3, 9) else ['int'], ''),
            ({'return': 'typing.Union["str", 1]'},
             ['str'] if sys.version_info >= (3, 11) else [], ''),
            ({'return': 'typing.Optional[str]'}, ['NoneType', 'str'], ''),
            ({'return': 'typing.Optional[str, int]'}, [], ''),  # Takes only one arg
            ({'return': 'typing.Any'},
             ['_AnyMeta'] if sys.version_info >= (3, 11) else [], ''),
    
            ({'return': 'typing.Tuple[int, str]'},
             ['Tuple' if sys.version_info[:2] == (3, 6) else 'tuple'], ''),
            ({'return': 'typing.Tuple[int, str]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.Tuple[int, str]'}, ['str'], 'x()[1]'),
            ({'return': 'typing.Tuple[int, str]'}, [], 'x()[2]'),
    
            ({'return': 'typing.List'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.List[int, str]'}, [], 'x()[0]'),
    
            ({'return': 'typing.Iterator[int]'}, [], 'x()[0]'),
            ({'return': 'typing.Iterator[int]'}, ['int'], 'next(x())'),
            ({'return': 'typing.Iterable[float]'}, ['float'], 'list(x())[0]'),
    
            ({'return': 'decimal.Decimal'}, [], ''),
            ({'return': 'lalalalallalaa'}, [], ''),
            ({'return': 'lalalalallalaa.lala'}, [], ''),
        ]
    )
    def test_string_annotation(annotations, result, code):
        x = lambda foo: 1  # noqa: E731
        x.__annotations__ = annotations
        defs = jedi.Interpreter(code or 'x()', [locals()]).infer()
>       assert [d.name for d in defs] == result
E       AssertionError: assert [] == ['str']
E         
E         Right contains one more item: 'str'
E         Use -v to get more diff

test/test_api/test_interpreter.py:697: AssertionError
_______________ test_string_annotation[annotations11-result11-] ________________

annotations = {'return': 'typing.Optional[str]'}, result = ['NoneType', 'str']
code = ''

    @pytest.mark.parametrize(
        'annotations, result, code', [
            ({}, [], ''),
            (None, [], ''),
            ({'asdf': 'str'}, [], ''),
    
            ({'return': 'str'}, ['str'], ''),
            ({'return': 'None'}, ['NoneType'], ''),
            ({'return': 'str().upper'}, [], ''),
            ({'return': 'foo()'}, [], ''),
            ({'return': 'bar()'}, ['float'], ''),
    
            # typing is available via globals.
            ({'return': 'typing.Union[str, int]'}, ['int', 'str'], ''),
            ({'return': 'typing.Union["str", int]'},
             ['int', 'str'] if sys.version_info >= (3, 9) else ['int'], ''),
            ({'return': 'typing.Union["str", 1]'},
             ['str'] if sys.version_info >= (3, 11) else [], ''),
            ({'return': 'typing.Optional[str]'}, ['NoneType', 'str'], ''),
            ({'return': 'typing.Optional[str, int]'}, [], ''),  # Takes only one arg
            ({'return': 'typing.Any'},
             ['_AnyMeta'] if sys.version_info >= (3, 11) else [], ''),
    
            ({'return': 'typing.Tuple[int, str]'},
             ['Tuple' if sys.version_info[:2] == (3, 6) else 'tuple'], ''),
            ({'return': 'typing.Tuple[int, str]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.Tuple[int, str]'}, ['str'], 'x()[1]'),
            ({'return': 'typing.Tuple[int, str]'}, [], 'x()[2]'),
    
            ({'return': 'typing.List'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['list'], 'list'),
            ({'return': 'typing.List[int]'}, ['int'], 'x()[0]'),
            ({'return': 'typing.List[int, str]'}, [], 'x()[0]'),
    
            ({'return': 'typing.Iterator[int]'}, [], 'x()[0]'),
            ({'return': 'typing.Iterator[int]'}, ['int'], 'next(x())'),
            ({'return': 'typing.Iterable[float]'}, ['float'], 'list(x())[0]'),
    
            ({'return': 'decimal.Decimal'}, [], ''),
            ({'return': 'lalalalallalaa'}, [], ''),
            ({'return': 'lalalalallalaa.lala'}, [], ''),
        ]
    )
    def test_string_annotation(annotations, result, code):
        x = lambda foo: 1  # noqa: E731
        x.__annotations__ = annotations
        defs = jedi.Interpreter(code or 'x()', [locals()]).infer()
>       assert [d.name for d in defs] == result
E       AssertionError: assert [] == ['NoneType', 'str']
E         
E         Right contains 2 more items, first extra item: 'NoneType'
E         Use -v to get more diff

test/test_api/test_interpreter.py:697: AssertionError
=========================== short test summary info ============================
FAILED test/test_api/test_interpreter.py::test_string_annotation[annotations8-result8-]
FAILED test/test_api/test_interpreter.py::test_string_annotation[annotations9-result9-]
FAILED test/test_api/test_interpreter.py::test_string_annotation[annotations10-result10-]
FAILED test/test_api/test_interpreter.py::test_string_annotation[annotations11-result11-]
==== 4 failed, 3770 passed, 22 skipped, 96 deselected, 1 xfailed in 46.74s =====

[frenzymadness]

Python 3.14 is in the release candidate state, and the final release is expected in October.

[davidhalter]

It seems like accessing Union/Optional has changed. I am happy to help with this one, but I'm a bit lazy to setup CI properly.

[frenzymadness]

It seems like accessing Union/Optional has changed. I am happy to help with this one, but I'm a bit lazy to setup CI properly.

I can help you setup CI for Python 3.14, if that's what you meant.

[davidhalter]

Yes :)

[frenzymadness]

Yes :)

#2070

[frenzymadness]

I wanted to take a deeper look and created this simple reproducer from the first failed test:

from jedi import Interpreter

x = lambda foo: 1
x.__annotations__ = {'return': 'typing.Union[str, int]'}

defs = Interpreter("x()", [locals()]).infer()
print([d.name for d in defs])

but it prints an empty list in Python 3.14, 3.13, 3.12, as well as 3.11. What I'm doing wrong?

[davidhalter]

I'm also really confused by this, tried to look into it for 10 minutes, but couldn't figure it out :-/

[davidhalter]

I just realized: You have to import typing. (Which is questionable).

[frenzymadness]

Indeed, thank you. Now, the reproducer works and I can continue digging deeper.

[frenzymadness]

So, the root cause is very deep in the code. The problem is in the chain of calls: get_annotation_name_and_args > safe_getattr > getattr_static

In Python 3.13, the inspected object is typing._UnionGenericAlias while in Python 3.14 it is typing.Union. When it gets down to getattr_static and it tries to get the __module__ attribute, it does not work anymore because the typing.Union has no __dict__ and the code around investigating MRO and metaclasses does not help because the MRO is just (<class 'typing.Union'>, <class 'object'>).

I kinda understand why the code here dances around __dict__ to investigate objects without the attribute access but obj.__module__ works in both tested Pythons and might be a last resort in the code of getattr_static or just for a defined set of classes.

By the way, there is a discussion about the possibility of reverting the changes done to typing.Union:

• Issue: Merge 'types.UnionType' with typing._UnionGenericAlias, not typing.Union python/cpython#137065
• Discussion: https://discuss.python.org/t/types-uniontype-was-merged-with-wrong-class-not-even-a-class/102275

I don't understand the implementation details well enough to say what are the possible outcomes but I hope my findings help us to fix jedi.

[frenzymadness]

The problem is in fact here:

    def get_annotation_name_and_args(self):
        """
        Returns Tuple[Optional[str], Tuple[AccessPath, ...]]
        """
        name = None
        args = ()
        if safe_getattr(self._obj, '__module__', default='') == 'typing':
            m = re.match(r'typing.(\w+)\[', repr(self._obj))
            if m is not None:
                name = m.group(1)

                import typing
                if sys.version_info >= (3, 8):
                    args = typing.get_args(self._obj)
                else:
                    args = safe_getattr(self._obj, '__args__', default=None)
        return name, tuple(self._create_access_path(arg) for arg in args)

args = typing.get_args(self._obj) works fine in Python 3.14 so the only thing we need here is to change how the module is detected here and how we get the name because the repr(self._obj) returns str | int for <class 'typing.Union'>.

[davidhalter]

Since this uses heuristics anyway I think it would even be reasonable to special case this in a very simple way.

However I would personally wait with merging until it's clear if the change gets reverted. I'm happy to review a PR, but would probably wait a bit until end of September to merge. How does that sound to you?

[frenzymadness]

The steering council has made the decision - the current behavior won't be reverted in the current 3.14 release.