errors with type narrowing

[jasonboh]

Hi, I am having problems understanding what I am seeing here in the below example.

Why can't mypy figure out that after the else statement, we definitely have a DataFrame?

from typing import Union

import pandas as pd

x: Union[pd.Series, pd.DataFrame, pd.Index] = pd.Series([1, 2, 3])
if isinstance(x, (pd.Series, pd.Index)):
    reveal_type(x)  # Revealed type is "Union[pandas.core.series.TimestampSeries, pandas.core.indexes.numeric.NumericIndex]"
    dtype = x.dtype
    reveal_type(dtype)  # Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype]"
else:  # a DataFrame
    reveal_type(x)  # Revealed type is "Union[pandas.core.series.Series[Any], pandas.core.frame.DataFrame, pandas.core.indexes.base.Index]"
    dtypes = x.dtypes  # yields error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr] 
    reveal_type(dtypes)  # "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype, pandas.core.series.Series[Any], Any]"
    dtypes_set = set(x.dtypes) # yield 2 errors: 
    # first error Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
    # second error: Argument 1 to "set" has incompatible type "Union[dtype[generic], ExtensionDtype, Series[Any], Any]"; expected "Iterable[Any]"  [arg-type]

[gandhis1]

This would appear to be a Mypy bug. It's probably worth trying pyright as well - I suspect they've got it right. Either way, not a pandas-stubs issue.

[Dr-Irv]

I modified the code a bit, and this is a mypy bug. One issue is that you bind x to a Series, so the else part is never entered. So I wrapped it in a function

import pandas as pd

def foo(x: Union[pd.Series, pd.DataFrame, pd.Index]): # = pd.Series([1, 2, 3])
    if isinstance(x, (pd.Series, pd.Index)):
        reveal_type(x)  # Revealed type is "Union[pandas.core.series.TimestampSeries, pandas.core.indexes.numeric.NumericIndex]"
        dtype = x.dtype
        reveal_type(dtype)  # Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype]"
    else:  # a DataFrame
        reveal_type(x)  # Revealed type is "Union[pandas.core.series.Series[Any], pandas.core.frame.DataFrame, pandas.core.indexes.base.Index]"
        dtypes = x.dtypes  # yields error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
        reveal_type(dtypes)  # "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype, pandas.core.series.Series[Any], Any]"
        dtypes_set = set(x.dtypes) # yield 2 errors:
        # first error Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
        # second error: Argument 1 to "set" has incompatible type "Union[dtype[generic], ExtensionDtype, Series[Any], Any]"; expected "Iterable[Any]"  [arg-type]

pyright correctly reports:

  issue516a.py:7:21 - information: Type of "x" is "Series[Unknown] | Index"
  issue516a.py:9:21 - information: Type of "dtype" is "dtype[generic] | ExtensionDtype"
  issue516a.py:11:21 - information: Type of "x" is "DataFrame"
  issue516a.py:13:21 - information: Type of "dtypes" is "Series[Unknown]"

mypy incorrectly reports:

issue516a.py:7: note: Revealed type is "Union[pandas.core.series.TimestampSeries, pandas.core.indexes.numeric.NumericIndex]"
issue516a.py:9: note: Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype]"
issue516a.py:11: note: Revealed type is "Union[pandas.core.series.Series[Any], pandas.core.frame.DataFrame, pandas.core.indexes.base.Index]"
issue516a.py:12: error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
issue516a.py:13: note: Revealed type is "Union[numpy.dtype[numpy.generic], pandas.core.dtypes.base.ExtensionDtype, pandas.core.series.Series[Any], Any]"
issue516a.py:14: error: Item "Index" of "Union[Series[Any], DataFrame, Index]" has no attribute "dtypes"  [union-attr]
issue516a.py:14: error: Argument 1 to "set" has incompatible type "Union[dtype[generic], ExtensionDtype, Series[Any], Any]"; expected "Iterable[Any]"  [arg-type]

[Dr-Irv]

Created a mypy issue: python/mypy#14502

[jasonboh]

thanks everyone! I have started using pyright in VS Code.

[Dr-Irv]

thanks everyone! I have started using pyright in VS Code.

You can also use pyright from the command line as well.

[Dr-Irv]

reopening so we can track if mypy ever fixes the bug

[Dr-Irv]

With mypy 1.14.1, this now works correctly. Using the original code:

play\issue516.py:7: note: Revealed type is "Union[pandas.core.series.Series[builtins.float], pandas.core.indexes.base.Index[builtins.int]]"
play\issue516.py:9: note: Revealed type is "Union[numpy.dtype[numpy.generic[Any]], pandas.core.dtypes.base.ExtensionDtype]"
play\issue516.py:11: note: Revealed type is "pandas.core.frame.DataFrame"
play\issue516.py:13: note: Revealed type is "pandas.core.series.Series[Any]"

Closing as a result.