fix(typing): Overhaul @overloads in nw.from_native
#3125
Conversation
Will close #1928
Absolutely not doing this with another class
wayyy more common in this case
The `TypedDict`s are clear enough
narwhals/typing.py
Outdated
| class NativeDataFrame(Sized, NativeFrame, Protocol): | ||
| def drop(self, *args: Any, **kwargs: Any) -> Any: ... | ||
|
|
||
| class NativeLazyFrame(NativeFrame, Protocol): | ||
| def explain(self, *args: Any, **kwargs: Any) -> Any: ... | ||
|
|
||
| # Needs to have something `NativeDataFrame` doesn't? | ||
| class NativeSeries(Sized, Iterable[Any], Protocol): | ||
| def filter(self, *args: Any, **kwargs: Any) -> Any: ... | ||
| # `pd.DataFrame` has this - the others don't | ||
| def value_counts(self, *args: Any, **kwargs: Any) -> Any: ... | ||
| # `pl.DataFrame` has this - the others don't | ||
| def unique(self, *args: Any, **kwargs: Any) -> Any: ... |
There was a problem hiding this comment.
Issue 1
NativeDataFrame and NativeSeries could overlap.
Although we don't define NativeDataFrame.filter - the classes that match it do:
That was leading to the strange @overload order and parameters changing the return type.
But now if we started with NativeSeries, then both series_only and allow_series will preserve that detail:
narwhals/translate.py
Outdated
| class OnlyEager(TypedDict, total=False): | ||
| pass_through: bool | ||
| eager_only: Required[Literal[True]] | ||
| series_only: bool | ||
| allow_series: bool | None |
There was a problem hiding this comment.
Issue 2
There's too many combinations we need to account for in @overloads.
- 3x different, true
native_objects - 3x narwhals objects pretending to be native
- 4x flags
- They all have a default, so needs
... allow_series: bool | Nonejust adds to it
- They all have a default, so needs
So instead of that:
dangotbanned
changed the title
from_native@overloads in nw.from_native
|
@MarcoGorelli this one isn't ready for a proper review yet, but I was curious about your thoughts on the idea generally? I did some testing locally and I think depending on how these issues have progressed - docs might be one more area for experimenting |
tbh I'm still not too keen on matching on the native object, i kind of feel like |
That isn't possible due to the issue from: Which is also why Possibly these sections of the spec may be helpful?
These are the issues that come from having such a heavily polymorphic function I'm afraid 😢 |
`v1`, `v2` can extend and reuse these defs
Need `OnlySeries` to finish the other replacements
ooooh I think we're almost ready
I've updated the PR description with a table showing this in more detail
Closes #1928
What type of PR is this? (check all applicable)
Related issues
DataFrame,LazyFrame#2944_native.py#3086Checklist
If you have comments or can explain your changes, please do so below
Note
The positive diff is from new quite exhaustive tests
narwhals.translatehas shrunk a lot 🥳TODO
Native{DataFrame,Series}nw._nativeNativeSeriesfiltermight not be needed anymoreTypedDicts intonw._translateTypedDicts forstablev1(thread)v1AllowAnyvariantsAllowSeriesvariantsOnlySeriesvariantsExcludeSeriesvariantsAllowLazyvariantsOnlyEagerOrInterchangevariantsv2mainfrom_native()@overloads 1-3 to useTypedDictsmkdocspresents the new typingTypedDicts?Unpackwork out-of-the-box?@overloads useful/noisy?Previews
fix(typing): More gracefully handle narwhals in overloads
Anywas previously being used on the first few overloads to solve a recursion issue.But that had the downside of swallowing completions until you matched an overload with
native_object:Show before
But now, we have something more useful
Show after
Changes to
Native{Series,DataFrame}As mentioned in (#3125 (comment)), these protocols needed to be
adapted to make them disjoint across
pandas,polars,pyarrow:dropfilteruniquevalue_countspd.Seriespl.Seriespa.ChunkedArraypd.DataFramepl.DataFramepa.Table