Skip to content

Conversation

@dangotbanned
Copy link
Member

@dangotbanned dangotbanned commented Aug 20, 2025

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • ✨ Feature
  • 🐛 Bug Fix
  • 🔧 Optimization
  • 📝 Documentation
  • ✅ Test
  • 🐳 Other

Related issues

Checklist

  • Code follows style guide (ruff)
  • Tests added
  • Documented the changes

Examples

This PR gives us static typing for Implementation at the narwhals-level:

from __future__ import annotations

from typing import Any

import pandas as pd
import polars as pl
import pyarrow as pa
from typing_extensions import reveal_type

import narwhals as nw

data: dict[str, Any] = {"a": [1, 2, 3]}

impl_1 = nw.from_native(pd.DataFrame(data)).implementation
impl_2 = nw.from_native(pl.DataFrame(data)).implementation
impl_3 = nw.from_native(pa.table(data)).implementation

reveal_type(impl_1)  # Type of "impl_1" is "Literal[Implementation.PANDAS]"
reveal_type(impl_2)  # Type of "impl_2" is "Literal[Implementation.POLARS]"
reveal_type(impl_3)  # Type of "impl_3" is "Literal[Implementation.PYARROW]"

Adapting (https://github.com/narwhals-dev/narwhals#example), the fun stuff happens when we use aliases/type variables.

IntoFrameT expands to all valid Implementation's:

if TYPE_CHECKING:
    from narwhals.typing import IntoFrameT


def agnostic_function(df_native: IntoFrameT) -> IntoFrameT:
    frame = nw.from_native(df_native)
    reveal_type(frame.implementation)  # Type of "frame.implementation" is "Literal[Implementation.PANDAS, Implementation.CUDF, Implementation.MODIN, Implementation.PYARROW, Implementation.POLARS, Implementation.PYSPARK, Implementation.SQLFRAME, Implementation.PYSPARK_CONNECT, Implementation.DASK, Implementation.DUCKDB, Implementation.IBIS]"
    return frame.to_native()

However, if we tweak that to use IntoDataFrameT it can be narrowed to all of EagerAllowed:

if TYPE_CHECKING:
    from narwhals.typing import IntoDataFrameT


def agnostic_function(df_native: IntoDataFrameT) -> IntoDataFrameT:
    frame = nw.from_native(df_native)
    reveal_type(frame.implementation)  # Type of "frame.implementation" is "Literal[Implementation.PANDAS, Implementation.CUDF, Implementation.MODIN, Implementation.PYARROW, Implementation.POLARS]"
    return frame.to_native()

The inverse also works, so when we use IntoLazyFrameT it can be narrowed to all of LazyAllowed:

if TYPE_CHECKING:
    from narwhals.typing import IntoLazyFrameT


def agnostic_function(df_native: IntoLazyFrameT) -> IntoLazyFrameT:
    frame = nw.from_native(df_native)
    reveal_type(frame.implementation)  # Type of "frame.implementation" is "Literal[Implementation.PYSPARK, Implementation.SQLFRAME, Implementation.PYSPARK_CONNECT, Implementation.DASK, Implementation.DUCKDB, Implementation.IBIS, Implementation.POLARS]"
    return frame.to_native()

Note

See tests/implementation_test.py for more exhaustive examples 😅

@dangotbanned dangotbanned added enhancement New feature or request fix typing labels Aug 20, 2025
@FBruzzesi
Copy link
Member

I didn't take a look at this yet, but I want to add some context if someone else is reviewing it and wonders what the hell is happening.

In #2983 I am creating a new series via a snippet such as:

x: SeriesT
impl = x.implementation
new_series(..., backend=impl)

Since in #3002 we too good, now the type checker complains since backend is supposed to be a eager Implementation (how can we blame it after all?), however the return type of Series.implementation is just Implementation.

I then started a series of adjustments which led to nowhere, so I asked for help and here is @dangotbanned making magic

Copy link
Member

@FBruzzesi FBruzzesi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dangotbanned - I left a few comments/questions, the main one being: I would already be happy to have the distinction between eager and lazy, so that we know that a DataFrame and a Series always returns a eager implementation (although a lazyframe might return any implementation in practice, since class EagerDataFrame(CompliantDataFrame[...], CompliantLazyFrame[...], ...)

@dangotbanned dangotbanned requested a review from FBruzzesi August 28, 2025 19:59
Copy link
Member

@FBruzzesi FBruzzesi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am quite happy with this! Thanks @dangotbanned

@MarcoGorelli do you have any objection and/or strong opinion against it?

"""Return True if `impl` allows eager operations."""
return impl in {
Implementation.CUDF,
Implementation.MODIN,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌🏼

@dangotbanned
Copy link
Member Author

I am quite happy with this! Thanks @dangotbanned

Thanks @FBruzzesi!

@MarcoGorelli do you have any objection and/or strong opinion against it?

I cannot stress this enough ...

This PR is easily the most complex bit of typing I've ever worked on 😳
I hope that all the docs and tests help break down the chunkier parts

But I had no idea this was possible a week ago when I started it 😂

Copy link
Member

@MarcoGorelli MarcoGorelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one comment, else looks good, thanks!

@dangotbanned dangotbanned merged commit 2869349 into main Sep 3, 2025
22 of 31 checks passed
@dangotbanned dangotbanned deleted the implementation-typing branch September 3, 2025 17:06
dangotbanned added a commit that referenced this pull request Sep 4, 2025
dangotbanned added a commit that referenced this pull request Sep 4, 2025
`v1` is different for the frame cases #3016 (comment)
@dangotbanned dangotbanned mentioned this pull request Sep 4, 2025
14 tasks
dangotbanned added a commit that referenced this pull request Sep 5, 2025
- Follow-up to #3016
- Doesn't quite fit into #3086
FBruzzesi pushed a commit that referenced this pull request Sep 6, 2025
- Follow-up to #3016
- Doesn't quite fit into #3086
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Question: Implementation/backend-related mypy complaint since narwhals 2.2.0

4 participants