Refine public `Compliant*` methods

[MarcoGorelli]

We're currently using the compliant classes to share some code between implementations

This is fine, but for protocols with other 3rd party backends should implement, I don't think this is ok:

• it will make it harder for us to change some internals
• it forces them to implement methods like _evaluate_expr which they really shouldn't need to care about

I'd like to suggest therefore that we aim to have:

• CompliantDataFrame, CompliantLazyFrame, CompliantSeries, CompliantExpr, CompliantNamespace: these should be Protocols which only implement the necessary functionality which we call from narwhals.DataFrame / ...
• we can subclass these to preserve some common functionality within Narwhals. For example, DuckDBExpr / SparkLikeExpr / ... all inherit from LazyExpr, which in turn inherits from CompliantExpr. It's fine to have some implementations in LazyExpr

PRs

• refactor: Move/remove private Compliant*Frame methods #2934
• refactor(typing): Remove CompliantDataFrame.collect #3041
• refactor: Organizing Compliant* APIs #3045
• refactor: Split up BaseFrame._extract_compliant #3054
• feat: Raise on mismatched *Frame joins #3055
• refactor: Add CompliantFrame protocol #3056
• refactor: Move CompliantExpr._eval_names_indices #3062
• chore: Add CompliantFrame._with_native #3140

[dangotbanned]

Sounds good, agreed!

I recall @FBruzzesi and I discussing something similar and there was a mermaid diagram with an idea
But it seems we forgot 🤦‍♂

I'll try to find it since it might help with where to move stuff

Update

Found it (#2563 (comment))

So we'd just be prefixing the new Protocols with something like Impl

Hidden, see the updated version in comment

@MarcoGorelli @dangotbanned I think we start to have a few piece of the code base (such as this one) which can be abstracted for duckdb, ibis and spark-like (dask tbd). I am having a hard time to come up with a name for such compliant class, mostly because EagerDataFrame inherits from CompliantLazyFrame 😂 LazyDataFrame does not sound right for some reason.

Well spotted @FBruzzesi!

The main blocker is that EagerDataFrame inherits both protocols.

So (PandasLike|Arrow)DataFrame have the awkward EagerDataFrame. The only (bad) idea I've had since then was LazyLazyFrame (🤦‍♂️) for the other side between between CompliantLazyFrame -> *LazyFrame to avoid conflicts.

But an alternative could be this tangled web 😄

graph LR;
    CompliantDataFrame-->PolarsDataFrame;
    CompliantDataFrame-->ImplDataFrame;
    CompliantLazyFrame-->ImplLazyFrame;
    CompliantLazyFrame-->ImplDataFrame;
    ImplDataFrame-->ArrowDataFrame;
    ImplDataFrame-->PandasLikeDataFrame;
    ImplLazyFrame-->SparkLikeLazyFrame;
    ImplLazyFrame-->DuckDBLazyFrame;
    ImplLazyFrame-->DaskLazyFrame;
    ImplLazyFrame-->IbisLazyFrame;
    CompliantLazyFrame-->PolarsLazyFrame;

Loading

[dangotbanned]

Rough target to aim for

Where I used (Data|Lazy)Frame before (#2563 (comment)), I think that missed out on another important part.

So here I'm using Expr, and everything in bold is proposed (but naming is fully open to better suggestions).

I think the extra parts will help with how dask is more like pandas (#2613), so we can avoid

dask quirks

narwhals/narwhals/_dask/expr.py

Lines 684 to 686 in 4bba50a

	_alias_native = not_implemented()
	window_function = not_implemented() # pyright: ignore[reportAssignmentType]
	_from_elementwise_horizontal_op = not_implemented()

narwhals/narwhals/_dask/expr.py

Lines 572 to 574 in 4bba50a

	def over(self, partition_by: Sequence[str], order_by: Sequence[str]) -> Self:
	# pandas is a required dependency of dask so it's safe to import this
	from narwhals._pandas_like.group_by import PandasLikeGroupBy

narwhals/narwhals/_dask/expr.py

Line 16 in 4bba50a

from narwhals._pandas_like.utils import native_to_narwhals_dtype

narwhals/narwhals/_dask/dataframe.py

Line 10 in 4bba50a

from narwhals._pandas_like.utils import native_to_narwhals_dtype, select_columns_by_name

narwhals/narwhals/_dask/dataframe.py

Lines 39 to 44 in 4bba50a

	Incomplete: TypeAlias = "Any"
	"""Using `_pandas_like` utils with `_dask`.
	Typing this correctly will complicate the `_pandas_like`-side.
	Very low priority until `dask` adds typing.
	"""

narwhals/narwhals/_dask/expr_dt.py

Lines 7 to 12 in 4bba50a

	from narwhals._pandas_like.utils import (
	UNIT_DICT,
	calculate_timestamp_date,
	calculate_timestamp_datetime,
	native_to_narwhals_dtype,
	)

graph TD;
    Compliant[CompliantExpr]---->Impl[**ImplExpr**];
    Compliant-->Polars[PolarsExpr];
    Impl-->DepthTracking["`DepthTrackingExpr
      rename me pls`"];
    DepthTracking-->Eager[EagerExpr];
    Eager-->Arrow[ArrowExpr];
    Eager-->PandasLike[PandasLikeExpr];
    DepthTracking-->PandasIsh["`**PandasIshExpr**
      rename me pls`"];
    PandasIsh-->PandasLike;
    PandasIsh-->Dask[DaskExpr];
    Impl-->Lazy[LazyExpr];
    Lazy-->Dask;
    Lazy-->SqlLike[**SqlLikeExpr**];
    SqlLike-->DuckDB[DuckDBExpr];
    SqlLike-->Ibis[IbisExpr];
    SqlLike-->SparkLike[SparkLikeExpr];

Loading

[dangotbanned]

Should we roll this issue in as a child?

• Alternative to CompliantDataFrame.__getitem__ #2423

All of this is still pretty dependent on our internals:

narwhals/narwhals/dataframe.py

Lines 1116 to 1132 in 79c00c0

	compliant = self._compliant_frame
	if isinstance(columns, (int, str)):
	if isinstance(rows, int):
	return self.item(rows, columns)
	col_name = columns if isinstance(columns, str) else self.columns[columns]
	series = self.get_column(col_name)
	return series[rows] if rows is not None else series
	if isinstance(rows, Series):
	rows = rows._compliant_series
	if isinstance(columns, Series):
	columns = columns._compliant_series
	if rows is None:
	return self._with_compliant(compliant[:, columns])
	if columns is None:
	return self._with_compliant(compliant[rows, :])
	return self._with_compliant(compliant[rows, columns])

Since we don't define how CompliantDataFrame.__getitem__ should handle each of the cases in the signature:

narwhals/narwhals/_compliant/dataframe.py

Lines 123 to 129 in 79c00c0

	def __getitem__(
	self,
	item: tuple[
	SingleIndexSelector | MultiIndexSelector[CompliantSeriesT],
	MultiColSelector[CompliantSeriesT],
	],
	) -> Self: ...

An extension would realistically need to copy/paste this to ensure they're consistent with what we would return:

narwhals/narwhals/_compliant/dataframe.py

Lines 433 to 483 in 79c00c0

	def _gather(self, rows: SizedMultiIndexSelector[NativeSeriesT]) -> Self: ...
	def _gather_slice(self, rows: _SliceIndex | range) -> Self: ...
	def _select_multi_index(
	self, columns: SizedMultiIndexSelector[NativeSeriesT]
	) -> Self: ...
	def _select_multi_name(
	self, columns: SizedMultiNameSelector[NativeSeriesT]
	) -> Self: ...
	def _select_slice_index(self, columns: _SliceIndex | range) -> Self: ...
	def _select_slice_name(self, columns: _SliceName) -> Self: ...
	def __getitem__( # noqa: C901, PLR0912
	self,
	item: tuple[
	SingleIndexSelector | MultiIndexSelector[EagerSeriesT],
	MultiColSelector[EagerSeriesT],
	],
	) -> Self:
	rows, columns = item
	compliant = self
	if not is_slice_none(columns):
	if isinstance(columns, Sized) and len(columns) == 0:
	return compliant.select()
	if is_index_selector(columns):
	if is_slice_index(columns) or is_range(columns):
	compliant = compliant._select_slice_index(columns)
	elif is_compliant_series(columns):
	compliant = self._select_multi_index(columns.native)
	else:
	compliant = compliant._select_multi_index(columns)
	elif isinstance(columns, slice):
	compliant = compliant._select_slice_name(columns)
	elif is_compliant_series(columns):
	compliant = self._select_multi_name(columns.native)
	elif is_sequence_like(columns):
	compliant = self._select_multi_name(columns)
	else:
	assert_never(columns)
	if not is_slice_none(rows):
	if isinstance(rows, int):
	compliant = compliant._gather([rows])
	elif isinstance(rows, (slice, range)):
	compliant = compliant._gather_slice(rows)
	elif is_compliant_series(rows):
	compliant = compliant._gather(rows.native)
	elif is_sized_multi_index_selector(rows):
	compliant = compliant._gather(rows)
	else:
	assert_never(rows)
	return compliant