refactor: pass `ExprMetadata` down to compliant Exprs from narwhals.Expr

[MarcoGorelli]

Currently, we track expression metadata at the compliant level:

• whether it aggregates, changes length, transforms
• the function name

This is quite error-prone, as we repeat logic in quite a few places.

I'd prefer it if:

• we didn't track the function name at all for most backends. It's only really needed for pandas/pyarrow/dask because of limitations in their group-by APIs, so we should only do it there
• one of the metadatas we could track is what kind of column selection we start with (nth, col, selector, all). Because in group_by-agg, selector and all require some special treatment. But this could be done much more simply at the Narwhals level, without tracking function names everywhere in DuckDB / PySpark / Ibis / whatever else we add, which I hope would also have a modern enough syntax

To do this, I haven't tried, but perhaps:

• _to_compliant_expr could take a metadata argument, which takes a TypeDict of metadata which we keep track of at the narwhals/expr.py and narwhals/functions.py level
• compliant expressions just read from the metadata passed by above to determine how and whether to broadcast

[dangotbanned]

Think I've come across this in #2149 (comment)

For example, in Expr.null_count we know that it is ExprKind.AGGREGATION.
But we repeat that as returns_scalar=True and then use that as a conditional in multiple places.

narwhals/narwhals/expr.py

Lines 1623 to 1652 in 1e45855

	def null_count(self: Self) -> Self:
	r"""Count null values.
	Returns:
	A new expression.
	Notes:
	pandas handles null values differently from Polars and PyArrow.
	See [null_handling](../pandas_like_concepts/null_handling.md/)
	for reference.
	Examples:
	>>> import pandas as pd
	>>> import narwhals as nw
	>>> df_native = pd.DataFrame(
	... {"a": [1, 2, None, 1], "b": ["a", None, "b", None]}
	... )
	>>> df = nw.from_native(df_native)
	>>> df.select(nw.all().null_count())
	┌──────────────────┐
	|Narwhals DataFrame|
	|------------------|
	| a b |
	| 0 1 2 |
	└──────────────────┘
	"""
	return self.__class__(
	lambda plx: self._to_compliant_expr(plx).null_count(),
	self._metadata.with_kind(ExprKind.AGGREGATION),
	)

Gets even trickier with ArrowExpr as that turns it into _return_py_scalar for ArrowSeries

narwhals/narwhals/_expression_parsing.py

Lines 160 to 163 in 1e45855

	# For PyArrow.Series, we return Python Scalars (like Polars does) instead of PyArrow Scalars.
	# However, when working with expressions, we keep everything PyArrow-native.
	extra_kwargs = (
	{"_return_py_scalar": False}

narwhals/narwhals/_arrow/series.py

Lines 532 to 534 in 1e45855

	def null_count(self: Self, *, _return_py_scalar: bool = True) -> int:
	return maybe_extract_py_scalar(self.native.null_count, _return_py_scalar)

Eventually resolving it in maybe_extract_py_scalar

narwhals/narwhals/_arrow/series.py

Lines 91 to 96 in 1e45855

	def maybe_extract_py_scalar(value: Any, return_py_scalar: bool) -> Any: # noqa: FBT001
	if TYPE_CHECKING:
	return value.as_py()
	if return_py_scalar:
	return getattr(value, "as_py", lambda: value)()
	return value

[dangotbanned]

I know things have changed somewhat since the issue has opened - I think some window functions are already passing kind down?

[MarcoGorelli]

yup - but you're right, we might be able to use this for the pyarrow scalar stuff too