Add spec for generic dataframe-agnostic schema API - #2401
Open
cosmicbboy wants to merge 3 commits into
Open
cosmicbboy wants to merge 3 commits into
cosmicbboy wants to merge 3 commits into
Conversation
Design document for a dataframe-agnostic DataFrameSchema/DataFrameModel API built on the existing narwhals backend and engine: dynamic backend resolution keyed on narwhals compatibility, mypy/pyright typing story, native-typed custom checks, a three-tier frame-type registration model, a phased roadmap for replacing the per-library schema implementations, and a generic NDArrayLike protocol for the xarray API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generate-and-paste pyright workflow was removed from the fix-series-typing branch as poor devex; the spec now scopes column-level inference to the mypy plugin, with pyright relying on plugin-free generics only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Updated the typing section (§5): removed the |
The generic (narwhals-default) DataFrameSchema/DataFrameModel API takes over pandera/api/dataframe/; the abstract base classes currently there move to a new private package, pandera/api/_base_dataframe/. Public accessor becomes pandera.dataframe instead of pandera.narwhals. Adds §3.5 documenting the migration mechanics of the takeover: internal import rewrite (~20 modules), mypy-plugin fullname updates, and the isinstance-compatibility caveat for external imports of the old paths until Phase 3 re-parents the per-library classes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
specs/generic-dataframe.md, a design document for a dataframe-agnosticDataFrameSchema/DataFrameModelAPI built on the existing narwhals backend (pandera/backends/narwhals/) and engine (pandera/engines/narwhals_engine.py).Today narwhals is wired in only as an alternate validation backend beneath the polars/ibis/pyspark schema classes (
use_narwhals_backend). This spec defines the missing top half — a generic schema/model API surface, its typing story, and the roadmap by which it eventually replaces the per-library schema implementations.Key design points:
pandera/api/dataframe/with public accessorpandera.dataframe; the abstract base classes currently inpandera/api/dataframe/move to a new private packagepandera/api/_base_dataframe/(migration mechanics in §3.5: internal import rewrite, mypy-plugin fullname updates, isinstance-compatibility caveat). The existing narwhals backend is reused unmodified; the backend remains customizable through the existingBACKEND_REGISTRYmechanism.validate()(TFrame -> TFrame), mypy plugin fullname extensions, pyright-clean generics — following the approach from the static-typing work onnielsb/fix-series-typing(Improve DataFrame/Series static typing for mypy and Pyright #2398). Column-level inference is a mypy-plugin feature.NarwhalsData/nw.Exprchecks by default, plus native-typed check signatures (e.g.pl.LazyFrame) with boundary conversion and singledispatch-style per-library overrides.register_frame_type(...), full backend override — plus apandera.frame_typesentry-point group.Index/MultiIndexproblem explicitly flagged and options laid out.NDArrayLikeprotocol +NDArray[Model]annotation for the xarray API, so xarray-like libraries can reuseDataArraySchema.Open questions are listed at the end of the spec (base-class naming/module placement, error-report unification, entry-point security posture, strategies scope, narwhals version policy).
Spec only — no code changes.
🤖 Generated with Claude Code