Skip to content

Fix optional and nullable fields validation in PydanticModel - #2410

Open
Apuravag07 wants to merge 2 commits into
unionai-oss:mainfrom
Apuravag07:fix-pydantic-optional-fields
Open

Fix optional and nullable fields validation in PydanticModel#2410
Apuravag07 wants to merge 2 commits into
unionai-oss:mainfrom
Apuravag07:fix-pydantic-optional-fields

Conversation

@Apuravag07

@Apuravag07 Apuravag07 commented Jul 9, 2026

Copy link
Copy Markdown

This PR fixes a bug where DataFrame validation fails on optional/nullable fields in a PydanticModel because columns built dynamically during DataFrameSchema.init defaulted to required=True and nullable=False. By extracting metadata from the pydantic model's fields, we now correctly map their required and nullable attributes.

@RedZapdos123

RedZapdos123 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Do the DCO sign off.

@Apuravag07
Fix: correctly handle optional/nullable fields in PydanticModel
e36a7ef 
Signed-off-by: Anant Agrawal <apuravagrawal07@gmail.com>
@Apuravag07
Apuravag07 force-pushed the fix-pydantic-optional-fields branch from 75016b6 to e36a7ef Compare July 12, 2026 18:26
@codecov

codecov Bot commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.21%. Comparing base (5f20ac1) to head (e36a7ef).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
pandera/engines/pandas_engine.py 95.23% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2410      +/-   ##
==========================================
+ Coverage   90.92%   91.21%   +0.28%     
==========================================
  Files         192      192              
  Lines       17121    17145      +24     
==========================================
+ Hits        15568    15638      +70     
+ Misses       1553     1507      -46
Flag Coverage Δ
unit-tests-base-Linux-py3.10 15.38% <12.50%> (-0.02%) ⬇️
unit-tests-base-Linux-py3.11 15.38% <12.50%> (-0.02%) ⬇️
unit-tests-base-Linux-py3.12 15.38% <12.50%> (-0.02%) ⬇️
unit-tests-base-Linux-py3.13 15.38% <12.50%> (-0.02%) ⬇️
unit-tests-base-Linux-py3.14 15.25% <12.50%> (-0.02%) ⬇️
unit-tests-base-Windows-py3.10 15.30% <12.50%> (-0.02%) ⬇️
unit-tests-base-Windows-py3.11 15.30% <12.50%> (-0.02%) ⬇️
unit-tests-base-Windows-py3.12 15.30% <12.50%> (-0.02%) ⬇️
unit-tests-base-Windows-py3.13 15.30% <12.50%> (-0.02%) ⬇️
unit-tests-base-Windows-py3.14 15.17% <12.50%> (-0.02%) ⬇️
unit-tests-base-macOS-py3.10 15.32% <12.50%> (-0.02%) ⬇️
unit-tests-base-macOS-py3.11 15.32% <12.50%> (-0.02%) ⬇️
unit-tests-base-macOS-py3.12 15.32% <12.50%> (-0.02%) ⬇️
unit-tests-base-macOS-py3.13 15.32% <12.50%> (-0.02%) ⬇️
unit-tests-base-macOS-py3.14 15.18% <12.50%> (-0.02%) ⬇️
unit-tests-dask-Linux-py3.10-pandas2.3.3-polars 24.23% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Linux-py3.11-pandas2.3.3-polars 24.23% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Linux-py3.12-pandas2.3.3-polars 24.23% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Linux-py3.13-pandas2.3.3-polars 24.23% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Linux-py3.14-pandas2.3.3-polars 24.14% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Windows-py3.10-pandas2.3.3-polars 24.14% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Windows-py3.11-pandas2.3.3-polars 24.14% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Windows-py3.12-pandas2.3.3-polars 24.14% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Windows-py3.13-pandas2.3.3-polars 24.14% <12.50%> (-0.03%) ⬇️
unit-tests-dask-Windows-py3.14-pandas2.3.3-polars 24.05% <12.50%> (-0.03%) ⬇️
unit-tests-fastapi-Linux-py3.11-pandas3.0.0-pydantic2.12.3 24.73% <12.50%> (-0.02%) ⬇️
unit-tests-fastapi-Linux-py3.12-pandas2.3.3-pydantic2.12.3 24.66% <12.50%> (-0.03%) ⬇️
unit-tests-fastapi-Linux-py3.12-pandas3.0.0-pydantic2.12.3 24.73% <12.50%> (-0.01%) ⬇️
unit-tests-fastapi-Linux-py3.13-pandas2.3.3-pydantic2.12.3 24.66% <12.50%> (-0.01%) ⬇️
unit-tests-fastapi-Linux-py3.14-pandas2.3.3-pydantic2.12.3 24.58% <12.50%> (-0.01%) ⬇️
unit-tests-fastapi-Windows-py3.10-pandas2.3.3-pydantic2.12.3 24.58% <12.50%> (-0.02%) ⬇️
unit-tests-geopandas-Linux-py3.10-pandas2.3.3-pydantic2.12.3 30.04% <12.50%> (-0.04%) ⬇️
unit-tests-geopandas-Linux-py3.11-pandas2.3.3-pydantic2.12.3 30.04% <12.50%> (-0.04%) ⬇️
unit-tests-geopandas-Linux-py3.11-pandas3.0.0-pydantic2.12.3 30.04% <12.50%> (-0.03%) ⬇️
unit-tests-geopandas-Linux-py3.12-pandas2.3.3-pydantic2.12.3 30.04% <12.50%> (-0.04%) ⬇️
unit-tests-geopandas-Linux-py3.12-pandas3.0.0-pydantic2.12.3 30.04% <12.50%> (-0.02%) ⬇️
unit-tests-geopandas-Linux-py3.13-pandas2.3.3-pydantic2.12.3 30.04% <12.50%> (-0.04%) ⬇️
unit-tests-geopandas-Linux-py3.13-pandas3.0.0-pydantic2.12.3 30.04% <12.50%> (-0.04%) ⬇️
unit-tests-geopandas-Windows-py3.11-pandas2.3.3-pydantic2.12.3 29.94% <12.50%> (-0.03%) ⬇️
unit-tests-hypotheses-Linux-py3.10-pandas2.3.3-pydantic2.12.3 22.47% <12.50%> (+0.01%) ⬆️
unit-tests-hypotheses-Linux-py3.11-pandas2.3.3-pydantic2.12.3 22.47% <12.50%> (-0.03%) ⬇️
unit-tests-hypotheses-Linux-py3.11-pandas3.0.0-pydantic2.12.3 22.51% <12.50%> (+<0.01%) ⬆️
unit-tests-hypotheses-Linux-py3.12-pandas2.3.3-pydantic2.12.3 22.47% <12.50%> (-0.03%) ⬇️
unit-tests-hypotheses-Linux-py3.12-pandas3.0.0-pydantic2.12.3 22.51% <12.50%> (-0.01%) ⬇️
unit-tests-hypotheses-Linux-py3.13-pandas2.3.3-pydantic2.12.3 22.46% <12.50%> (-0.03%) ⬇️
unit-tests-hypotheses-Linux-py3.13-pandas3.0.0-pydantic2.12.3 22.51% <12.50%> (-0.01%) ⬇️
unit-tests-hypotheses-Linux-py3.14-pandas2.3.3-pydantic2.12.3 22.37% <12.50%> (-0.03%) ⬇️
unit-tests-hypotheses-Linux-py3.14-pandas3.0.0-pydantic2.12.3 22.41% <12.50%> (+0.03%) ⬆️
unit-tests-hypotheses-Windows-py3.10-pandas2.3.3-pydantic2.12.3 22.40% <12.50%> (-0.01%) ⬇️
unit-tests-hypotheses-Windows-py3.11-pandas2.3.3-pydantic2.12.3 22.40% <12.50%> (-0.03%) ⬇️
unit-tests-ibis-Linux-py3.10-pandas2.3.3-polars 28.82% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Linux-py3.11-pandas2.3.3-polars 28.82% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Linux-py3.12-pandas2.3.3-polars 28.82% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Linux-py3.13-pandas2.3.3-polars 28.82% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Linux-py3.14-pandas2.3.3-polars 28.63% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Windows-py3.10-pandas2.3.3-polars 28.75% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Windows-py3.11-pandas2.3.3-polars 28.75% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Windows-py3.12-pandas2.3.3-polars 28.75% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Windows-py3.13-pandas2.3.3-polars 28.75% <12.50%> (-0.04%) ⬇️
unit-tests-ibis-Windows-py3.14-pandas2.3.3-polars 28.56% <12.50%> (-0.04%) ⬇️
unit-tests-io-Linux-py3.10-pandas2.3.3-pydantic2.12.3 38.05% <12.50%> (-0.05%) ⬇️
unit-tests-io-Linux-py3.11-pandas2.3.3-pydantic2.12.3 38.09% <12.50%> (-0.05%) ⬇️
unit-tests-io-Linux-py3.11-pandas3.0.0-pydantic2.12.3 38.06% <12.50%> (-0.05%) ⬇️
unit-tests-io-Linux-py3.12-pandas2.3.3-pydantic2.12.3 38.09% <12.50%> (-0.05%) ⬇️
unit-tests-io-Linux-py3.12-pandas3.0.0-pydantic2.12.3 38.06% <12.50%> (-0.05%) ⬇️
unit-tests-io-Linux-py3.13-pandas2.3.3-pydantic2.12.3 38.09% <12.50%> (-0.05%) ⬇️
unit-tests-io-Linux-py3.13-pandas3.0.0-pydantic2.12.3 38.06% <12.50%> (?)
unit-tests-io-Linux-py3.14-pandas2.3.3-pydantic2.12.3 37.98% <12.50%> (-0.04%) ⬇️
unit-tests-io-Windows-py3.10-pandas2.3.3-pydantic2.12.3 37.69% <12.50%> (-0.05%) ⬇️
unit-tests-io-Windows-py3.11-pandas3.0.0-pydantic2.12.3 37.71% <12.50%> (-0.05%) ⬇️
unit-tests-modin-dask-Linux-py3.10-pandas2.3.3-polars 30.66% <12.50%> (-0.04%) ⬇️
unit-tests-modin-dask-Linux-py3.12-pandas2.3.3-polars 30.66% <12.50%> (-0.01%) ⬇️
unit-tests-modin-dask-Windows-py3.10-pandas2.3.3-polars 30.57% <12.50%> (-0.04%) ⬇️
unit-tests-modin-ray-Linux-py3.10-pandas2.3.3-polars 30.64% <12.50%> (-0.04%) ⬇️
unit-tests-modin-ray-Linux-py3.11-pandas2.3.3-polars 30.64% <12.50%> (-0.04%) ⬇️
unit-tests-modin-ray-Linux-py3.12-pandas2.3.3-polars 30.64% <12.50%> (-0.04%) ⬇️
unit-tests-mypy-Linux-py3.10-pandas2.3.3-pydantic2.12.3 23.43% <12.50%> (-0.02%) ⬇️
unit-tests-mypy-Linux-py3.11-pandas2.3.3-pydantic2.12.3 23.43% <12.50%> (+0.03%) ⬆️
unit-tests-mypy-Linux-py3.11-pandas3.0.0-pydantic2.12.3 23.50% <12.50%> (?)
unit-tests-mypy-Linux-py3.12-pandas2.3.3-pydantic2.12.3 23.43% <12.50%> (-0.03%) ⬇️
unit-tests-narwhals-Linux-py3.10 28.23% <12.50%> (-0.03%) ⬇️
unit-tests-narwhals-Linux-py3.11 28.23% <12.50%> (-0.03%) ⬇️
unit-tests-narwhals-Linux-py3.12 28.23% <12.50%> (-0.03%) ⬇️
unit-tests-narwhals-Linux-py3.13 28.23% <12.50%> (-0.03%) ⬇️
unit-tests-narwhals-Linux-py3.14 28.01% <12.50%> (-0.03%) ⬇️
unit-tests-narwhals-backend-ibis-py3.10 31.77% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-ibis-py3.11 31.66% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-ibis-py3.12 31.66% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-ibis-py3.13 31.66% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-ibis-py3.14 31.45% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-polars-py3.10 34.40% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-polars-py3.11 34.39% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-polars-py3.12 34.39% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-polars-py3.13 34.39% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-polars-py3.14 34.24% <12.50%> (-0.04%) ⬇️
unit-tests-narwhals-backend-pyspark-py3.10 43.42% <12.50%> (?)
unit-tests-narwhals-backend-pyspark-py3.11 43.45% <12.50%> (?)
unit-tests-pandas-Linux-py3.10-pandas2.3.3-pydantic1.10.11 42.79% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Linux-py3.11-pandas2.3.3-pydantic1.10.11 42.79% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Linux-py3.11-pandas3.0.0-pydantic1.10.11 42.81% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Linux-py3.12-pandas2.3.3-pydantic1.10.11 42.78% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Linux-py3.12-pandas3.0.0-pydantic1.10.11 42.79% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Linux-py3.13-pandas2.3.3-pydantic1.10.11 42.78% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Linux-py3.13-pandas3.0.0-pydantic1.10.11 42.79% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Linux-py3.14-pandas2.3.3-pydantic2.12.3 42.97% <83.33%> (+0.06%) ⬆️
unit-tests-pandas-Linux-py3.14-pandas3.0.0-pydantic2.12.3 42.98% <83.33%> (+0.06%) ⬆️
unit-tests-pandas-Windows-py3.10-pandas2.3.3-pydantic1.10.11 42.71% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Windows-py3.11-pandas2.3.3-pydantic1.10.11 42.71% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Windows-py3.11-pandas3.0.0-pydantic1.10.11 42.72% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Windows-py3.12-pandas2.3.3-pydantic1.10.11 42.69% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Windows-py3.12-pandas3.0.0-pydantic1.10.11 42.71% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Windows-py3.13-pandas2.3.3-pydantic1.10.11 42.69% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Windows-py3.13-pandas3.0.0-pydantic1.10.11 42.71% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-Windows-py3.14-pandas2.3.3-pydantic2.12.3 42.88% <83.33%> (+0.06%) ⬆️
unit-tests-pandas-Windows-py3.14-pandas3.0.0-pydantic2.12.3 42.90% <83.33%> (+0.06%) ⬆️
unit-tests-pandas-macOS-py3.10-pandas2.3.3-pydantic1.10.11 42.73% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-macOS-py3.11-pandas2.3.3-pydantic1.10.11 42.73% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-macOS-py3.11-pandas3.0.0-pydantic1.10.11 42.74% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-macOS-py3.12-pandas2.3.3-pydantic1.10.11 42.72% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-macOS-py3.12-pandas3.0.0-pydantic1.10.11 42.73% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-macOS-py3.13-pandas2.3.3-pydantic1.10.11 42.72% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-macOS-py3.13-pandas3.0.0-pydantic1.10.11 42.73% <50.00%> (+0.01%) ⬆️
unit-tests-pandas-macOS-py3.14-pandas2.3.3-pydantic2.12.3 42.91% <83.33%> (+0.06%) ⬆️
unit-tests-pandas-macOS-py3.14-pandas3.0.0-pydantic2.12.3 42.92% <83.33%> (+0.06%) ⬆️
unit-tests-polars-Linux-py3.10-pandas2.3.3-polars1.33.1 31.00% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Linux-py3.11-pandas2.3.3-polars1.33.1 30.99% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Linux-py3.12-pandas2.3.3-polars1.33.1 30.99% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Linux-py3.13-pandas2.3.3-polars1.33.1 30.99% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Linux-py3.14-pandas2.3.3-polars1.33.1 30.86% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Windows-py3.10-pandas2.3.3-polars1.33.1 30.92% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Windows-py3.11-pandas2.3.3-polars1.33.1 30.91% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Windows-py3.12-pandas2.3.3-polars1.33.1 30.91% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Windows-py3.13-pandas2.3.3-polars1.33.1 30.91% <12.50%> (-0.04%) ⬇️
unit-tests-polars-Windows-py3.14-pandas2.3.3-polars1.33.1 30.79% <12.50%> (-0.04%) ⬇️
unit-tests-pyspark-Linux-py3.10-pandas2.3.3-polars 38.96% <12.50%> (?)
unit-tests-pyspark-Linux-py3.11-pandas2.3.3-polars 38.96% <12.50%> (?)
unit-tests-strategies-Linux-py3.10-pandas2.3.3-pydantic2.12.3 34.14% <12.50%> (-0.03%) ⬇️
unit-tests-strategies-Linux-py3.11-pandas2.3.3-pydantic2.12.3 34.14% <12.50%> (-0.03%) ⬇️
unit-tests-strategies-Linux-py3.11-pandas3.0.0-pydantic2.12.3 34.13% <12.50%> (-0.04%) ⬇️
unit-tests-strategies-Linux-py3.12-pandas2.3.3-pydantic2.12.3 34.14% <12.50%> (?)
unit-tests-strategies-Linux-py3.14-pandas3.0.0-pydantic2.12.3 34.08% <12.50%> (-0.05%) ⬇️
unit-tests-xarray-Linux-py3.10-pandas2.3.3-polars 31.72% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Linux-py3.11-pandas2.3.3-polars 31.72% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Linux-py3.12-pandas2.3.3-polars 31.72% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Linux-py3.13-pandas2.3.3-polars 31.72% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Linux-py3.14-pandas2.3.3-polars 31.65% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Windows-py3.10-pandas2.3.3-polars 31.65% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Windows-py3.11-pandas2.3.3-polars 31.65% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Windows-py3.12-pandas2.3.3-polars 31.65% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Windows-py3.13-pandas2.3.3-polars 31.65% <12.50%> (-0.04%) ⬇️
unit-tests-xarray-Windows-py3.14-pandas2.3.3-polars 31.58% <12.50%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Apuravag07
Fix: check types.UnionType for PEP 604 unions in Pydantic fields meta…
68cc64c 
…data

Signed-off-by: Anant Agrawal <apuravagrawal07@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Apuravag07 @RedZapdos123