Create documentation about data validation

[noklam]

Description

Add a dedicated page discuss Data Validation, and different options in Kedro.

Context

These questions has been ask repeatly:

• What are the current status of kedro-great? It seems unmaintained
• What are Kedro's opinion about GE or pandera, which one is the go-to plugin?
• Should I use Pydantic?
• How to validate config, type checking etc?

Kedro is all about best practice for data/ML project, and data validation is no longer an optional thing. The current status is that we have some mention about Great Expectation, a sample hook, unmaintained plugin and not-so-active plugin in the wild. While we cannot give a default path for users, it would be still beneficial to discuss different options and tradeoff to provide some guidance and let users make their own choice.

Pages:

• https://docs.kedro.org/en/stable/hooks/index.html
• https://docs.kedro.org/en/stable/hooks/examples.html#v2-api
• https://docs.kedro.org/en/stable/hooks/examples.html#v3-api

Plugin:

• https://pypi.org/project/kedro-great/ (unmaintain, last update 2020)
• https://pypi.org/project/kedro-expectations/ (actively maintained, but we don't know much about it)
• https://pypi.org/project/kedro-great-expectations/ (from @deepyaman but at version 0.0.1 2023)
• kedro-pandera https://github.com/Galileo-Galilei/kedro-pandera, last update Jul 2024, somewhat active but feature are limited

It's also important to keep in mind that plugin/libraries are one of the options but not the only one, it's still possible to write custom function to do assertion, or even do it as a unit test (with pytest for example)

[rashidakanchwala]

Hi team,

Just wanted to align with you all on the direction we’re taking for the data validation documentation.

We’re planning to include the following examples/tutorials:

• Data validation via Pandera and Hooks (Issue - Add documentation for Pandera Integration #5194)

  • Demonstrates how to integrate Pandera validation logic using Kedro hooks. (PR in review- Add docs on working with Pandera for data validation #5142 )

• Data validation via Great Expectations and Hooks (Issue - Add documentation for Great Expectations integration #5170)

  • Mirrors the Pandera example closely, showing an equivalent workflow using Great Expectations for consistency and easy comparison. (PR in review - Add docs on working with Great Expectations for data validation #5181)

• Data validation via a Separate Pipeline (Pandera-based, no hooks)

  • Useful for Spark or parallel runners. Felipe Oliveira shared that using hooks in these scenarios can introduce some caveats — for instance, running data validation within hooks may cause unnecessary re-execution of Spark transformations or create performance bottlenecks, since each validation rule can trigger additional Spark actions.
  • A separate validation pipeline keeps processing and validation decoupled and avoids chaining multiple Spark actions during node execution.

Felipe will support this example, as it reflects a pattern already used by some verticals where hooks are avoided for scalability and parallel execution efficiency. He’s also happy to help us write these docs (or a blog post), as Nok suggested.

• Data validation via Pure Hooks (no external integrations)
  • Optional example for users who prefer not to install additional dependencies.
  • Please let us know if this one makes sense to include.

---

We also noticed a user suggestion about adding a GreatExpectationsDataset example ((#5181 (comment)_) wrapping an existing dataset to apply expectations before saving. We hadn’t considered this approach yet, so we’d appreciate your thoughts on whether to include it or to focus on the four above.

Given the number of possible data validation patterns, our aim is to keep the documentation cohesive and avoid overwhelming users with too many parallel approaches.

Would love your feedback especially on whether we should include the dataset wrapper example or stick to the current plan.

@deepyaman @datajoely @noklam

cc - @merelcht , @lrcouto , @AliceCima10

[datajoely]

So there are two different ways of doing data checks:

1. Data at rest offline (Great expectations, similar to dbt test)
2. Runtime (Pandera)

The latter is supported already because it works on a Python level not a data level. I'm actually against a GreatExpectationDataSet since it would require the user to essentially duplicate another existing dataset pointing to the same path.

I would much prefer that Kedro exposes a generic checks API which allows users to define:

1. Offline checks on persisted data or when data is written/loaded.
2. Runtime checks similar to pandera.

This should integrate into the Kedro run lifecycle and be exposed in Kedro Viz. The difference here is that Kedro doesn't care how a dataset / parameter etc is checked (the mechanism is an implementation detail), but it has a consistent way of exposing checks to the user. I would consider shipping 1st part Pandera / GE plugins down the line.

[datajoely]

Dagster also has a similar idea
https://docs.dagster.io/dagster-basics-tutorial/asset-checks

[lrcouto]

My feeling re: data validation is that, because of the way Kedro's execution flow is structured, it'd make more sense for any kind of validation, be it Great Expectations, Pandera or something else, to exist withing a pipeline run.

The data validation becomes part of the same data flow that includes loading from the catalog, processing, etc. rather than something that happens "on the side".

This should integrate into the Kedro run lifecycle and be exposed in Kedro Viz. The difference here is that Kedro doesn't care how a dataset / parameter etc is checked (the mechanism is an implementation detail), but it has a consistent way of exposing checks to the user.

This is something I was thinking about too, having this visibility would be good.