Resources on data validation in the full data science pipelines

[chendaniely]

We've kind of used unit tests to help validate things in the course, but those were mainly single assert or testthat cases in a script or part of a package's unit testing workflow.

We should also use tools specifically to validate our data to test our data's assumptions (e.g., no missing values, has a specific distribution, etc)

• R: https://github.com/rstudio/pointblank
• Python: https://pandera.readthedocs.io/en/stable/

Posit conf this past year had 2 workshops in r and python that might be nice reference material as well:

• R: https://github.com/posit-conf-2023/ds-workflows-r
• Python: https://github.com/posit-conf-2023/ds-workflows-python