Skip to content

Set default Check n_failure_cases to None and add documentation for n_failure_cases behavior #577

Description

@jgirault-qs

Describe the bug
pa.errors.SchemaErrors.failure_cases only returns the first 10 failure_cases

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of pandera. 0.6.5
  • (optional) I have confirmed this bug exists on the master branch of pandera.

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
import pandera as pa

df = pd.DataFrame({'n': range(20)})
schema = pa.DataFrameSchema({
    "n": pa.Column(pa.Int, pa.Check.greater_than(30))
})
try:
    schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as err:
    print(err.failure_cases)

Expected behavior

err.failure_cases should be 20 lines long

Usage

Using pandera for data validation at the start of data processing. When errors appear I want the clean data to be able to continue the pipeline, while corrupted data is removed from the dataframe and indexes stored somewhere so I can fix the issue and plan a recovery pipeline run later.
Was trying the following code when I discovered the issue. (I've got MultiIndexes)

import pandas as pd
import pandera as pa

df = pd.DataFrame({'n': range(20), 'a': range(20), 'b': range(20)}).set_index(['a', 'b'])
schema = pa.DataFrameSchema({
   "n": pa.Column(pa.Int, pa.Check.greater_than(30))
})

try:
   schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as err:
   f = err.failure_cases  # dataframe of schema errors
   d = err.data  # invalid dataframe
   df_failures = pd.DataFrame(err.failure_cases['index'].apply(literal_eval).values.tolist(), columns=err.data.index.names)
   print(df_failures)

df_failures could contain the indexes that I need to remove the corrupted data from the dataframe and store to prepare the recovery run, but it only has the first 10.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinggood first issueGood for newcomershelp wantedExtra attention is needed

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions