Describe the bug
pa.errors.SchemaErrors.failure_cases only returns the first 10 failure_cases
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
import pandas as pd
import pandera as pa
df = pd.DataFrame({'n': range(20)})
schema = pa.DataFrameSchema({
"n": pa.Column(pa.Int, pa.Check.greater_than(30))
})
try:
schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as err:
print(err.failure_cases)
Expected behavior
err.failure_cases should be 20 lines long
Usage
Using pandera for data validation at the start of data processing. When errors appear I want the clean data to be able to continue the pipeline, while corrupted data is removed from the dataframe and indexes stored somewhere so I can fix the issue and plan a recovery pipeline run later.
Was trying the following code when I discovered the issue. (I've got MultiIndexes)
import pandas as pd
import pandera as pa
df = pd.DataFrame({'n': range(20), 'a': range(20), 'b': range(20)}).set_index(['a', 'b'])
schema = pa.DataFrameSchema({
"n": pa.Column(pa.Int, pa.Check.greater_than(30))
})
try:
schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as err:
f = err.failure_cases # dataframe of schema errors
d = err.data # invalid dataframe
df_failures = pd.DataFrame(err.failure_cases['index'].apply(literal_eval).values.tolist(), columns=err.data.index.names)
print(df_failures)
df_failures could contain the indexes that I need to remove the corrupted data from the dataframe and store to prepare the recovery run, but it only has the first 10.
Describe the bug
pa.errors.SchemaErrors.failure_cases only returns the first 10 failure_cases
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
Expected behavior
err.failure_cases should be 20 lines long
Usage
Using pandera for data validation at the start of data processing. When errors appear I want the clean data to be able to continue the pipeline, while corrupted data is removed from the dataframe and indexes stored somewhere so I can fix the issue and plan a recovery pipeline run later.
Was trying the following code when I discovered the issue. (I've got MultiIndexes)
df_failures could contain the indexes that I need to remove the corrupted data from the dataframe and store to prepare the recovery run, but it only has the first 10.