`unexpected_index_column_names` on `ValidationDefinition` level when using `Checkpoint`

[oleksandr-kinship]

I'm new in Gx framework and data management itself, tough trying to build data-quality framework on my project.

Initially im use all basic abstractions to build correct sequence of these to validate data in MSSQL,
At end all finished with run of Checkpoint that runs ValidationDefinition then Action, im also use really useful unexpected_index_column_names (with _id coumn) in result_format for Checkpoint.run() to include it into results and see in my dataDoc html.

It works perfect when I have 1 table (i mean all set of abstractions that end with VD), then added second again all great.
But when I have added a third table that doesn't have _id column it's failed
Then I start finding the way how to define unexpected_index_column_names on table level, I have found that I can define it only in run() function for Checkpoint, VD or batch.

It means if I want to use Checkpoint I have to include there only VDs with tables that have my column from unexpected_index_column_names. otherwise check for tables that not include the index column will be errored.

So if miss something let me know, but I would use full cycle of abstractions with predefined actions, but for that I need to use Checkpoint, but I can't skip unexpected_index_column_names it looks really useful

Describe the solution you'd like
I would like to set unexpected_index_column_names on any of table level abstractions

Describe alternatives you've considered
Right now im use loop with run() for all my VD, to each add result_format with unexpected_index_column_names correct for each table that using. After that separately build site context.build_data_docs(site_names=site_name) (with no action, before im used gx.checkpoint.actions.UpdateDataDocsAction)

[adeola-ak]

hi, thanks for the detailed explanation. unexpected_index_column_names is part of result_format, and when you run a Checkpoint, that result_format is applied uniformly to all ValidationDefinitions executed by that checkpoint. If a checkpoint includes multiple ValidationDefinitions and one or more of the underlying tables does not contain the specified index column, the validation will fail for those tables

At the moment, Great Expectations does not support defining unexpected_index_column_names at the table / asset / ValidationDefinition level in a persisted way.

you may wish to split checkpoints by schema by creating separate checkpoints for:

-Tables that include _id (and use unexpected_index_column_names=["_id"])
-Tables that do not include that column

or run validations individually

[oleksandr-kinship]

Thanks @adeola-ak , yes, I think that's exactly what I discovered,
That's a bit unfortunate because I built automation that's bulk grap many tables for creating some basic checks sets in my case additional conditions just add unnecessity complexity.
As workaround I just decided end up on Validation Definition abstraction, so Im to execute them with individual result_format. It's means I need separately implement actions and some other hand abstraction, it's not critical.

But you might consider is it worth to include in some backlog as improvement, workarounds looks kind of lame.

Anyway, thanks for your response and confirming my results.