[discussion]: Which quality checks should we perform on the converted data?

[signekb]

To ensure that the conversion has gone as expected, we should perform some quality checks to compare the data in the original SAS files with the converted Parquet files. The question is: Which checks?

I suggest, per register (not per SAS file):

• nrows
• colnames (will be lowercased in the Parquets + source_file and year is added in Parquets)
• col types (using mapping from create_arrow_schema())
• values:
  • categorical cols: how many of each unique value + na count
  • numeric cols: min, max, mean, median, quantiles. na count
  • datetime cols: min, max, na count
• other checks?

Questions:

• How to do this in the easiest way? Add generic function to do these checks?
• How to determine categorial cols (with n_unique_values <= x?)

[lwjohnst86]

Hmm, sounds like some logging utility would be helpful here, to record this type of data as each file is converted and store that QC data somewhere to compare with the output Parquet files... 🤔 Will have to think about that 🤔

[signekb]

So, we might want to add some logging to the targets pipeline and then output some kind of "report" file to be inspected manually afterwards?

[lwjohnst86]

Yea, that might be useful. Maybe low priority though

[signekb]

Hm, I would say high priority, since if we expect fastreg to be usable, then we should ensure that no data is lost of changed in the conversion?

[lwjohnst86]

Hmm, yea, I'm just hesitate to go down that road as I think it might lead to lots of issues... It's important for sure, but maybe not our responsibility..? I had started to go down this path when I first started writing the initial version of this code when working on DST, but it was not pleasant. So people have currently been using the Parquet as is with this limitation (and so far I haven't heard any issues). I'm just worried, what will it mean if we find that different years within a register have different data types..? How would we solve that? Who would actually know? How much time would it take to figure out? Lots of unknowns there for this being a collaborating project where we don't have a lot of control on the downstream data source. It is a constant battle between them making potential changes and us not knowing what those changes mean or even if there were changes...

[signekb]

That's some good points. My suggestion would be to add some simple logging in a report.txt file (or the like), maybe through a param in the convert_file() function for every chunk (both SAS and Parquet). Then, that can be inspected manually by e.g., data managers.
We could add a note in the "report" that if they find a difference or a bug, they can file an Issue.

[lwjohnst86]

That's a smart idea 👍

[signekb]

Great. Will give that a go 👍