Performance

[cegladanych]

Hi

We have started to use DQX for data validation and first impression is that is really slow, our process has slowed down about 10 times when implemented apply_checks_by_metadata_and_split(). I am curious is it us or there is generic problem with the tool?

Does anyone has similar issue?

[mwojtyczka]

It's the first time a performance issue is reported. We are using native Spark Core APIs, no UDFs so the performance impact should be minimal. We have customers using DQX with PTs of data. Can you share a bit more regarding your setup?

We plan to publish performance benchmarks.

[cegladanych]

I will double check, and run few test to monitor what is going on.

[cegladanych]

I know what I did wrong, I added validation as additional step to my previous transformations, chaining them toghether, and this caused in slow down, there are too many transformations at the same time. Instead I use dqx as separate step to read from disk and writing validated data that way there is not performance penalty. To read 5milion rows from disk validation and write back it takes 30 or so seconds.

[mwojtyczka]

we are working on creating perf benchmarks: #56 (comment)