External use of sqllogictest machinery?

[tshauck]

Hi -- I was wondering if there were thoughts on how one might be able to use some of the code in sqllogictest, but externally?

For context, I have a number of UDFs I've developed for specific domains (e.g. SELECT gc_content('ATCG') for the GC content of a DNA sequence).

To the end of testing the registered UDFs, I was wondering how I might, if possible, go about using code like https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sqllogictests/src/engines/datafusion/normalize.rs#L30 so that I could use sqllogictest-rs similar to how datafusion does, but w/o having to reimplement some of the "nitty-gritty" stuff. If it's not, any initial thoughts on making it into a crate?

Thanks!

[alamb]

Hi @tshauck -- this is a really neat idea.

I think there are two ways to go:

1. make a new crate (datafusion-sqllogictest??) with the datafusion machinery (and hooks to setup the context specially)
2. port some/all of the functionality you are looking upstream into https://github.com/risinglightdb/sqllogictest-rs -- it might be worth starting a conversation in that crate for interest.

Maybe @melgenek or @xxchan have some additional opinions

[tshauck]

If I'm following then the direction is a crate in this repo for item 1, "datafusion-sqllogictest for the implementation of the Datafusion client", then I can open an issue in the risinglightdb repo about adding DataFusion as a supported engine along side postgres? -- or do you'all that would go in this repo too?

[xxchan]

If you only want to use sqllogictest as sth like cargo test/run, like arrow-datafusion, you don't need to change sqllogictest-rs repo. You only need if you want to use the sqllogictest-rs CLI tool.

You can reuse logic like:

let mut runner = sqllogictest::Runner::new(|| async {
      Ok(datafusion-sqllogictest::DataFusion::new(...))
  });

and you implement your own db server setup / test file discovery logic.

[tshauck]

Ok cool, that jibes w/ what I'm doing here https://github.com/wheretrue/exon/blob/f98941e0df7a25327e59eccc0dd6b89b06ada07b/exon/tests/sqllogictests/src/main.rs#L116 (that I mostly adapted from here 😄 )... just seeing that sqllogictest exposes an external engine for postgres I wasn't sure if that's what was meant.

So I'll try to bottle up some of the internal to datafusion stuff into a crate for feedback?

[alamb]

I would recommend you at least start with a crate in the datafusion repo so that we can get on the regular release timeline of the rest of the project.

Maybe that is what you mean by "internal to datafusion stuff" but figured I would add the suggestion

[tshauck]

Sorry yes, that's what I meant -- will do that.

[alamb]

BTW @tshauck -- https://www.wheretrue.dev/docs/exon/ looks like a very cool project. It is a great example of a specific analytics tool reusing the datafusion machinery, Would you mind if we added it to the list in https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users ?

[tshauck]

That sounds good to me -- thanks for all the work you and the other collaborators have done on this project. I've wanted to build something like exon for several years and it's only been realistic in the last year or so.

[alamb]

#7103

[alamb]

To follow up here -- this was completed (thanks @tshauck ) in #7134