How do I bring dependencies in my binding?

[dariocurr]

Hi guys, I'm Dario.
I have been struggling with an issue and I am trying to understand it.

I am trying to create my own cross-language library on top of datafusion and datafusion-python.
Let's call this library my-library.

I created a rust workspace and I have two crates:

• my-library, to be used by other rust crates
• my-library-python, to be used by other Python packages

my-library has datafusion as a dependency and has just one function returning a datafusion::execution::context::SessionContext

my-library-python, has datafusion-python as a dependency and has just one function wrapping the datafusion::execution::context::SessionContext in a datafusion_python::context::PySessionContext

Now. when I install my-library-python in my python env through maturin develop and try to play with the SessionContext returned by the binding as follows:

python
ctx = my_library.get_context()
datafusion_df = ctx.sql(query)

I get the following error

pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'datafusion'"), traceback: None }

My question then is:
Why should I add datafusion as a dependency in my python package, duplicating the library?
Is there a way to bring the dependency in my binding?

[dariocurr]

The binding is created as follows:

#[pymodule]
fn module(_: Python<'_>, main_module: &PyModule) -> PyResult<()> {
    main_module.add_function(wrap_pyfunction!(get_context, main_module)?)?;
    Ok(())
}

[Michael-J-Ward]

I haven't previously done what you're trying to do, so a minimum github repo to reproduce would be helpful.

This error is the python runtime trying to import datafusion, and it isn't clear to me from your example why / where that would happen.

ModuleNotFoundError("No module named 'datafusion'")

[Michael-J-Ward]

I haven't done any digging to see why the code is like this, but the end-result is that you probably will need datafusion as a python dependency.

datafusion-python/src/utils.rs

Lines 26 to 37 in 45a6844

	pub(crate) fn get_tokio_runtime(py: Python) -> PyRef<TokioRuntime> {
	let datafusion = py.import_bound("datafusion._internal").unwrap();
	let tmp = datafusion.getattr("runtime").unwrap();
	match tmp.extract::<PyRef<TokioRuntime>>() {
	Ok(runtime) => runtime,
	Err(_e) => {
	let rt = TokioRuntime(tokio::runtime::Runtime::new().unwrap());
	let obj: Bound<'_, TokioRuntime> = Py::new(py, rt).unwrap().into_bound(py);
	obj.extract().unwrap()
	}
	}
	}

[Michael-J-Ward]

Storing the tokio runtime on the python heap to ensure it only gets created once, which provided performance improvements.

It maybe could be created once per SessionContext or similar, but that would be a decent lift of a refactor.

#341

[Michael-J-Ward]

@andygrove confirmed that your use case is something that datafusion-python should support, and points to datafusion-ballista as an example.

@dariocurr, do you have a repo link to share, so that I can investigate further?

[dariocurr]

I just created an MRE here my-library-datafusion.

Following the instructions and running:

1. conda env create
2. maturin develop
3. pytest tests

You will get:

ModuleNotFoundError("No module named 'datafusion'")

I really don't know how it should work, I am here to ask and learn from you.

Thank you for your time

[dariocurr]

Hi, any news about this?

[timsaucer]

I’m focusing right now on finishing up some udf improvements, but after this my next big focus is improving the interoperability of datafusion-python with other libraries. We have at least 4 related use cases that need support.
I hope to start work on this next week

[timsaucer]

By the way, what is the scope of the extensions you’re trying to build on top of it. Depending on what all you need we may have some support already.

[dariocurr]

Hi, thanks for your response.
We are trying to build our enriched SessionContext in a cross-language library, so that we have a consistent system in each component of our platforms

[timsaucer]

You will need datafusion. I downloaded your repo and created an env for it. After doing

pip install datafusion==38.0.1

Your unit tests will run.

[dariocurr]

Yeah I know. That was my original question.

Why should I add datafusion as a dependency in my python package, duplicating the library?
Is there a way to bring the dependency in my binding?

[timsaucer]

Ok, I don't think it's going to work in the way that you'd like. The pyo3 package has some assumptions about how one project will expose one package. The DataFusion python libraries have their module paths in the bindings. I don't think you'll ever get the package my-library to expose both my-library and datafusion. I think there are two paths forward.

1. You could change your library to expose under datafusion. This should work right now. You might have to do a little bit of work to make sure the modules are getting added to your package in the same way we do at the top level of datafusion-python. Admittedly, this is not ideal.
2. We are working on exposing parts of DataFusion through python bindings. This will not work as you have described above, which is why I was trying to ask the question of what the scope of your extensions are. We can currently extend DataFusion by adding functions only. We know for sure we want to add the ability to extend table providers, and there may be other portions that we want to expose as well. These additions are non-trivial and may take some time.

If you could tell us what kinds of extensions you plan to bring we can try to make sure they are included in the work.

[Michael-J-Ward]

@timsaucer, the specific error that he's encountering happens because the rust side of datafusion-python assumes that datafusion is installed a python library.

This happens in exactly one spot (see my two comments from June 21st above).
#737 (comment)

I have a draft PR up #895 that removes this assumption, which also makes pytest pass for the MRE.

Here's my repo for the MRE using that feature branch:
https://github.com/Michael-J-Ward/my-library-datafusion

@dariocurr, could please "push" your use case further using my PR branch? I'd like to see if you encounter any unexpected errors with it.