Farewell pandas, and thanks for all the fish. – Ibis

[giscus[bot]]

Farewell pandas, and thanks for all the fish. – Ibis

the portable Python dataframe library

https://ibis-project.org/posts/farewell-pandas/

[jonmmease]

Without pandas and dask, which backends are left that aren't SQL based? Are you aiming to stop supporting non-sql backends in general?

[gforsyth]

It will only be the polars backend that isn't SQL based, and once their SQL support is at feature-parity, we'll probably move to using SQL there, too. We aren't aiming to stop supporting non-SQL backends, though -- if there's a deferred query engine we're happy to wire it up.

And while it was a joke, Phillip Cloud's ibish is a pretty great demonstration that the Ibis IR is flexible enough to talk to just about anything: https://ibis-project.org/posts/unix-backend/

[kszucs]

It is somewhat bittersweet for me to see the pandas backend go since I've rewritten it from the ground up a couple months ago #7797 vastly simplifying it, improving its memory management and operation support, it was still a burden that we had to handle in order to ship ibis' internal refactors.

The main point where the ibis backend stands out of the rest, is that ibis actually implements an execution engine on top of pandas functions as compute kernels. Although both the dask and polars backends use the rewrite rules from the pandas backend to transform the logical plan to a dataframe API compatible flavor, they don't need to implement an execution engine, just building up a lazy execution plan of that particular backend.

[gregorywaynepower]

I certainly hope that daft will replace dask at some point in the future.

[gregorywaynepower]

For folks interested in the progress here's the issue on daft support as a backend: #8904

[jgbradley1]

With pandas backend gone, will that impact export functionality like .to_pandas()? I see a lot of value in keeping at least this functionality around to allow users to perform downstream tasks and interface with dataframe libraries that were designed around pandas (Seaborn, Bokeh, Plotly, etc.).

[jcrist]

Pandas will remain around as an input and output type of ibis. You can still pass pandas dataframes to create_table or memtable, and you can still get results in pandas with to_pandas. The only thing that's going away is using pandas as the backing compute engine (the ibis.pandas backend).

[NickCrews]

As a contributor who has had to add/modify pandas special cased stuff when submitting a PR, I am excited that this cruft is getting removed.