Fantastic book here! also curious to here your thoughts on ibis

[cboettig]

Hi friends! Fantastic book here; I have really been enjoying seeing your approach to 'tidy' python. Some quick background context, though maybe @ttimbers remembers me from rOpenSci stuff years ago; I've taught R for data science at Berkeley for ~ 10 years now based roughly around the R4DS book with a domain-applications-flavor (R-based course site) and have just this year switched into python while seeking to retain the foundational tidy principles you espouse so clearly in your book!

Given what I think are our very similar objectives/standpoints, I am really keen to hear your opinions about teaching ibis rather than pandas. Yes yes I know the entire planet uses pandas and very few use ibis, but of course no one used dplyr or pandas at the beginning either. I struggle to find the balance just right, but I wanted to share my experience with ibis in case it was helpful or could lead to further discussion. It definitely isn't perfect but I think these things are worth us discussing as a community.

You do discuss ibis in the context of SQL backends, but as you probably know with the recent duckdb backend at least it works very nicely with formats like csv and parquet (and even any spatial vector format) and gives us a more dplyr-esque syntax (e.g. .filter(), .select(), .mutate(), .group_by(), .aggregate(), .pivot_longer(), left_join() etc). Using duckdb on the backend, it's both faster and able to handle larger datasets with minimal RAM requirements, which is particularly useful for teaching real-world examples on the kind of small-RAM machines my students typically have access to in the classroom.

On the R side I actually also stick with dplyr->dbplyr->duckdb (or somewhat similarly, dplyr+arrow backend), so the students just learn the same dplyr syntax but it works much faster and remains applicable with larger-than-RAM datasets. Yes, this means certain dplyr-functions that can't translate to SQL don't work, just as you note in your python book that some pandas functions like tail don't translate to ibis; precisely because they don't have any analog in SQL. Overall though I think this is more of a feature than a bug. I find SQL syntax incredibly difficult to read, and the syntax of dplyr (and ibis, and now many similar variants like polars, prql etc etc) much easier. But the logic of what it does and doesn't do is very tidy -- after all, as you know Hadley's notion of "tidy" data is as he explains just a more catchy term for what the SQL people/relational database community have called "Cobb's third normal form" for decades -- but both R and python syntax grew up organically out of their own communities.

Anyway, didn't mean for this to be a 🧼 📦 on ibis, and it's not perfect (e.g. duckdb's read_csv() isn't quite a configurable as pandas read_csv(), though it's gotten better recently), but as tidy data people I'd be really curious what you make of it's current abilities. I think I can get pretty close to the same content I cover in my R4DS course just in ibis now, but am still figuring out how to navigate when and how much of pandas students should learn. (I guess in R we eventually teach some base R concepts, but dplyr was always compatible with a base data.frame while ibis needs a call to create_table to get rolling...)

[joelostblom]

Thanks for the kind words @cboettig !
I haven't used Ibis much myself so far, but dabbled around a bit with polars and duckdb so it is exiting that ibis now supports these as backends. My main draw for venturing outside pandas has been learning a new potentially more expressive syntax for wrangling and that I wanted to be one of the cool kids and use rusty new tools. From a quick glance on the Ibis docs, it is unclear to me if they provide a more expressive syntax, but you have gotten me interested to try it out more in other contexts.

As you point out, I believe the key balance point is when does it becomes beneficial to switch from a tool that is abundant to a tool that seems to be theoretically and/or practically superior. Personally, I hold off when there are too many cases where I find myself switching back to the more popular tool to get work done. We did have a similar case in one of our data viz courses where we switched to the Altair library which was tricky for a year or two, but has paid off long term. In terms of teaching to data science novices, I think there is also the additional consideration of how easy it is for them to find help (docs, SO, genAI, etc) on that package and how useful it will be in the future courses/work that they learn one package over the other.

Although I haven't used Ibis enough to say if I personally find that it is worthwhile switching currently, I did notice that Voltron Data supports it wholeheartedly and they lay out some sound arguments in their post (which you might already have seen).

[cboettig]

Thanks @joelostblom ! Yup, I also played for a while with polars before settling on ibis. Syntax-wise they are very similar, though coming from dplyr I have a slight preference for ibis having .mutate() vs polars .with_column(). Both avoid the NSE issue of dplyr, but in polars it looks like:

df.with_columns((pl.col("a") ** 2).alias("a_squared"))

while ibis looks a lot more like dplyr:

df.mutate(a_squared = _.a ** 2)

note the _. is just an alias for the 'current data frame', (we could have used df.a instead), which avoids NSE while also being more compact than pl.col("a") and echoing the usual python way of indexing columns. you can also do df["a"] or _["a"] in ibis.

polars and rust definitely have momentum, but using duckdb by ibis totally sealed the deal for me. duckdb beats polars handily in most benchmarks. polars has both lazy (i.e. on disk) and non-lazy data.frames, which can help with RAM but can also be confusing. At least a year ago the support wasn't there with polars to access large remote parquet files over s3 / https in streaming mode -- that is, you couldn't just point it at a bunch large files. e.g. here's a reprex with a dataset that's over 250 GB:

import ibis
from ibis import _
con = ibis.duckdb.connect()
parquet = "s3://gbif-open-data-us-east-1/occurrence/2024-10-01/occurrence.parquet/**"
gbif = con.read_parquet(parquet)
(gbif
.group_by(_.countrycode)
.agg(n = _.count())
.order_by(_.n.desc())
.execute()
)

on a good (e.g. cloud-based, like codespaces) internet connection this takes only a few minutes, runs in less than 2 GB of RAM and no disk space. (on a local copy or local network this is even faster, ~ 15 seconds). the ibis syntax isn't far off the same R syntax for duckdb+dplyr:

library(duckdbfs)
library(dplyr)
parquet = "s3://gbif-open-data-us-east-1/occurrence/2024-10-01/occurrence.parquet/**"
gbif = open_dataset(parquet)
gbif |> 
  group_by(countrycode) |>
  summarise(n = n()) |>
  order_by(desc(n)) |>
  collect()

there is a geopolars effort paralleling geopandas but doesn't seem that mature. duckdb has a spatial extension that implements most of the postgis (i.e. sf like operations from R-world) syntax which is also nice. (of course duckdb has it's own python wrapper too, but doesn't provide the nice piped syntax that way).

These are my rather half-baked tutorials from teaching ibis this semester: https://espm-157.carlboettiger.info/ibis-1 (the official docs are better of course, though honestly not really all that great either. Honestly one of the main challenges this semester has been that the students always ask the LLMs, and the LLMs know pandas well but aren't particularly good in ibis)

[joelostblom]

Thanks for expanding on your thoughts and sharing the tutorial you have made @cboettig ! Maybe be something for us to consider in the future. You might also be interested in some of the discussion in this issue about a better duckdb python interface based on pyspark (so less flexible, but also less dependencies than ibis) duckdb/duckdb#12134 (comment)