duckplyr 1.0.0

content/blog/duckplyr-1-0-0/index.Rmd

@@ -0,0 +1,209 @@
+---
+output: hugodown::hugo_document
+
+slug: duckplyr-1-0-0
+title: duckplyr fully joins the tidyverse!
+date: 2025-02-13
+author: Kirill Müller and Maëlle Salmon
+description: >
+    duckplyr 1.0.0 is on CRAN and part of the tidyverse!
+    A drop-in replacement for dplyr, powered by DuckDB for speed.
+    It is the most dplyr-like of dplyr backends.
+
+photo:
+  url: https://www.pexels.com/photo/a-mallard-duck-on-water-6918877/
+  author: Kiril Gruev
+
+# one of: "deep-dive", "learn", "package", "programming", "roundup", or "other"
+categories: [package]
+tags:
+  - duckplyr
+  - dplyr
+  - tidyverse
+---
+
+<!--
+TODO:
+* [x] Look over / edit the post's title in the yaml
+* [x] Edit (or delete) the description; note this appears in the Twitter card
+* [x] Pick category and tags (see existing with `hugodown::tidy_show_meta()`)
+* [x] Find photo & update yaml metadata
+* [x] Create `thumbnail-sq.jpg`; height and width should be equal
+* [x] Create `thumbnail-wd.jpg`; width should be >5x height
+* [x] `hugodown::use_tidy_thumbnails()`
+* [x] Add intro sentence, e.g. the standard tagline for the package
+* [x] `usethis::use_tidy_thanks()`
+-->
+
+We're very chuffed to announce the release of [duckplyr](https://duckplyr.tidyverse.org) 1.0.0.
+duckplyr is a drop-in, fully compatible replacement for dplyr, powered by [DuckDB](https://duckdb.org/) for speed.
+It joins the rank of dplyr backends together with [dtplyr](https://dtplyr.tidyverse.org) and [dbplyr](https://dbplyr.tidyverse.org).
+You can use it instead of dplyr for data small or large.
+
+<!-- FIXME:
+
+We have many more dplyr backends, the two above are just from the tidyverse.
+GitHub search: https://github.com/search?q=org%3Acran+%2FS3method%5B%28%5D%28mutate%7Csummarise%29+*%2C%2F&type=code
+Do we need an "awesome dplyr" like https://github.com/krlmlr/awesome-vctrs/?
+
+-->
+
+You can install it from CRAN with:
+
+```{r, eval = FALSE}
+install.packages("duckplyr")
+```
+
+In this article, we'll show how duckplyr can help you with data of different size, explain how you can help improve the package, and ... .
+
+## A drop-in replacement for dplyr
+
+The duckplyr package is a _drop-in replacement for dplyr_ that uses _DuckDB for speed_.
+You can simply _drop_ duckplyr into your pipeline by loading it, then computations will be efficiently carried out by DuckDB.
+DuckDB is a fast in-memory analytical database system[^duckdb].
+
+[^duckdb]: If you haven't heard about it, you can watch [Hannes Mühleisen's keynote at posit::conf(2024)](https://www.youtube.com/watch?v=GELhdezYmP0&feature=youtu.be).
+
+
+```{r}
+library(conflicted)
+library(duckplyr)
+conflict_prefer("filter", "dplyr", quiet = TRUE)
+library(babynames)
+
+out <- babynames |>
+  mutate(prevalence = if_else(prop >= 0.01, "frequent", "rare")) |>
+  summarize(
+    .by = c(sex, year, prevalence),
+    babies_n = sum(n)
+  ) |>
+  filter(sex == "F")
+class(out)
+out
+```
+
+Like with other dplyr backends like dtplyr and dbplyr, duckplyr allows you to get faster results.
+Unlike other dplyr backends, duckplyr does not require you to learn a different syntax.
+
+The duckplyr package is fully compatible with dplyr: if an operation cannot be carried out with DuckDB, it is automatically outsourced to dplyr.
+Over time, we expect fewer and fewer fallbacks to dplyr to be needed.
+
+## How to use duckplyr
+
+To _replace_ dplyr with duckplyr, you can:
+
+- Load duckplyr and then keep your pipeline as is. Calling `library(duckplyr)` overwrites dplyr methods, enabling duckplyr for the entire session no matter how data.frames are created.
+    This is shown in the example above.
+
+- Create individual "duck frames" using _conversion functions_ like `duckdb_tibble()` or `as_duckdb_tibble()`, or _ingestion functions_ like `read_csv_duckdb()`.
+
+
+Then, the data manipulation pipeline uses the exact same syntax as a dplyr pipeline.
+The duckplyr package performs the computation using DuckDB.
+
+```{r}
+# Undo the effect of library(duckplyr)
+methods_restore()
+
+out <- babynames |>
+  as_duckdb_tibble() |>
+  mutate(prevalence = if_else(prop >= 0.01, "frequent", "rare")) |>
+  summarize(
+    .by = c(sex, year, prevalence),
+    babies_n = sum(n)
+  ) |>
+  filter(sex == "F")
+class(out)
+```
+
+In both cases, printing the result only shows the first few rows, as with dbplyr.
+
+```{r}
+out
+```
+
+The result can finally be materialized to memory, or computed temporarily, or computed to a file.
+
+```{r}
+# to memory
+nrow(out)
+# or for instance collect(out)
+
+# to a file
+csv_file <- withr::local_tempfile()
+compute_csv(out, csv_file)
+fs::file_size(csv_file)
+```
+
+When duckplyr itself does not support specific functionality, it falls back to dplyr.
+For instance, filtering on grouped data is not supported yet, still it works thanks to the fallback mechanism.
+
+```{r}
+babynames |>
+  filter(n > 10000, .by = "name")
+```
+
+For performance reasons, the output order of the result is not guaranteed to be stable.
+If you need a stable order, you can use `arrange()` or force output order stability by setting an environment variable.
+This and other limitations are documented in [`vignette("limits")`](https://duckplyr.tidyverse.org/articles/limits.html).
+
+## Large data
+
+For large data, duckplyr is a legitimate alternative to dtplyr and dbplyr.
+
+With large datasets, you want:
+
+- input data in an efficient format, like Parquet files, which duckplyr allows thanks to its ingestion functions like `read_parquet_duckdb()`.
+- efficient computation, which duckplyr provides via DuckDB's holistic optimization, without your having to use another syntax than dplyr.
+- the output to not clutter all the memory, which duckplyr supports through two features:
+    - computation to files using  [`compute_parquet()`](https://duckplyr.tidyverse.org/reference/compute_file.html) or [`compute_csv()`](https://duckplyr.tidyverse.org/reference/compute_file.html).
+    - the control of automatic materialization (collection of results into memory). You can disable automatic materialization completely or, as a compromise, disable it up to a certain output size. See [`vignette("prudence")`](https://duckplyr.tidyverse.org/articles/prudence.html) for details.
+
+See [`vignette("large")`](https://duckplyr.tidyverse.org/articles/large.html) for a walkthrough and more details.
+
+A drawback of analyzing large data with duckplyr is that the limitations of duckplyr won't be compensated by fallbacks, since fallbacks to dplyr necessitate putting data into memory.
+Therefore, if your pipeline encounters fallbacks, you might want to work around them by converting the duck frame into a table through `compute()` then running SQL code through the experimental `read_sql_duckdb()` function.
+Again, over time, we expect more native support for dplyr functionality.
+
+```{r}
+data <-
+  duckdb_tibble(a = 2) |>
+  mutate(b = 3)
+
+computed_data <-
+  data |>
+  compute(name = "computed_data")
+
+sql_data <-
+  read_sql_duckdb("SELECT *, a * b AS c FROM computed_data")
+
+sql_data
+```
+
+## Help us improve duckplyr!
+
+Our goals for future development of duckplyr include:
+
+- Enabling users to provide [custom translations](https://github.com/tidyverse/duckplyr/issues/158) of dplyr functionality;
+- Making it easier to contribute code to duckplyr;
+- Supporting more dplyr and tidyr functionality natively in DuckDB.
+
+You can help!
+
+- Please report any issues, especially regarding unknown incompabilities. See [`vignette("limits")`](https://duckplyr.tidyverse.org/articles/limits.html).
+- Contribute to the codebase after reading duckplyr's [contributing guide](https://duckplyr.tidyverse.org/CONTRIBUTING.html).
+- Turn on telemetry to help us hear about the most frequent fallbacks so we can prioritize working on the corresponding missing dplyr translation. See [`vignette("telemetry")`](https://duckplyr.tidyverse.org/articles/telemetry.html) and the [`duckplyr::fallback_sitrep()`](https://duckplyr.tidyverse.org/reference/fallback.html) function.
+
+## Acknowledgements and additional resources
+
+A big thanks to all folks who filed issues, created PRs and generally helped to improve duckplyr!
+
+<!-- FIXME: Can we use_tidy_thanks also for the duckdb repo?, and perhaps merge the two? -->
+
+[&#x0040;adamschwing](https://github.com/adamschwing), [&#x0040;andreranza](https://github.com/andreranza), [&#x0040;apalacio9502](https://github.com/apalacio9502), [&#x0040;apsteinmetz](https://github.com/apsteinmetz), [&#x0040;barracuda156](https://github.com/barracuda156), [&#x0040;beniaminogreen](https://github.com/beniaminogreen), [&#x0040;bob-rietveld](https://github.com/bob-rietveld), [&#x0040;brichards920](https://github.com/brichards920), [&#x0040;cboettig](https://github.com/cboettig), [&#x0040;davidjayjackson](https://github.com/davidjayjackson), [&#x0040;DavisVaughan](https://github.com/DavisVaughan), [&#x0040;Ed2uiz](https://github.com/Ed2uiz), [&#x0040;eitsupi](https://github.com/eitsupi), [&#x0040;era127](https://github.com/era127), [&#x0040;etiennebacher](https://github.com/etiennebacher), [&#x0040;eutwt](https://github.com/eutwt), [&#x0040;fmichonneau](https://github.com/fmichonneau), [&#x0040;hadley](https://github.com/hadley), [&#x0040;hannes](https://github.com/hannes), [&#x0040;hawkfish](https://github.com/hawkfish), [&#x0040;IndrajeetPatil](https://github.com/IndrajeetPatil), [&#x0040;JanSulavik](https://github.com/JanSulavik), [&#x0040;JavOrraca](https://github.com/JavOrraca), [&#x0040;jeroen](https://github.com/jeroen), [&#x0040;jhk0530](https://github.com/jhk0530), [&#x0040;joakimlinde](https://github.com/joakimlinde), [&#x0040;JosiahParry](https://github.com/JosiahParry), [&#x0040;larry77](https://github.com/larry77), [&#x0040;lnkuiper](https://github.com/lnkuiper), [&#x0040;lorenzwalthert](https://github.com/lorenzwalthert), [&#x0040;luisDVA](https://github.com/luisDVA), [&#x0040;maelle](https://github.com/maelle), [&#x0040;math-mcshane](https://github.com/math-mcshane), [&#x0040;meersel](https://github.com/meersel), [&#x0040;multimeric](https://github.com/multimeric), [&#x0040;mytarmail](https://github.com/mytarmail), [&#x0040;nicki-dese](https://github.com/nicki-dese), [&#x0040;PMassicotte](https://github.com/PMassicotte), [&#x0040;prasundutta87](https://github.com/prasundutta87), [&#x0040;rafapereirabr](https://github.com/rafapereirabr), [&#x0040;Robinlovelace](https://github.com/Robinlovelace), [&#x0040;romainfrancois](https://github.com/romainfrancois), [&#x0040;sparrow925](https://github.com/sparrow925), [&#x0040;stefanlinner](https://github.com/stefanlinner), [&#x0040;thomasp85](https://github.com/thomasp85), [&#x0040;TimTaylor](https://github.com/TimTaylor), [&#x0040;Tmonster](https://github.com/Tmonster), [&#x0040;toppyy](https://github.com/toppyy), [&#x0040;wibeasley](https://github.com/wibeasley), [&#x0040;yjunechoe](https://github.com/yjunechoe), [&#x0040;ywhcuhk](https://github.com/ywhcuhk), and [&#x0040;zhjx19](https://github.com/zhjx19).
+
+Special thanks to Joe Thorley ([&#x0040;joethorley](https://github.com/joethorley)) for help with choosing the right words.
+
+Eager to learn more about duckplyr -- beside by trying it out yourself?
+The pkgdown website of duckplyr features several [articles](https://duckplyr.tidyverse.org/articles/).
+Furthermore, the blog post ["duckplyr: dplyr Powered by DuckDB"](https://duckdb.org/2024/04/02/duckplyr.html) by Hannes Mühleisen provides some context on duckplyr including its inner workings, as also seen in a [section](https://blog.r-hub.io/2025/02/13/lazy-meanings/#duckplyr-lazy-evaluation-and-prudence) of the R-hub blog post ["Lazy introduction to laziness in R"](https://blog.r-hub.io/2025/02/13/lazy-meanings/) by Maëlle Salmon, Athanasia Mo Mowinckel and Hannah Frick.