Error in Materialization printing so many times

[durraniu]

I have 6 parquet files in my data folder and I'm trying to read them with duckplyr but I get this error and it continues to print:

library(geoarrow)
library(duckplyr)

urls <- list.files(
  path = "data/",
  pattern = "\\.parquet$",
  full.names = TRUE
)
# [1] "data/geofabrik_canada.parquet"
# [2] "data/us-midwest.parquet"      
# [3] "data/us-northeast.parquet"    
# [4] "data/us-pacific.parquet"      
# [5] "data/us-south.parquet"        
# [6] "data/us-west.parquet" 

roads <- read_parquet_duckdb(urls)
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.
# Error: Materialization would result in 250001 rows, which exceeds the limit of 250000. Use collect() or as_tibble() to materialize.

I am on R 4.4.2 and duckplyr version 1.0.0. I don't want to read all of the data in memory. What can I do to easily filter/select, etc. without loading all data in memory?

[krlmlr]

Thanks. This looks like a bug. Does this also occur with a subset of the files, or is one single file driving the error? What are the column types? How are these files created?

[durraniu]

Does this also occur with a subset of the files, or is one single file driving the error?

Yes it occurs with a single file also.

What are the column types?

> arrow::open_dataset(urls[1])
FileSystemDataset with 1 Parquet file
4 columns
name: string
highway: string
other_tags: string
geometry: geoarrow.linestring <CRS: {
  "$schema": "https://pro...

How are these files created?

I downloaded pbf files from geofabrick website. Reading the linestrings only automatically generated gpkg files in the same folder. I then converted the gpkg files to parquet:

library(geoarrow)
library(osmextract)

all_files <- list.files(
  path = "data/",
  pattern = "\\.gpkg$"
)
all_files <- sapply(all_files, 
       function(x){sub("\\.gpkg$", "", x)}, 
       USE.NAMES = FALSE)

convert_to_parquet <- function(filename){
  foo <- oe_read(paste0("data/", filename, ".gpkg")) |>
    tibble::as_tibble()   
  foo |> 
    arrow::write_parquet(paste0("data/", filename, ".parquet"))
    rm(foo)
}
lapply(
  all_files,
  convert_to_parquet
)

Perhaps the issue is the geometry column?

[krlmlr]

The error behavior should be better. Can you please help me replicate?

I see, with both duckdb 1.1.3-2 and the upcoming 1.2.0:

library(geoarrow)
library(osmextract)
#> Data (c) OpenStreetMap contributors, ODbL 1.0. https://www.openstreetmap.org/copyright.
#> Check the package website, https://docs.ropensci.org/osmextract/, for more details.

curl::curl_download(
  "https://download.geofabrik.de/north-america/greenland-latest.osm.pbf",
  "greenland-latest.osm.pbf"
)
url <- "greenland.parquet"

oe_read("greenland-latest.osm.pbf") |>
  arrow::write_parquet(url)
#> Starting with the vectortranslate operations on the input file!
#> 0...10...20...30...40...50...60...70...80...90...100 - done.
#> Finished the vectortranslate operations on the input file!
#> Reading layer `lines' from data source 
#>   `/private/var/folders/dj/yhk9rkx97wn_ykqtnmk18xvc0000gn/T/RtmpG17ahf/reprex-1464311fd9c4a-full-husky/greenland-latest.gpkg' 
#>   using driver `GPKG'
#> Simple feature collection with 17183 features and 10 fields
#> Geometry type: LINESTRING
#> Dimension:     XY
#> Bounding box:  xmin: -79.45219 ymin: 47.80979 xmax: -7.5 ymax: 83.87517
#> Geodetic CRS:  WGS 84

duckplyr::read_parquet_duckdb(url)
#> # A duckplyr data frame: 11 variables
#>    osm_id   name  highway waterway aerialway barrier man_made railway z_order
#>    <chr>    <chr> <chr>   <chr>    <chr>     <chr>   <chr>    <chr>     <int>
#>  1 22797155 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  2 22797189 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  3 22797220 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  4 22797333 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  5 22797349 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  6 22798102 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  7 22798274 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  8 22798329 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#>  9 22798350 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#> 10 22798369 <NA>  <NA>    <NA>     <NA>      <NA>    <NA>     <NA>          0
#> # ℹ more rows
#> # ℹ 2 more variables: other_tags <chr>, geometry <list>

^{Created on 2025-02-18 with reprex v2.1.1}

[durraniu]

Thanks for looking into it. In my example, I had already converted the gpkg files to parquet, and in your example, you saved from memory to parquet. In both cases, the geometry column is not read in correctly. For example, if I extend your example for creating a plot:

library(geoarrow)
library(osmextract)

curl::curl_download(
  "https://download.geofabrik.de/north-america/greenland-latest.osm.pbf",
  "greenland-latest.osm.pbf"
)
url <- "greenland.parquet"

oe_read("greenland-latest.osm.pbf") |>
  arrow::write_parquet(url)

roads <- duckplyr::read_parquet_duckdb(url)

roads |> select(name, geometry) |> #sf::st_as_sf()
  ggplot2::ggplot() +
  ggplot2::geom_sf()
  # Error in `ggplot2::geom_sf()`:
#   ! Problem while computing stat.
# ℹ Error occurred in the 1st layer.
# Caused by error in `compute_layer()`:
#   ! `stat_sf()` requires the following missing
# aesthetics: geometry.

I am unable to convert to sf and hence cannot use the geometry column.

So, it seems that duckplyr wasn't able to properly read the geometry column`.

However, this works:

library(geoarrow)

arrow::open_dataset("greenland.parquet") |>
  dplyr::select(name, geometry) |>
  sf::st_as_sf() |>
  ggplot2::ggplot() +
  ggplot2::geom_sf()

[krlmlr]

How can I "convert the gpkg files to parquet", like you did?

I'd like duckplyr to be compatible with sf, the error message in the OP is more important to me at this stage.

[durraniu]

Actually, there isn't any difference of what I did. I converted the gpkg to parquet in the similar manner as you did with the pbf file:

library(geoarrow)
library(osmextract)

all_files <- list.files(
  path = "data/",
  pattern = "\\.gpkg$"
)

all_files <- sapply(all_files, 
       function(x){sub("\\.gpkg$", "", x)}, 
       USE.NAMES = FALSE)

convert_to_parquet <- function(filename){
  foo <- oe_read(paste0("data/", filename, ".gpkg")) |>
    tibble::as_tibble() 
  
  foo |> 
    arrow::write_parquet(paste0("data/", filename, ".parquet"))
  
  rm(foo)
}

lapply(
  all_files,
  convert_to_parquet
)

[krlmlr]

I already saw this code. Where do I get the .gpkg files? It's not obvious from https://download.geofabrik.de/north-america.html

[durraniu]

Oh sorry. When you read the pbf file with oe_read, it automatically saves the gpkg file in the same folder. Look for the greenland-latest.gpkg file where you had the greenland-latest.osm.pbf file.

[durraniu]

@krlmlr Were you able to find the gpkg file? I'm not concerned with reading gpkg specifically. I can work with pbf and parquet files as you showed in the example. Thanks again for looking into it.

[krlmlr]

Thanks for the heads-up. I still really like to see a reproducible example of the OP, this is clearly wrong behavior.