duckdb 1.4.0 Error: Assertion '__n < this->size()' failed.

[erydit]

After an update to 1.4.0, duckdb started to fail with an error:

/usr/include/c++/15.2.1/bits/stl_vector.h:1282: std::vector<_Tp, _Alloc>::const_reference std::vector<_Tp, _Alloc>::operator[](size_type) const [with _Tp = unsigned char*; _Alloc = std::allocator<unsigned char*>; const_reference = unsigned char* const&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
[70398:70399:20250924,153933.666930:ERROR file_io_posix.cc:148] open /home/rstanislav/.r/crashpad_database/pending/76ad2fc4-02f5-4946-8e94-22f3491f1216.lock: File exists (17)
[70398:70399:20250924,153933.667109:ERROR file_io_posix.cc:148] open /home/rstanislav/.r/crashpad_database/pending/08f41697-72e6-49a3-8b0a-ea07ee068eb0.lock: File exists (17)
[70398:70398:20250924,153933.670075:ERROR process_memory_range.cc:86] read out of range
[70398:70398:20250924,153933.670086:ERROR elf_image_reader.cc:558] missing nul-terminator
[70398:70398:20250924,153933.670164:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.672565:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.672641:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.672709:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.672777:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.672855:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.673309:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.673380:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674083:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674192:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674245:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674298:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674352:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674412:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674464:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674515:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674569:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674624:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.674778:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676369:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676418:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676468:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676516:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676567:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676621:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676677:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676726:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676775:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676825:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676875:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676925:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.676977:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677024:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677069:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677115:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677167:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677213:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677262:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677312:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677358:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677405:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677452:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677507:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677565:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677612:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677662:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677708:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677771:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677819:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677864:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677915:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.677955:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.678003:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.678047:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.678102:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70398:20250924,153933.678149:ERROR elf_dynamic_array_reader.h:61] tag not found
[70398:70399:20250924,153933.683178:ERROR directory_reader_posix.cc:42] opendir: No such file or directory (2)

And I cant find the exact reason to provide a reproducible example. I have tried to get logs using:

dbExecute(features_conn, "CALL enable_logging(level='debug');")
dbExecute(features_conn, "CALL enable_logging(storage='file', storage_config={'path': logs'});")

But these logs are pretty useless

The error arises in the code when I am trying to compute tbl_base_integral:

walk(.x = 1:nrow(df_chunks), .f = ~{
      x_i <- .x
      tile_num_i <- df_chunks$tile_num[x_i]
      tile_cell_min_i <- df_chunks$tile_cell_min[x_i]
      tile_cell_max_i <- df_chunks$tile_cell_max[x_i]
      
     tbl_base <- tbl(features_conn, "source.tbl_cells") %>%
        filter(
          tile_num == {{ tile_num_i }}, 
          tile_cell >= {{ tile_cell_min_i }},
          tile_cell <= {{ tile_cell_max_i }}) %>%
        select(tile_num, tile_cell, band_date, {{ db_vars }}) %>%
        mutate(band_year = as.integer(band_date / 10000),
               band_date = as.Date(strptime(as.character(band_date), "%Y%m%d"))) %>%
        compute(name = "temp_base", temporary = T, overwrite = T)
      
      tbl_base_stats <- tbl_base %>%
        group_by(tile_num, tile_cell, band_year) %>%
        summarise(across(any_of(db_vars), 
                         list(
                           mean = ~ as.integer( mean(.x, na.rm = TRUE) ),
                           sd = ~ as.integer( sd(.x, na.rm = TRUE) ),
                           min = ~ as.integer( min(.x, na.rm = TRUE) ),
                           max = ~ as.integer( max(.x, na.rm = TRUE) ),
                           range = ~ as.integer( max(.x, na.rm = TRUE) - min(.x, na.rm = TRUE)) ),
                         .names = "{.col}_{.fn}"),
                  .groups = "drop") %>%
        compute(name = "temp_base_stats", temporary = T, overwrite = T)
      
      tbl_base_integral <- tbl_base %>%
        group_by(tile_num, tile_cell, band_year) %>%
        arrange(band_date) %>%
        mutate(across(any_of(db_vars), 
                      .fns = ~ (band_date - lag(band_date)) * (lag(.x) + .x), 
                      .names = "{.col}_prod")) %>%
        arrange() %>%
        summarise(across(any_of(paste0(db_vars, "_prod")), 
                         list(prod = ~ as.integer(sum(.x, na.rm = TRUE))),
                         .names = "{.col}_{.fn}"),
                  .groups = "drop") %>%
        compute(name = "temp_base_integral", temporary = T, overwrite = T)
      
      dat_features <- left_join(tbl_base_stats, tbl_base_integral, by = c("tile_num", "tile_cell", "band_year")) %>%
        collect() %>%
        dbWriteTable(features_conn, name = "tbl_cells", value = ., append = T)
      
      if (x_i %% 10 == 0) {
        dbExecute(features_conn, "CHECKPOINT;")
      }
      
      invisible(gc())
      
      p()
    }) # END walk()

This code is executed in the purrr::walk cycle and the error occurs at 81st iteration by some reason (nothing special in the data of the 81st iteration). I cant provide the source database as it is 4.5 Gb size. But if I save 81st data chunk into separate db and try to reproduce the issue the error does not appears.

sessionInfo()

R version 4.5.1 (2025-06-13) Platform: x86_64-pc-linux-gnu Running under: Manjaro Linux

Matrix products: default
BLAS/LAPACK: /usr/lib/libopenblas.so.0.3; LAPACK version 3.12.0

locale:
[1] LC_CTYPE=ru_RU.UTF-8 LC_NUMERIC=C LC_TIME=ru_RU.UTF-8 LC_COLLATE=ru_RU.UTF-8
[5] LC_MONETARY=ru_RU.UTF-8 LC_MESSAGES=ru_RU.UTF-8 LC_PAPER=ru_RU.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=ru_RU.UTF-8 LC_IDENTIFICATION=C

time zone: Europe/Moscow
tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

after downgrade to duckdb 1.3.2 the code works as expected.

[krlmlr]

Thanks. A DuckDB update is scheduled for October 6. I'll resume vendoring soon so that you can check with an updated version from GitHub or r-universe.

If you're very brave, you could try dblog: https://github.com/r-dbi/dblog . This might be able to produce code at one layer below -- calls to DBI with pure SQL. Ideally, this DBI code could be made to work and reproduce the same error. Happy to assist.

[erydit]

@krlmlr Thanks for the tip! I didn't know about the dblog package. Below I'm attaching two logs: one for DuckDB version 1.4.0, the second for DuckDB version 1.3.2. The difference starts at line 1369, where DuckDB 1.4.0 throws an error. Version 1.3.2 successfully inserts a chunk of data into the new table.
Unfortunately, I don't see the reason for the error in these logs.

duckdb_1.3.2.txt

duckdb_1.4.0.txt

[krlmlr]

Thanks. Can you reproduce the error when running the generated codes directly?

I wonder if this can be turned into a simple .sql file that shows the same behavior with the DuckDB CLI. Another thing to try: the latest and greatest duckdb 1.4.1. release candidate from r-universe: https://duckdb.r-universe.dev/duckdb .

Those dbQuoteIdentifier() calls aren't helpful. I wish dblog had an option to filter out certain calls.