unique gives segfault from C stack overflow

[quantitative-technologies]

# Minimal reproducible example

data <- readRDS('data_debug.rds')
dt <- unique(data)

crashes RStudio, or in the R console:

Error: segfault from C stack overflow

I can provide data_debug.rds (~300MB) if desired.

# Output of sessionInfo()

sessionInfo()
R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.10

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0

locale:
[1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
[5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
[7] LC_PAPER=en_CA.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] data.table_1.12.8

loaded via a namespace (and not attached):
[1] compiler_3.6.2

[shrektan]

Would you mind try to call setDT(data) before the unique call and report the results back here?

Just check if it’s due to the null self pointer.

[quantitative-technologies]

I tried calling setDT(data) first, but the result was the same.

[shrektan]

Oops, having your data would be necessary then.

[quantitative-technologies]

Please find the data set here

[shrektan]

Thanks for the data, I can reproduce this.

---

UPDATE: the error is thrown from

data.table/src/forder.c

Line 736 in b1b1832

radix_r(0, nrow-1, 0); // top level recursive call: (from, to, radix)

[shrektan]

OK, I create a smaller data set (23MB) that can reproduce this. The cause, I believe, is the recursive function radix_r() is called for too many times (too deep), leading to an overflow on the calling stack.

I don't know why this happens for this specific dataset as I can't reproduce it with randomly generated data.

Code

library(data.table)
dt <- readRDS('~/Downloads/data_debug_tan.rds')
dt2 <- data.table:::duplicated.data.table(dt)

Data

data_debug_tan.rds.zip

Debugged message

Error: C stack usage  8094212 is too close to the limit

[ben-schwen]

Its reproducible with the following setting:

• A wide data.table here with at least 500 columns (at least for my stack size)
• Duplicated rows

As @shrektan pointed out it's forderv and there void radix_r() what actually errors.

Example

x = matrix(rnorm(5000), nrow=10)
idx = sample(10, 20, TRUE)
DT = as.data.table(x[idx,])
forderv(DT, by=names(DT), sort=FALSE, retGrp=TRUE)
# Error: segfault from C stack overflow

Calling it a second time also changes the error to

x = matrix(rnorm(5000), nrow=10)
idx = sample(10, 20, TRUE)
DT = as.data.table(x[idx,])
forderv(DT, by=names(DT), sort=FALSE, retGrp=TRUE)
# Error in colnamesInt(x, by, check_dups = FALSE) : 
#   Internal error: savetl_init checks failed (0 100 0x556d9d49d540 0x556d9e05f430). please report to data.table issue tracker.

Error seems to have been introduced by 05c0d45, no segfault at 092fec3

edit:
What seems to be the problem here is that nradix gets so big that the recursion radix += 1 until radix+1 == nradix leads to the stack overflow.