predict crash on row count > 10000 and col count = 396

[liuauto]

I have a modestly wide data set, which has 10 categorical features and the total columns after the one hot transformation is 396. The predict function crashes R in this use case.

I am not sure where is the problem. The train works on the similar size of the data. Not sure what the root cause is. Has this package been through some stress testing?

[liuauto]

Turns out that gc() must be called very frequently to ensure the predict() function not crashing R. Interesting to investigate...

[gravesee]

Can you post specifics of your dataset? When I run the following code everything is fine:

n <- 10000
df <- data.frame(replicate(10, sample(make.names(1:40), n, T)))
oh <- onehot(df, max_levels = 100)
for (i in 1:100) m <- predict(oh, data = df)

The levels of your factor variables would be a start, I think. Are the columns balanced (i.e. are the level counts similar for all fields)?

[liuauto]

Do I need to set the max_levels properly? It is kind of imbalanced though.

When the # of rows exceeds 10,000 it will start to crash

Cannot share the actual data but here is the output of
str(onehot.model) , with actual names replaced/removed:

List of 10
$ F1 :List of 3
..$ name : chr "F1"
..$ type : chr "factor"
..$ levels: chr [1:22]
$ C1 :List of 3
..$ name : chr "C1"
..$ type : chr "factor"
..$ levels: chr [1:101]
$ S1 :List of 3
..$ name : chr "S1"
..$ type : chr "factor"
..$ levels: chr [1:246]
$ P1 :List of 3
..$ name : chr "P1"
..$ type : chr "factor"
..$ levels: chr [1:3]
$ R1 :List of 3
..$ name : chr "R1"
..$ type : chr "factor"
..$ levels: chr [1:4]
$ CU1 :List of 3
..$ name : chr "CU1"
..$ type : chr "factor"
..$ levels: chr [1:3]
$ OP1 :List of 3
..$ name : chr "OP1"
..$ type : chr "factor"
..$ levels: chr [1:2]
$ TE1 :List of 3
..$ name : chr "TE1"
..$ type : chr "factor"
..$ levels: chr [1:2]
$ DI1:List of 3
..$ name : chr "DI1"
..$ type : chr "factor"
..$ levels: chr [1:2]
$ B1 :List of 3
..$ name : chr "B1"
..$ type : chr "factor"
..$ levels: chr [1:2]
- attr(*, "class")= chr "onehot"
- attr(*, "call")= language onehot(data = train.df[, factor.features], stringsAsFactors = TRUE, max_levels = 1000)

The workaround is to split the data into smaller chunks and call predict(), also need to call gc() after each step. It will crash if running a single predict over the entire data

[gravesee]

Sorry, I cannot replicate this... I just ran this script with N set to 100,000:

library(onehot)
N <- 100000
v <- c(F1=22, C1=101, S1=246, P1=3, R1=4, CU1=3, OP1=2, TE1=2, DI1=2, B1=2)
lvls <- lapply(v, function(x) make.names(1:x))
df <- data.frame(lapply(lvls, sample, N, TRUE))
oh <- onehot(df[,names(v)], max_levels = 1000)
microbenchmark::microbenchmark(m <- predict(oh, df))

And the timing output is shown below:

Unit: milliseconds
                 expr      min       lq     mean   median       uq      max neval
 m <- predict(oh, df) 587.3384 670.9076 881.4207 712.2625 941.0636 2156.051   100

Here's my version info. I'm on windows 10. What OS are you using?

platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          5.1                         
year           2018                        
month          07                          
day            02                          
svn rev        74947                       
language       R                           
version.string R version 3.5.1 (2018-07-02)
nickname       Feather Spray               

[liuauto]

Thanks for testing on your side!
I tried the similar operations ( train and score the same data), and it has no issues. So I think it is not a performance/scaling issue.
It only crashes when I applied the trained model to the test case.
My training set and the test set has no overlaps. I guess there must be some missing levels in some of the categorical variables. I need to go back and remove/add some features to locate the root cause.

[liuauto]

I think I can reproduce the crash now. If there are too many unseen levels, the predict will crash. I guess I need to use addNA option in this scenario.

library(onehot)
N <- 100000
v <- c(F1=22, C1=101, S1=246, P1=3, R1=4, CU1=3, OP1=2, TE1=2, DI1=2, B1=2)
lvls <- lapply(v, function(x) make.names(1:x))
df <- data.frame(lapply(lvls, sample, N, TRUE))
oh <- onehot(df[,names(v)], max_levels = 1000)`

If we add new levels to S1, it will crash. (It will not crash if there are only a few new levels though).

 N <- 100000
 v <- c(F1=22, C1=101, S1=300, P1=3, R1=4, CU1=3, OP1=2, TE1=2, DI1=2, B1=2)
 lvls <- lapply(v, function(x) make.names(1:x))
 test.df <- data.frame(lapply(lvls, sample, N, TRUE))
 m <- predict(oh, test.df)

*** caught segfault ***
address 0x7f30ee162440, cause 'invalid permissions'

Traceback:
1: .Call("predict_onehot", object, data[names(object)])
2: c_predict_onehot(object, data)
3: predict.onehot(oh, test.df)
4: predict(oh, test.df)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
R is aborting now ...
Segmentation fault

[gravesee]

Excellent, thank you for the test case. I will troubleshoot and get back to you.

[gravesee]

Ok, so the CRAN version used a lot of C code that is likely causing this issue with levels that don't match. If you use the latest version from github installed with devtools there is now an error and not a crash because everything was refactored to R.

[liuauto]

Thanks! It works for me now.
Are there any future plans to handle this unseen level automatically? Something like generating all zeros or adding a column X_unknown.

[liuauto]

Seems that enforcing the levels can solve the problem.

 test.df$S1 <- factor( test.df$S1, levels = levels(df$S1))