R: setting weights_column = NULL causes unwanted variables to be used as predictors

[exalate-issue-sync]

Setting "weights_column = NULL" explicitly when calling h2o.gbm (or h2o.randomForest) causes unwanted variables to be used as predictors. The following example illustrates this issue using the prostate data that comes with the h2o R package.

library(h2o)
h2o.init()
pros_path <- system.file("extdata", "prostate.csv", package="h2o")
pros_data <- h2o.importFile(pros_path)

pros_data$CAPSULE <- as.factor(pros_data$CAPSULE)

WITHOUT setting "weights_column = NULL" explicitly in h2o.gbm, things work fine.

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli")

h2o.varimp(model)
#Variable Importances:

variable relative_importance scaled_importance percentage

#1 GLEASON 112.808281 1.000000 0.352302
#2 PSA 70.384674 0.623932 0.219813
#3 VOL 51.909214 0.460154 0.162113
#4 DPROS 42.125877 0.373429 0.131560
#5 AGE 36.616043 0.324586 0.114353
#6 RACE 3.215139 0.028501 0.010041
#7 DCAPS 3.143716 0.027868 0.009818

Setting "weights_column = NULL" explicitly in h2o.gbm causes ID (not selected # initially) to be used as a predictor.

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", weights_column = NULL)

h2o.varimp(model)
#Variable Importances:

variable relative_importance scaled_importance percentage

#1 GLEASON 114.578041 1.000000 0.331926
#2 ID 55.921295 0.488063 0.162001
#3 PSA 54.399891 0.474785 0.157593
#4 VOL 39.352005 0.343452 0.114000
#5 DPROS 37.653904 0.328631 0.109081
#6 AGE 34.698425 0.302837 0.100519
#7 DCAPS 6.719040 0.058642 0.019465
#8 RACE 1.869231 0.016314 0.005415

[exalate-issue-sync]

Arno Candel commented: model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli")

Method: POST , URI: /3/ModelBuilders/gbm/parameters, route: /3/ModelBuilders/gbm/parameters, parms: {distribution=bernoulli, training_frame=prostate.hex_136, response_column=CAPSULE, ignored_columns=["ID"]}

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", weights_column = NULL)

Method: POST , URI: /3/ModelBuilders/gbm/parameters, route: /3/ModelBuilders/gbm/parameters, parms: {distribution=bernoulli, training_frame=prostate.hex_136, response_column=CAPSULE, ignored_columns=[""]}

Bug: ignored_columns is set to "" by the R client in models.R .h2o.prepareModelParameters, not clear how and when though.

[exalate-issue-sync]

Erin LeDell commented: This affects all supervised algos (glm, gbm, deeplearning, randomForest) in the h2o R package and can be triggered by setting any of the following three args explicitly to NULL: weights_column, fold_column, offset_column

This is now fixed in master: 221bdf0

[exalate-issue-sync]

Erin LeDell commented: R Code that shows weights_column is fixed for all algos:

library(h2o)
conn <- h2o.init()
pros_path <- system.file("extdata", "prostate.csv", package="h2o")
pros_data <- h2o.importFile(pros_path)

WITHOUT setting "weights_column = NULL" explicitly in h2o.gbm

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli")
h2o.varimp(model)

Variable Importances:

variable relative_importance scaled_importance percentage

1 GLEASON 112.808281 1.000000 0.352302

2 PSA 70.384674 0.623932 0.219813

3 VOL 51.909214 0.460154 0.162113

4 DPROS 42.125877 0.373429 0.131560

5 AGE 36.616043 0.324586 0.114353

6 RACE 3.215139 0.028501 0.010041

7 DCAPS 3.143716 0.027868 0.009818

Setting "weights_column = NULL" now gives same results

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", weights_column = NULL)
h2o.varimp(model)

Variable Importances:

variable relative_importance scaled_importance percentage

1 GLEASON 112.808281 1.000000 0.352302

2 PSA 70.384674 0.623932 0.219813

3 VOL 51.909214 0.460154 0.162113

4 DPROS 42.125877 0.373429 0.131560

5 AGE 36.616043 0.324586 0.114353

6 RACE 3.215139 0.028501 0.010041

7 DCAPS 3.143716 0.027868 0.009818

Correctly removes from the model

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", weights_column = "DPROS")
h2o.varimp(model)

Variable Importances:

variable relative_importance scaled_importance percentage

1 GLEASON 269.331573 1.000000 0.328470

2 PSA 232.714966 0.864046 0.283813

3 VOL 156.291687 0.580295 0.190609

4 AGE 139.613373 0.518370 0.170269

5 DCAPS 12.626597 0.046881 0.015399

6 RACE 9.379818 0.034826 0.011439

model <- h2o.gbm(x = c(1:9), y = 2, pros_data, distribution = "bernoulli", weights_column = "DPROS")

WITHOUT setting "weights_column = NULL" explicitly in h2o.randomForest

model <- h2o.randomForest(x = c(3:9), y = 2, pros_data)
h2o.varimp(model)

Variable Importances:

variable relative_importance scaled_importance percentage

1 PSA 819.725952 1.000000 0.275316

2 AGE 635.977844 0.775842 0.213602

3 GLEASON 605.755920 0.738974 0.203451

4 VOL 414.318298 0.505435 0.139154

5 DPROS 371.036652 0.452635 0.124618

6 DCAPS 86.069611 0.104998 0.028908

7 RACE 44.518166 0.054309 0.014952

Setting "weights_column = NULL" now gives same results

model <- h2o.randomForest(x = c(3:9), y = 2, pros_data, weights_column = NULL)
h2o.varimp(model)

Variable Importances:

variable relative_importance scaled_importance percentage

1 PSA 860.105530 1.000000 0.285723

2 AGE 644.754333 0.749622 0.214184

3 GLEASON 562.964172 0.654529 0.187014

4 VOL 404.870544 0.470722 0.134496

5 DPROS 383.326355 0.445674 0.127339

6 DCAPS 103.269661 0.120066 0.034306

7 RACE 50.991951 0.059286 0.016939

Correctly removes DPROS from the model

model <- h2o.randomForest(x = c(3:9), y = 2, pros_data, weights_column = "DPROS")
h2o.varimp(model)

Variable Importances:

variable relative_importance scaled_importance percentage

1 PSA 2283.783936 1.000000 0.327249

2 AGE 1664.619629 0.728887 0.238527

3 GLEASON 1404.228149 0.614869 0.201215

4 VOL 1111.190918 0.486557 0.159225

5 DCAPS 365.907684 0.160220 0.052432

6 RACE 149.008423 0.065246 0.021352

model <- h2o.randomForest(x = c(1:9), y = 2, pros_data, weights_column = "DPROS")
h2o.varimp(model)

WITHOUT setting "weights_column = NULL" explicitly in h2o.glm

model <- h2o.glm(x = c(3:9), y = 2, pros_data)
h2o.varimp(model)

Standardized Coefficient Magnitudes:

names coefficients sign

1 GLEASON 0.159514 POS

2 DPROS 0.092240 POS

3 PSA 0.073741 POS

4 VOL 0.036598 NEG

5 DCAPS 0.030156 POS

6 RACE 0.025248 NEG

7 AGE 0.011957 NEG

Setting "weights_column = NULL" now gives same results

model <- h2o.glm(x = c(3:9), y = 2, pros_data, weights_column = NULL)
h2o.varimp(model)

Standardized Coefficient Magnitudes:

names coefficients sign

1 GLEASON 0.159514 POS

2 DPROS 0.092240 POS

3 PSA 0.073741 POS

4 VOL 0.036598 NEG

5 DCAPS 0.030156 POS

6 RACE 0.025248 NEG

7 AGE 0.011957 NEG

Correctly removes DPROS from the model

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, weights_column = "DPROS")
h2o.varimp(model)

Variable Importances:

variable relative_importance scaled_importance percentage

1 GLEASON 269.331573 1.000000 0.328470

2 PSA 232.714966 0.864046 0.283813

3 VOL 156.291687 0.580295 0.190609

4 AGE 139.613373 0.518370 0.170269

5 DCAPS 12.626597 0.046881 0.015399

6 RACE 9.379818 0.034826 0.011439

model <- h2o.gbm(x = c(1:9), y = 2, pros_data, weights_column = "DPROS")
h2o.varimp(model)

WITHOUT setting "weights_column = NULL" explicitly in h2o.deeplearning

model <- h2o.deeplearning(x = c(3:9), y = 2, pros_data, distribution = "bernoulli")
model@parameters #since we dont have varimp in DL, just look at params instead

$training_frame

[1] "prostate.hex_1"

$seed

[1] 3.457833e+18

$distribution

[1] "bernoulli"

$x

[1] "AGE" "RACE" "DPROS" "DCAPS" "PSA" "VOL" "GLEASON"

$y

[1] "CAPSULE"

Setting "weights_column = NULL" now gives same results

model <- h2o.deeplearning(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", weights_column = NULL)
model@parameters

$training_frame

[1] "prostate.hex_1"

$seed

[1] 3.457833e+18

$distribution

[1] "bernoulli"

$x

[1] "AGE" "RACE" "DPROS" "DCAPS" "PSA" "VOL" "GLEASON"

$y

[1] "CAPSULE"

Correctly removes from the model (still shows up in args$x, but that is correct)

model <- h2o.deeplearning(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", weights_column = "DPROS")
model@parameters
model <- h2o.deeplearning(x = c(1:9), y = 2, pros_data, distribution = "bernoulli", weights_column = "DPROS")
model@parameters

[exalate-issue-sync]

Erin LeDell commented: R Code that shows fold_column is fixed for all algos:

library(h2o)
conn <- h2o.init()
pros_path <- system.file("extdata", "prostate.csv", package="h2o")
pros_data <- h2o.importFile(pros_path)

WITHOUT setting "fold_column = NULL" explicitly in h2o.gbm

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli")
h2o.varimp(model)

Setting "fold_column = NULL" now gives same results

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", fold_column = NULL)
h2o.varimp(model)

Correctly removes from the model

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", fold_column = "DPROS")
h2o.varimp(model)
model <- h2o.gbm(x = c(1:9), y = 2, pros_data, distribution = "bernoulli", fold_column = "DPROS")
h2o.varimp(model)

WITHOUT setting "fold_column = NULL" explicitly in h2o.randomForest

model <- h2o.randomForest(x = c(3:9), y = 2, pros_data)
h2o.varimp(model)

Setting "fold_column = NULL" now gives same results

model <- h2o.randomForest(x = c(3:9), y = 2, pros_data, fold_column = NULL)
h2o.varimp(model)

Correctly removes DPROS from the model

model <- h2o.randomForest(x = c(3:9), y = 2, pros_data, fold_column = "DPROS")
h2o.varimp(model)
model <- h2o.randomForest(x = c(1:9), y = 2, pros_data, fold_column = "DPROS")
h2o.varimp(model)

WITHOUT setting "fold_column = NULL" explicitly in h2o.glm

model <- h2o.glm(x = c(3:9), y = 2, pros_data)
h2o.varimp(model)

Setting "fold_column = NULL" now gives same results

model <- h2o.glm(x = c(3:9), y = 2, pros_data, fold_column = NULL)
h2o.varimp(model)

Correctly removes DPROS from the model

model <- h2o.gbm(x = c(3:9), y = 2, pros_data, fold_column = "DPROS")
h2o.varimp(model)
model <- h2o.gbm(x = c(1:9), y = 2, pros_data, fold_column = "DPROS")
h2o.varimp(model)

WITHOUT setting "fold_column = NULL" explicitly in h2o.deeplearning

model <- h2o.deeplearning(x = c(3:9), y = 2, pros_data, distribution = "bernoulli")
model@parameters #since we dont have varimp in DL, just look at params instead
model <- h2o.deeplearning(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", fold_column = NULL)
model@parameters
model <- h2o.deeplearning(x = c(3:9), y = 2, pros_data, distribution = "bernoulli", fold_column = "DPROS")
model@parameters
model <- h2o.deeplearning(x = c(1:9), y = 2, pros_data, distribution = "bernoulli", fold_column = "DPROS")

[exalate-issue-sync]

Erin LeDell commented: R Code that shows offset_column is fixed for all algos:

library(h2o)
conn <- h2o.init()
pros_path <- system.file("extdata", "prostate.csv", package="h2o")
pros_data <- h2o.importFile(pros_path)

Add an offset col that has offsets of a reasonable size

pros_data$offset_col <- pros_data$DPROS/10

WITHOUT setting "offset_column = NULL" explicitly in h2o.gbm

model <- h2o.gbm(x = c(3:10), y = 2, pros_data, distribution = "bernoulli")
h2o.varimp(model)

Setting "offset_column = NULL" now gives same results

model <- h2o.gbm(x = c(3:10), y = 2, pros_data, distribution = "bernoulli", offset_column = NULL)
h2o.varimp(model)

Correctly removes from the model

model <- h2o.gbm(x = c(3:10), y = 2, pros_data, distribution = "bernoulli", offset_column = "offset_col")
h2o.varimp(model)
model <- h2o.gbm(x = c(1:10), y = 2, pros_data, distribution = "bernoulli", offset_column = "offset_col")
h2o.varimp(model)

Note: offset_column not supporrted for DRF binomial

WITHOUT setting "weights_column = NULL" explicitly in h2o.glm

model <- h2o.glm(x = c(3:10), y = 2, pros_data)
h2o.varimp(model)

Setting "offset_column = NULL" now gives same results

model <- h2o.glm(x = c(3:10), y = 2, pros_data, offset_column = NULL)
h2o.varimp(model)

Correctly removes offset_col from the model

model <- h2o.gbm(x = c(3:10), y = 2, pros_data, offset_column = "offset_col")
h2o.varimp(model)
model <- h2o.gbm(x = c(1:10), y = 2, pros_data, offset_column = "offset_col")
h2o.varimp(model)

WITHOUT setting "offset_column = NULL" explicitly in h2o.deeplearning

model <- h2o.deeplearning(x = c(3:10), y = 2, pros_data, distribution = "bernoulli")
model@parameters #since we dont have varimp in DL, just look at params instead
model <- h2o.deeplearning(x = c(3:10), y = 2, pros_data, distribution = "bernoulli", offset_column = NULL)
model@parameters
model <- h2o.deeplearning(x = c(3:10), y = 2, pros_data, distribution = "bernoulli", offset_column = "offset_col")
model@parameters
model <- h2o.deeplearning(x = c(1:10), y = 2, pros_data, distribution = "bernoulli", offset_column = "offset_col")

[DinukaH2O]

JIRA Issue Migration Info

Jira Issue: PUBDEV-1972
Assignee: Erin LeDell
Reporter: Avni Wadhwa
State: Resolved
Fix Version: N/A
Attachments: N/A
Development PRs: N/A