Storing xgboost models in list columns fails

[be-marc]

Not sure if an xgboost or data.table bug. Storing xgboost models in list columns fails with the new xgboost version 3.1.2.1.

library(data.table)
library(xgboost)

data(mtcars)

model_regression = xgboost(mtcars[, -1], mtcars$mpg, nthreads = 1, nrounds = 3)

data.table(model = list(model_regression))

# Error in y[] <- lapply(y, reallocate) : 
#   ALTLIST classes must provide a Set_elt method [class: XGBAltrepPointerClass, pkg: xgboost]

See also dmlc/xgboost#11849

[MichaelChirico]

This looks at a glance to be an {xgboost} bug -- you should be able to reproduce the error without any {data.table} code, IINM.

The error is telling us that an ALTREP class provided by {xgboost} is incomplete, again IINM.

[aitap]

data.table::copy() works recursively just in case there is another data.table inside the list being copied (#4205), which conflicts with xgboost's ALTREP wrapper. They need the wrapper so that their models could be serialized transparently, and want the altlist to be immutable. There's probably a way to let data.table::copy() avoid trying to overwrite list entries that don't need overwriting, but it's not trivial. (Ideally we'd use reference counts instead having copy() construct special attributes with pointers, but that's far ahead in the future.) If you need a horrible workaround right now, try hiding the object inside the attributes of the list item, which copy() is currently blind to: data.table(model = list( structure('wrapper', real_model = model_regression) ))

[david-cortes]

This looks at a glance to be an {xgboost} bug -- you should be able to reproduce the error without any {data.table} code, IINM.

The error is telling us that an ALTREP class provided by {xgboost} is incomplete, again IINM.

This is the intended behavior in xgboost. The model objects now have an altlist class on them which has a single entry containing an external C++ pointer. This list is not meant to be modified by users, as it contains logic for serialization of the C++ object behind the pointer. Adding other altlist methods would break the logic, because then R might turn the altlist to a regular list as the result of some operations.

It would need a fix from the data.table side to not attempt to modify these objects.

[aitap]

A quick workaround on our side would be to skip any further processing if inherits(newx, 'xgb.Booster'). Is this the right class? Any other classes we should skip as well?

[david-cortes]

As far as the list part of xgboost is concerned: yes, that should probably work. But other than the list structure, xgboost model objects can also have R attributes, some of which can contain data.table objects. Not sure if that'd be an issue.

But it feels like this would require more of a solution to avoid modifying objects when not needed instead of just adding a workaround for xgboost in specific. Other objects from other packages may also follow a similar logic, and would still break in this case.

[jangorecki]

Haven't look into it much but sounds like having generic method copy() could allow for custom classes to control their behavior