Consistent Replacement of List Column with NULL (#6167)

Co-authored-by: Michael Chirico <[email protected]>

Author: joshhwuu

NEWS.md

@@ -133,6 +133,28 @@ rowwiseDT(
 
 19. Grouped queries on keyed tables no longer return an incorrectly keyed result if the _ad hoc_ `by=` list has some function call (in particular, a function which happens to return a strictly decreasing function of the keys), e.g. `by=.(a = rev(a))`, [#5583](https://github.com/Rdatatable/data.table/issues/5583). Thanks @AbrJA for the report and @MichaelChirico for the fix.
 
+20. Assigning `list(NULL)` to a list column now replaces the column with `list(NULL)`, instead of deleting the column [#5558](https://github.com/Rdatatable/data.table/issues/5558). This behavior is now consistent with base `data.frame`. Thanks @tdhock for the report and @joshhwuu for the fix. This is due to a fundamental ambiguity from both allowing list columns _and_ making the use of `list()` to wrap `j=` arguments optional. We think that the code behaves as expected in all cases now. See the below for some illustration:
+
+    ```r
+    DT = data.table(L=list(1L), i=2L, c='a')
+
+    DT[, i := NULL] # delete i
+    DT[, L := NULL] # delete L
+
+    DT[, i := list(NULL)] # overwrite: identical(DT$i, list(NULL))
+    # ^ ** THIS IS A CHANGE FROM PREVIOUS BEHAVIOR WHICH WOULD DELETE i **
+    DT[, L := list(NULL)] # assignment: identical(DT$L, list(NULL))
+
+    DT[, i := .(3L)]         # assignment: identical(DT$i, 3L)
+    DT[, i := .('a')]        # overwrite: identical(DT$i, 'a')
+    DT[, L := .(list(NULL))] # assignment: identical(DT$L, list(NULL))
+
+    DT[, c('L', 'i') := list(NULL, NULL)]       # delete L,i
+    DT[, c('L', 'i') := list(list(NULL), 3L)]   # assignment: identical(DT$L, list(NULL)), identical(DT$i, 3L)
+    DT[, c('L', 'i') := list(NULL, 3L)]         # delete L, assign to i
+    DT[, c('L', 'i') := list(list(NULL), NULL)] # assign to L, delete i
+    ```
+
 ## NOTES
 
 1. There is a new vignette on joins! See `vignette("datatable-joins")`. Thanks to Angel Feliz for authoring it! Feedback welcome. This vignette has been highly requested since 2017: [#2181](https://github.com/Rdatatable/data.table/issues/2181).

R/data.table.R

@@ -1153,7 +1153,8 @@ replace_dot_alias = function(e) {
             }
           }
           names(jsub)=""
-          jsub[[1L]]=as.name("list")
+          # dont wrap the RHS in list if it is a singular NULL and if not creating a new column
+          if (length(jsub[-1L]) == 1L && as.character(jsub[-1L]) == 'NULL' && all(lhs %chin% names_x)) jsub[[1L]]=as.name("identity") else jsub[[1L]]=as.name("list")
         }
         av = all.vars(jsub,TRUE)
         if (!is.atomic(lhs)) stopf("LHS of := must be a symbol, or an atomic vector (column names or positions).")

inst/tests/tests.Rraw

@@ -15447,7 +15447,7 @@ L = list(1:3, NULL, 4:6)
 test(2058.18, length(L), 3L)
 test(2058.19, as.data.table(L), data.table(V1=1:3, V2=4:6))  # V2 not V3        # no
 DT = data.table(a=1:3, b=c(4,5,6))
-test(2058.20, DT[,b:=list(NULL)], data.table(a=1:3))                            # no
+test(2058.20, DT[,b:=list(NULL)], data.table(a=1:3, b=list(NULL)))              # no
 
 # rbindlist improved error message, #3638
 DT = data.table(a=1)
@@ -20764,3 +20764,281 @@ test(2303.1, DT[, .N, by=.(b=rev(a))], data.table(b=2:1, N=1L))
 test(2303.2, DT[, .(N=1L), by=.(b=rev(a))], data.table(b=2:1, N=1L)) # ensure no interaction with GForce
 DT = data.table(a=2:3, b=1:0, key=c('a', 'b'))
 test(2303.3, DT[, .N, by=.(ab=a^b, d=c(1L, 1L))], data.table(ab=c(2, 1), d=1L, N=1L))
+
+# tests for new consistent replacement of list columns with list(NULL), #5558
+# replacement of a list column with list(NULL) in a single-row data.table, using different assignment methods
+DT = data.table(L=list("A"), i=1L)
+ans = data.table(L=list(NULL), i=1L)
+# test using replacement with $ operator
+DT$L = list(NULL)
+test(2304.001, DT, ans)
+DT = data.table(L=list("A"), i=1L)
+# standard form with := operator
+test(2304.002, copy(DT)[, L := list(NULL)], ans)
+# functional form with := operator
+test(2304.003, copy(DT)[, `:=`(L=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.004, copy(DT)[, let(L=list(NULL))], ans)
+# using set()
+test(2304.005, set(copy(DT), j="L", value=list(NULL)), ans)
+
+# replacement of multiple list columns with list(NULL) in a single-row data.table, using different assignment methods
+DT = data.table(L1=list("A"), L2=list("B"), i=1L)
+ans = data.table(L1=list(NULL), L2=list(NULL), i=1L)
+DT$L1 = list(NULL)
+DT$L2 = list(NULL)
+test(2304.006, DT, ans)
+DT = data.table(L1=list("A"), L2=list("B"), i=1L)
+# standard form with := operator
+test(2304.007, copy(DT)[, c("L1", "L2") := list(list(NULL), list(NULL))], ans)
+# functional form with := operator
+test(2304.008, copy(DT)[, `:=`(L1=list(NULL), L2=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.009, copy(DT)[, let(L1=list(NULL), L2=list(NULL))], ans)
+# using set()
+test(2304.010, set(copy(DT), j=c("L1", "L2"), value=list(list(NULL), list(NULL))), ans)
+
+# replacement of a list column with list(NULL) in a multi-row data.table, using different assignment methods
+DT = data.table(L=list("A", "B"), i=1L)
+ans = data.table(L=list(NULL, NULL), i=1L)
+# test using replacement with $ operator
+DT$L = list(NULL)
+test(2304.011, DT, ans)
+DT = data.table(L=list("A", "B"), i=1L)
+# standard form with := operator
+test(2304.012, copy(DT)[, L := list(NULL)], ans)
+# functional form with := operator
+test(2304.013, copy(DT)[, `:=`(L=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.014, copy(DT)[, let(L=list(NULL))], ans)
+# using set()
+test(2304.015, set(copy(DT), j="L", value=list(NULL)), ans)
+
+# replacement of multiple list columns with list(NULL) in a multi-row data.table, using different assignment methods
+DT = data.table(L1=list("A", "B"), L2=list("B", "C"), i=1L)
+ans = data.table(L1=list(NULL, NULL), L2=list(NULL, NULL), i=1L)
+DT$L1 = list(NULL)
+DT$L2 = list(NULL)
+test(2304.016, DT, ans)
+DT = data.table(L1=list("A", "B"), L2=list("B", "C"), i=1L)
+# standard form with := operator
+test(2304.017, copy(DT)[, c("L1", "L2") := list(list(NULL), list(NULL))], ans)
+# functional form with := operator
+test(2304.018, copy(DT)[, `:=`(L1=list(NULL), L2=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.019, copy(DT)[, let(L1=list(NULL), L2=list(NULL))], ans)
+# using set()
+test(2304.020, set(copy(DT), j=c("L1", "L2"), value=list(list(NULL), list(NULL))), ans)
+
+# Adding an empty list column to a single-row data.table, using different assignment methods
+DT = data.table(L=list("A"), i=1L)
+ans = data.table(L=list("A"), i=1L, D=list(NULL))
+warn = "Tried to assign NULL to column 'D', but this column does not exist to remove"
+# try to add a new empty list by list(NULL) with := in standard form, warns and does not change
+test(2304.021, copy(DT)[, D := list(NULL)], DT, warning=warn)
+test(2304.022, set(copy(DT), j="D", value=NULL), DT, warning=warn)
+test(2304.023, set(copy(DT), j="D", value=list(NULL)), DT, warning=warn)
+# add a new column by wrapping list(NULL), consistent with old behavior
+DT$D = list(list(NULL))
+test(2304.024, DT, ans)
+DT = data.table(L=list("A"), i=1L)
+# test adding empty list column in standard form with := operator
+test(2304.025, copy(DT)[, D := .(list(NULL))], ans)
+# functional form with := operator
+test(2304.026, copy(DT)[, `:=`(D=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.027, copy(DT)[, let(D=list(NULL))], ans)
+# using set()
+test(2304.028, set(copy(DT), j="D", value=list(list(NULL))), ans)
+
+# Adding multiple empty list columns to a single-row data.table, using different assignment methods
+DT = data.table(L=list("A"), i=1L)
+ans = data.table(L=list("A"), i=1L, D=list(NULL), R=list(NULL))
+DT$D = list(list(NULL))
+DT$R = list(list(NULL))
+test(2304.029, DT, ans)
+DT = data.table(L=list("A"), i=1L)
+# standard form with := operator
+test(2304.030, copy(DT)[, c("D", "R") := .(list(NULL))], ans)
+test(2304.031, copy(DT)[, c("D", "R") := .(list(NULL), list(NULL))], ans)
+# functional form with := operator
+test(2304.032, copy(DT)[, `:=`(D=list(NULL), R=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.033, copy(DT)[, let(D=list(NULL), R=list(NULL))], ans)
+# using set()
+test(2304.034, set(copy(DT), j=c("D", "R"), value=list(list(NULL))), ans)
+test(2304.035, set(copy(DT), j=c("D", "R"), value=list(list(NULL), list(NULL))), ans)
+
+# Adding an empty list column to a multi-row data.table, using different assignment methods
+DT = data.table(L=list("A", "B"), i=1L)
+ans = data.table(L=list("A", "B"), i=1L, D=list(NULL, NULL))
+warn = "Tried to assign NULL to column 'D', but this column does not exist to remove"
+# try to add a new empty list by list(NULL) with := in standard form, warns and does not change
+test(2304.036, copy(DT)[, D := list(NULL)], DT, warning=warn)
+test(2304.037, set(copy(DT), j="D", value=NULL), DT, warning=warn)
+test(2304.038, set(copy(DT), j="D", value=list(NULL)), DT, warning=warn)
+# add a new column by wrapping list(NULL), consistent with old behavior
+DT$D = list(list(NULL))
+test(2304.039, DT, ans)
+DT = data.table(L=list("A", "B"), i=1L)
+# test adding empty list column in standard form with := operator
+test(2304.040, copy(DT)[, D := .(list(NULL))], ans)
+# functional form with := operator
+test(2304.041, copy(DT)[, `:=`(D=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.042, copy(DT)[, let(D=list(NULL))], ans)
+# using set()
+test(2304.043, set(copy(DT), j="D", value = list(list(NULL))), ans)
+
+# Adding multiply empty list columns to a multi-row data.table, using different assignment methods
+DT = data.table(L=list("A", "B"), i=1L)
+ans = data.table(L=list("A", "B"), i=1L, D=list(NULL, NULL), R=list(NULL, NULL))
+DT$D = list(list(NULL))
+DT$R = list(list(NULL))
+test(2304.044, DT, ans)
+DT = data.table(L=list("A", "B"), i=1L)
+# standard form with := operator
+test(2304.045, copy(DT)[, c("D", "R") := .(list(NULL))], ans)
+test(2304.046, copy(DT)[, c("D", "R") := .(list(NULL), list(NULL))], ans)
+# functional form with := operator
+test(2304.047, copy(DT)[, `:=`(D=list(NULL), R=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.048, copy(DT)[, let(D=list(NULL), R=list(NULL))], ans)
+# using set()
+test(2304.049, set(copy(DT), j=c("D", "R"), value=list(list(NULL))), ans)
+test(2304.050, set(copy(DT), j=c("D", "R"), value=list(list(NULL), list(NULL))), ans)
+
+# Removal of a list column in a single-row data.table, using different assignment methods
+# NOTE: There is only one way to remove columns now, by assigning to NULL
+DT = data.table(L=list("A"), i=1L)
+ans = data.table(i=1L)
+# test removing a list column by assigning to NULL
+DT$L = NULL
+test(2304.051, DT, ans)
+DT = data.table(L=list("A"), i=1L)
+# standard form with := operator
+test(2304.052, copy(DT)[, L := NULL], ans)
+# functional form with := operator
+test(2304.053, copy(DT)[, `:=`(L=NULL)], ans)
+# functional form with 'let' alias
+test(2304.054, copy(DT)[, let(L=NULL)], ans)
+# using set()
+test(2304.055, set(copy(DT), j="L", value=NULL), ans)
+
+# Removal of multiple list columns in a single-row data.table, using different assignment methods
+DT = data.table(L1=list("A"), L2=list("B"), i=1L)
+# test removing two list columns by assigning to NULL
+DT$L1 = NULL
+DT$L2 = NULL
+test(2304.056, DT, ans)
+DT = data.table(L1=list("A"), L2=list("B"), i=1L)
+# standard form with := operator
+test(2304.057, copy(DT)[, c("L1", "L2") := NULL], ans)
+test(2304.058, copy(DT)[, c("L1", "L2") := .(NULL, NULL)], ans)
+# functional form with := operator
+test(2304.059, copy(DT)[, `:=`(L1=NULL, L2=NULL)], ans)
+# functional form with 'let' alias
+test(2304.060, copy(DT)[, let(L1=NULL, L2=NULL)], ans)
+# using set()
+test(2304.061, set(copy(DT), j=c("L1", "L2"), value=NULL), ans)
+test(2304.062, set(copy(DT), j=c("L1", "L2"), value=list(NULL, NULL)), ans)
+
+# Removal of a list column in a multi-row data.table, using different assignment methods
+DT = data.table(L=list("A", "B"), i=1L)
+ans = data.table(i=c(1L, 1L))
+# test removing a list column by assigning to NULL
+DT$L = NULL
+test(2304.063, DT, ans)
+DT = data.table(L=list("A", "B"), i=1L)
+# standard form with := operator
+test(2304.064, copy(DT)[, L := NULL], ans)
+# functional form with := operator
+test(2304.065, copy(DT)[, `:=`(L=NULL)], ans)
+# functional form with 'let' alias
+test(2304.066, copy(DT)[, let(L=NULL)], ans)
+# using set()
+test(2304.067, set(copy(DT), j="L", value=NULL), ans)
+
+# Removal of multiple list columns in a multi-row data.table, using different assignment methods
+DT = data.table(L1=list("A", "B"), L2=list("B", "C"), i=1L)
+# test removing two list columns by assigning to NULL
+DT$L1 = NULL
+DT$L2 = NULL
+test(2304.068, DT, ans)
+DT = data.table(L1=list("A", "B"), L2=list("B", "C"), i=1L)
+# standard form with := operator
+test(2304.069, copy(DT)[, c("L1", "L2") := NULL], ans)
+test(2304.070, copy(DT)[, c("L1", "L2") := .(NULL, NULL)], ans)
+# functional form with := operator
+test(2304.071, copy(DT)[, `:=`(L1=NULL, L2=NULL)], ans)
+# functional form with 'let' alias
+test(2304.072, copy(DT)[, let(L1=NULL, L2=NULL)], ans)
+# using set()
+test(2304.073, set(copy(DT), j=c("L1", "L2"), value=NULL), ans)
+test(2304.074, set(copy(DT), j=c("L1", "L2"), value=list(NULL, NULL)), ans)
+
+# Combining queries (add/remove/replace columns in the same query) for a single-row data.table
+
+# test for adding a new empty list column D and removing column L in the same query
+DT = data.table(L=list("A"), i=1L)
+ans = data.table(i=1L, D=list(NULL))
+test(2304.075, copy(DT)[, c("L", "D") := list(NULL, list(NULL))], ans)
+test(2304.076, copy(DT)[, `:=`(L=NULL, D=list(NULL))], ans)
+test(2304.077, copy(DT)[, let(L=NULL, D=list(NULL))], ans)
+test(2304.078, set(copy(DT), j=c("L", "D"), value=list(NULL, list(NULL))), ans)
+
+# test for adding a new empty list column D and replacing column L with empty list in the same query
+DT = data.table(L=list("A"), i=1L)
+ans = data.table(L=list(NULL), i=1L, D=list(NULL))
+test(2304.079, copy(DT)[, c("L", "D") := list(list(NULL), list(NULL))], ans)
+test(2304.080, copy(DT)[, `:=`(L=list(NULL), D=list(NULL))], ans)
+test(2304.081, copy(DT)[, let(L=list(NULL), D=list(NULL))], ans)
+test(2304.082, set(copy(DT), j=c("L", "D"), value=list(list(NULL), list(NULL))), ans)
+
+# test for replacing column L with an empty list and removing list column D in the same query
+DT = data.table(L=list("A"), D=list("B"), i=1L)
+ans = data.table(L=list(NULL), i=1L)
+test(2304.083, copy(DT)[, c("L", "D") := list(list(NULL), NULL)], ans)
+test(2304.084, copy(DT)[, `:=`(L=list(NULL), D=NULL)], ans)
+test(2304.085, copy(DT)[, let(L=list(NULL), D=NULL)], ans)
+test(2304.086, set(copy(DT), j=c("L", "D"), value=list(list(NULL), NULL)), ans)
+
+# test for combining add, replace, remove in the same query
+DT = data.table(L=list("A"), D=list("B"), i=1L)
+ans = data.table(L=list(NULL), i=1L, E=list(NULL))
+test(2304.087, copy(DT)[, c("L", "D", "E") := list(list(NULL), NULL, list(NULL))], ans)
+test(2304.088, copy(DT)[, `:=`(L=list(NULL), D=NULL, E=list(NULL))], ans)
+test(2304.089, copy(DT)[, let(L=list(NULL), D=NULL, E=list(NULL))], ans)
+test(2304.090, set(copy(DT), j=c("L", "D", "E"), value=list(list(NULL), NULL, list(NULL))), ans)
+
+# sub-assignment of list column with list(NULL) in a multi-row data.table, using different assignment methods
+DT = data.table(L=list("A", "B"), i=1L)
+ans = data.table(L=list("A", NULL), i=1L)
+# test using replacement with $ operator
+DT$L[2L] = list(NULL)
+test(2304.091, DT, ans)
+DT = data.table(L=list("A", "B"), i=1L)
+# standard form with := operator
+test(2304.092, copy(DT)[2L, L := list(list(NULL))], ans)
+# functional form with := operator
+test(2304.093, copy(DT)[2L, `:=`(L=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.094, copy(DT)[2L, let(L=list(NULL))], ans)
+# using set()
+test(2304.095, set(copy(DT), i=2L, j="L", value=list(list(NULL))), ans)
+
+# sub-assignment of multiple list columns with list(NULL) in a multi-row data.table, using different assignment methods
+DT = data.table(L1=list("A", "B"), L2=list("B", "C"), i=1L)
+ans = data.table(L1=list("A", NULL), L2=list("B", NULL), i=1L)
+DT$L1[2L] = list(NULL)
+DT$L2[2L] = list(NULL)
+test(2304.096, DT, ans)
+DT = data.table(L1=list("A", "B"), L2=list("B", "C"), i=1L)
+# standard form with := operator
+test(2304.097, copy(DT)[2L, c("L1", "L2") := list(list(NULL), list(NULL))], ans)
+# functional form with := operator
+test(2304.098, copy(DT)[2L, `:=`(L1=list(NULL), L2=list(NULL))], ans)
+# functional form with 'let' alias
+test(2304.099, copy(DT)[2L, let(L1=list(NULL), L2=list(NULL))], ans)
+# using set()
+test(2304.100, set(copy(DT), i=2L, j=c("L1", "L2"), value=list(list(NULL), list(NULL))), ans)

man/assign.Rd

@@ -78,6 +78,8 @@ When \code{LHS} is a factor column and \code{RHS} is a character vector with ite
 Unlike \code{<-} for \code{data.frame}, the (potentially large) LHS is not coerced to match the type of the (often small) RHS. Instead the RHS is coerced to match the type of the LHS, if necessary. Where this involves double precision values being coerced to an integer column, a warning is given when fractional data is truncated. It is best to get the column types correct up front and stick to them. Changing a column type is possible but deliberately harder: provide a whole column as the RHS. This RHS is then \emph{plonked} into that column slot and we call this \emph{plonk syntax}, or \emph{replace column syntax} if you prefer. By needing to construct a full length vector of a new type, you as the user are more aware of what is happening and it is clearer to readers of your code that you really do intend to change the column type; e.g., \code{DT[, colA:=as.integer(colA)]}. A plonk occurs whenever you provide a RHS value to `:=` which is \code{nrow} long. When a column is \emph{plonked}, the original column is not updated by reference because that would entail updating every single element of that column whereas the plonk is just one column pointer update.
 
 \code{data.table}s are \emph{not} copied-on-change by \code{:=}, \code{setkey} or any of the other \code{set*} functions. See \code{\link{copy}}.
+
+While in most cases standard and functional form of \code{:=} are interchangeable, there are some minor differences in the way that \code{RHS} is handled. In the functional form, \code{:=} operator behaves like an alias to \code{list}. This means that when \code{RHS} is a list, \code{LHS} is assigned a list. Avoid this by using the standard form when \code{RHS} is a list, or use a vector. See \href{../doc/datatable-reference-semantics.html}{\code{vignette("datatable-reference-semantics")}} for examples.
 }
 
 \section{Advanced (internals):}{It is easy to see how \emph{sub-assigning} to existing columns is done internally. Removing columns by reference is also straightforward by modifying the vector of column pointers only (using memmove in C). However adding (new) columns is more tricky as to how the \code{data.table} can be grown \emph{by reference}: the list vector of column pointers is \emph{over-allocated}, see \code{\link{truelength}}. By defining \code{:=} in \code{j} we believe update syntax is natural, and scales, but it also bypasses \code{[<-} dispatch and allows \code{:=} to update by reference with no copies of any part of memory at all.

src/assign.c

@@ -549,6 +549,11 @@ SEXP assign(SEXP dt, SEXP rows, SEXP cols, SEXP newcolnames, SEXP values)
   for (int i=0; i<length(cols); ++i) {
     coln = INTEGER(cols)[i]-1;
     SEXP thisvalue = RHS_list_of_columns ? VECTOR_ELT(values, i) : values;
+    // if values is list(NULL), then replace with a list of NULLs instead of deleting, #5558
+    if (RHS_list_of_columns && length(values)==1 && TYPEOF(VECTOR_ELT(values, 0))==NILSXP && coln < oldncol) {
+        SET_VECTOR_ELT(dt, coln, targetcol=allocNAVector(VECSXP, length(VECTOR_ELT(dt, coln))));
+        continue;
+    }
     if (TYPEOF(thisvalue)==NILSXP) {
       if (!isNull(rows)) internal_error(__func__, "earlier error 'When deleting columns, i should not be provided' did not happen"); // # nocov
       ndelete++;