Skip to content

incorrect output when using pivot_longer with values_drop_na = TRUE #484

@wjchulme

Description

@wjchulme

There appears to be an issue with the values_drop_na = TRUE option when using pivot_longer with dtplyr. The names_to columns end up as odd character-rendered list-cols things, rather than simple vectors. See example:

library('dtplyr')
library('dplyr')
library('tidyr')

example.data <- data.frame(
  pid = c(1,2,3,1,2,3),
  valueA_1 = runif(6),
  valueA_2 = NA_real_,
  valueB_1 = runif(6),
  valueA_2 = NA_real_,
  type_1 = c("A","B", "C", "C", "B", "B"),
  type_2 = NA_character_
)

example.data %>%
  lazy_dt() %>%
  pivot_longer(
    cols = -pid,
    names_to = c(".value", "index"),
    names_pattern = "^(.*)_(\\d+)",
    values_drop_na = TRUE,
  ) %>% 
  as_tibble()

Which produces:

# A tibble: 6 × 5
    pid index                                                                                   valueA valueB type 
  <dbl> <chr>                                                                                    <dbl>  <dbl> <chr>
1     1 "c(\"1\", \"1\", \"1\", \"1\", \"1\", \"1\", \"2\", \"2\", \"2\", \"2\", \"2\", \"2\")"  0.266 0.945  A    
2     2 "c(\"1\", \"1\", \"1\", \"1\", \"1\", \"1\", \"2\", \"2\", \"2\", \"2\", \"2\", \"2\")"  0.372 0.661  B    
3     3 "c(\"1\", \"1\", \"1\", \"1\", \"1\", \"1\", \"2\", \"2\", \"2\", \"2\", \"2\", \"2\")"  0.573 0.629  C    
4     1 "c(\"1\", \"1\", \"1\", \"1\", \"1\", \"1\", \"2\", \"2\", \"2\", \"2\", \"2\", \"2\")"  0.908 0.0618 C    
5     2 "c(\"1\", \"1\", \"1\", \"1\", \"1\", \"1\", \"2\", \"2\", \"2\", \"2\", \"2\", \"2\")"  0.202 0.206  B    
6     3 "c(\"1\", \"1\", \"1\", \"1\", \"1\", \"1\", \"2\", \"2\", \"2\", \"2\", \"2\", \"2\")"  0.898 0.177  B    
Warning message:
In `[.data.table`(melt(`_DT27`, measure.vars = list(c("valueA_1",  :
  Coercing 'list' RHS to 'character' to match the type of column 2 named 'index'.

The correct output (which can be seen by removing the lazy_dt() line in the above code) should be:

# A tibble: 6 × 5
    pid index valueA valueB type 
  <dbl> <chr>  <dbl>  <dbl> <chr>
1     1 1      0.266 0.945  A    
2     2 1      0.372 0.661  B    
3     3 1      0.573 0.629  C    
4     1 1      0.908 0.0618 C    
5     2 1      0.202 0.206  B    
6     3 1      0.898 0.177  B   

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugan unexpected problem or unintended behavior

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions