Bug 16750: improve summary.default() for character vectors

[bastistician]

On Bugzilla: https://bugs.r-project.org/show_bug.cgi?id=16750
Currently,

R> summary(LETTERS)
   Length     Class      Mode 
       26 character character 

is not very useful. This is also the only information that is presented for character columns in data frames (which occur more frequently since the stringsAsFactors=FALSE change in R 4.0.0), compare:

R> summary(data.frame("character" = LETTERS, "factor" = factor(LETTERS)))
  character             factor  
 Length:26          A      : 1  
 Class :character   B      : 1  
 Mode  :character   C      : 1  
                    D      : 1  
                    E      : 1  
                    F      : 1  
                    (Other):20  

A character vector is not necessarily categorical, so always using the factor-style summary is not ideal either. Furthermore, it should remain clear from the data-frame summary if a variable is a factor or "bare" character.

summary.default() has an argument quantile.type to customize numeric summaries. For character input, the idea would be to have an argument character.proxy = c("nchar", "factor", "none"), say, where "none" corresponds to the current generic summary (for back-compatibility), "nchar" produces a summary based on nchar(object), e.g.

       Mode   nchar.min    nchar.max    NA's
  character           0            1       2

(where the last element is only included if >0, similar to numeric summaries) and "factor" uses summary.factor (but still includes the Mode as the first element). An alternative would be to have options "nchar2" and "nchar5" to use range() and quantile() summaries, respectively.

[llrs]

Perhaps the summary could have length(unique(LETTERS)) if > 7 (to match the (Other) factor output)? Knowing the length and how many are repeated is often very helpful.

[bastistician]

I agree that the number of unique elements should be considered, too. Maybe a summary.character could have a variant character.proxy = "both" to be used by summary.data.frame that shows nchar.min, nchar.max, distinct, followed by summary.factor() with maxsum=2 or 3 to only show the most frequent element(s) and NA's. I think the number of summary elements in this case should not exceed 7 to align with the summaries for factor and numeric columns.

[gmbecker]

Currently, summary.factor only shows NA count if there are any NAs:

> summary(iris)
  Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
 Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
 1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
 Median :5.800   Median :3.000   Median :4.350   Median :1.300  
 Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
 3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
 Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
       Species  
 setosa    :50  
 versicolor:50  
 virginica :50  
                
                
                
> iris2 <- iris
> iris2$Species[3] <- "fake"
Warning message:
In `[<-.factor`(`*tmp*`, 3, value = c(1L, 1L, NA, 1L, 1L, 1L, 1L,  :
  invalid factor level, NA generated
> summary(iris2)
  Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
 Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
 1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
 Median :5.800   Median :3.000   Median :4.350   Median :1.300  
 Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
 3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
 Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
       Species  
 setosa    :49  
 versicolor:50  
 virginica :50  
 NA's      : 1  

I wonder if this is the same behavior we want for characters, or if explicitly saying NA's :0 would be useful. My intuition is that missing data is much more common in characters than factors generally.

[gmbecker]

@tinarozsos, @gguyver, and I worked on possible replacements for the current character summary behavior.

Design considerations

There are up to two core behaviors one might want for characters: treating them like pseudo-factors, and ... not doing so. For factors, we suspect that we can leverage summary.factor, though likely with maxsum of 5 or 6 to give space for Mode (and possibly Length).

For the non-factor/factor agnostic case we feel that a combination of the range of nchar(object), a count of unique values, and possibly a count of empty strings and NA's seems quite useful.

Example Implementation

char_summary <- function(object) {
    ncs <- nchar(object)
    unq <- unique(object)
    structure(c(Length = length(object),
    Mode = "character",
    unique.values = length(unq),
    nchar.min = min(ncs, na.rm = TRUE),
    nchar.max = max(ncs, na.rm = TRUE),
    empty.strings = sum(!nzchar(object)),
    NAs = sum(is.na(object))),
    class = "summaryDefault")
}

Some discussions with @bastistician:

• Want argument to control whether characters are treated as factors or not
• Not sure if length should be included, but we feel there may be value to compare length to number of unique values
• Other summary methods only conditionally show NA count, could easily modify this to do the same
• Some skepticism for empty string count, though we feel that it could be useful for free-text style data as empty string/no response data can often be a type of missing but not encoded as such (NA)

Heuristics for detecting factor-like character vectors

Possibility 1:

length(object) > 15 && length(unique(object)) <= log(length(object)) + 3

Very conservative, probably substantially overly so for large vectors

Possibility 2:

length(object) > 15 && length(unique(object)) < length(object) / 3

Reasonable I think, still fairly conservative but unlikely to have many false positives

Possibility 3:

dups <- duplicated(object)
tbl <- table(dups)
unname(tbl["TRUE"]/tbl["FALSE"]) < 2

Odds style instead of ratio but basically equivalent to possibilty 2 for the correct cutoff

[EllaKaye]

Working with @bastistician, we offer the following for discussion as an improvement:

summary.character <- function(
  object,
  ...,
  character.method = c("default", "factor")
) {
  if (is.null(character.method)) {
    return(summary.default(object, ...))
  }

  character.method <- match.arg(character.method)

  if (character.method == "factor") {
    return(summary(factor(object), ...))
  }

  ncs <- nchar(object)
  nna <- sum(is.na(object))

  structure(
    c(
      Mode = "character",
      Length = length(object),
      n.unique = length(unique(object)),
      nchar.min = min(ncs, na.rm = TRUE),
      nchar.med = median(ncs, na.rm = TRUE),
      nchar.max = max(ncs, na.rm = TRUE),
      NAs = if (nna > 0) nna else NULL
    ),
    class = c("summaryDefault", "table")
  )
}

Compared to earlier discussion, we are using character.method as opposed to character.proxy. If character.method is NULL, we return to the old default behaviour (this can be documented but don't need this to be visible in the function signature).

We dropped the "nchar" variant but added nchar.med to the "default" summary.

We have removed the count of empty.strings. We feel it is sufficient to indicate the presence of empty strings, and this is discernible from nchar.min.

We only show a count of NAs if this is > 0. This is for consistency with other summary methods. As a side note, it is grammatically correct for this to be NAs rather than NA's as in other summary methods and we are currently investigating whether it is feasible to correct this elsewhere.

We are not using heuristics to detect factor-like character vectors. The output of summary.character() should be consistent, whatever the values in object. We leave it to the user to decide whether they wish the character vector to be summarised as if it were a factor or not.