Compact printing in cut

[bkamins]

Currently we have:

julia> cut(1:10, 3)
10-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 4.0)"
 "Q1: [1.0, 4.0)"
 "Q1: [1.0, 4.0)"
 "Q2: [4.0, 6.999999999999999)"
 "Q2: [4.0, 6.999999999999999)"
 "Q2: [4.0, 6.999999999999999)"
 "Q3: [6.999999999999999, 10.0]"
 "Q3: [6.999999999999999, 10.0]"
 "Q3: [6.999999999999999, 10.0]"
 "Q3: [6.999999999999999, 10.0]"

which is not nice. It would be better to use compact printing.

Though we should make sure to correctly do this case:

julia> cut(1:10^-12:1+10^-11, 3)
11-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 1.0000000000033333)"
 "Q1: [1.0, 1.0000000000033333)"
 "Q1: [1.0, 1.0000000000033333)"
 "Q1: [1.0, 1.0000000000033333)"
 "Q2: [1.0000000000033333, 1.0000000000066667)"
 "Q2: [1.0000000000033333, 1.0000000000066667)"
 "Q2: [1.0000000000033333, 1.0000000000066667)"
 "Q3: [1.0000000000066667, 1.00000000001]"
 "Q3: [1.0000000000066667, 1.00000000001]"
 "Q3: [1.0000000000066667, 1.00000000001]"
 "Q3: [1.0000000000066667, 1.00000000001]"

[andreasnoack]

I just hit a similar case hwere the many digits made plotting labels look ugly. What about supporting rounding in the cut(array, ngroups) method instead of just changing the printing? I guess it might be confusing if the string here don't reflect the actual cuts.

[nalimilan]

This is tricky. Rounding could easily change radically the size of the groups if values are very close. AFAIK other packages don't do that. What R's cut does is that it tries with 3 digits by default, and if some breaks end up represented the same it increases the number of digits up to 12. This can be tweaked via the dig.lab argument. This sounds reasonable to me at least for quantiles, as you probably don't care about the exact value in that case. OTOH it could be more surprising when specifying breaks manually.

At the very least we could add that dig.lab argument.

[andreasnoack]

Rounding could easily change radically the size of the groups if values are very close.

I don't see the issue if it's just an option. Then the user can decide if the rounding is acceptable.

[nalimilan]

If you want quantiles and rouding ends up creating classes with different sizes, you no longer get quantiles. ;-) So the printing "Q1: ..." would be misleading. Rounding only the display seems less dramatic. Though of course we could support both as options.

[andreasnoack]

The Qx label doesn't specify the probability of the quantile anyway. Actually, I'd prefer a label without Qx part. I'm mostly interested in the interval information. It is easy to incorrectly think that Q1 is the first quartile.

[nalimilan]

I added Qx because when allowempty=true some intervals may be identical (in the presence of many duplicates), but we can't have levels with equal names. But that's a corner case so we could stop doing that by default. Still, returning quantiles that are not real quantiles due to rounding would be confusing IMO.

[nalimilan]

This is related to a discussion we had in 2020 at #245. @bkamins Do you have an opinion? I see several options:

• Keep adding Qx.
• Provide a formatter that adds Qx and advise using it when we throw an error because of duplicate intervals.
• Support an argument doing that.
• Automatically use that formatter by default when allowempty=true is passed.

[bkamins]

Given the discussion - I think adding an argument is most flexible.

[nalimilan]

Can you elaborate why you prefer an argument over a formatter? I find it hard to decide which is best.

[bkamins]

Ah - now I understand an issue. I understand that you want to predefine a function that you would pass to labels and it would provide a different way of handling this. I think this is OK. For me the key thing is to give some predefined to a user (i.e. do not just say that user could do this themselves by writing a proper formatte function)

[andreasnoack]

After thinking a bit further, I see two separate but somewhat related issues here.

1. Which cuts to use
2. Which labels to attach

This issue is of course about 2. but now I think that 1. is actually what makes 2. an bigger issue. At least to me. In my use cases, I'd prefer cuts at the values in the input vector, i.e. without linear interpolation. So R-1 in https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample. Typically, real data doesn't have the many decimals so using the R-1 definition would avoid very long representations of the cuts and therefore 2. is no longer as relevant (to me). Is that something you'd be willing to consider here?

I added Qx because when allowempty=true some intervals may be identical (in the presence of many duplicates), but we can't have levels with equal names.

I guess you have thought this through many years ago but having non-unique intervals seems unfortunate to me. I can see that something would have to go, though, but I think I'd prefer to change the definition such that cut produces at most n cuts.

Btw. found JuliaStats/Statistics.jl#184 when playing around with this issue.

[nalimilan]

This issue is of course about 2. but now I think that 1. is actually what makes 2. an bigger issue. At least to me. In my use cases, I'd prefer cuts at the values in the input vector, i.e. without linear interpolation. So R-1 in https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample. Typically, real data doesn't have the many decimals so using the R-1 definition would avoid very long representations of the cuts and therefore 2. is no longer as relevant (to me). Is that something you'd be willing to consider here?

That's an interesting idea. On the one hand we probably don't care a lot about the exact cutpoints as long as the resulting quantiles are identical, and using those from actual data makes sense. On the other one, getting results slightly different from quantile (which doesn't even support R-1 currently) is a bit annoying. And it wouldn't completely solve the problem, as even simple real data can contain many decimals (log income, average income per household member...), so we would need a solution for compact printing anyway. So I'm hesitant...

In terms of implementation, the cleanest solution would be to change quantile to support R-1 first, but it won't be available on old Julia versions that we want to support (like 1.6) to avoid creating a big pre-1.0 vs post-1.0 CategoricalArrays divide. Instead we would have to take the sorted array returned by quantile and find the data points corresponding to quantiles.

I guess you have thought this through many years ago but having non-unique intervals seems unfortunate to me. I can see that something would have to go, though, but I think I'd prefer to change the definition such that cut produces at most n cuts.

This is opt-in so we can offer several options. The idea behind the current allowempty=true behavior is that sometimes you really need to get the expected number of groups so that e.g. your tables and models are consistent across countries (this is something that happened to me in practice). I guess in other cases you'd like as many groups as possible up to ngroups so we could also support that (it's easy). Either we can add an argument like allowfewer=true, or maybe better use a single argument like duplicatebreaks=:error/:empty/:skip?

[andreasnoack]

And it wouldn't completely solve the problem, as even simple real data can contain many decimals (log income, average income per household member...), so we would need a solution for compact printing anyway. So I'm hesitant...

That is true but my guess is that it would be sufficient in many cases and it would have the advantage that the user could manually round that quantity before passing it to cut if needed.

In terms of implementation, the cleanest solution would be to change quantile to support R-1 first, but it won't be available on old Julia versions that we want to support (like 1.6) to avoid creating a big pre-1.0 vs post-1.0 CategoricalArrays divide.

I agree that it would be nice to have in quantile. The actual algorithm would be very easy to implement. I guess the main challenge would figure out the interface since the current interface is based on alpha and beta. Maybe we could have a separate method/definition keyword where the user is only allowed to specify either definition or alpha/beta. I will open an issue if Statistics to discuss this.