[ENH] Impute: Allow setting a default value for all numeric and time variables

[janezd]

Issue

Fixes #5065.

Includes

• Code changes
• Tests
• Documentation

[codecov]

Codecov Report

Merging #5102 (8286474) into master (d04e3cd) will increase coverage by 0.00%.
The diff coverage is 97.29%.

@@           Coverage Diff           @@
##           master    #5102   +/-   ##
=======================================
  Coverage   84.77%   84.77%           
=======================================
  Files         286      286           
  Lines       60074    60130   +56     
=======================================
+ Hits        50927    50977   +50     
- Misses       9147     9153    +6     

[ajdapretnar]

Try entering a value with a decimal comma. It breaks the widget.

[lanzagar]

A new class was introduced below with name = "Fixed value". Was this one intentionally changed to the same name too (short names remain different)?

[janezd]

Yes. It's more informative, looks better in the widget.

[lanzagar]

self.default_numeric is a string by default, but this changes it to a float. This is inconsistent and probably helped with the below bug on line 308 (and others?).

[janezd]

I initially planned to have a float, but then decided for str and forgot to remove this argument.

[lanzagar]

self.default_numeric can be 0 which evaluates to False so np.nan is used instead!

[janezd]

Oops. Fixed. Thanks.

[janezd]

Not imputing anything: it works for me. Which data did you try and how? Could it be that you tried to impute 0 (the bug discovered by @lanzagar) or you used a wrong delimiter (see next point)?

Decimal comma: I now changed the widget to use the current locale. Select Rows does the same. But this is inconsistent: I now need to enter 42,0 and the Table widget shows 42.0. Have we been discussing this before? We use . as decimal separator when displaying floats, so we should probably also fix US locale?

Discrete values: they should come from a combo, I guess. Which values would it contain? For, say, heart disease data? Also, this should then be a context setting that would depend upon whether the selected value is present or not. Using a new value for all discrete variables is also problematic. What if this particular value name already exist for some variables? I found this too messy.

[ajdapretnar]

Could it be that you tried to impute 0 (the bug discovered by @lanzagar).

You mean a bug discovered by me and reported by Mr @lanzagar? :D Yes, it was the case of 0.

Discrete values: this is why I deleted a comment, because I realized it is too difficult to implement. So, my use case is usually having N discrete variables with values 0, 1. Some values are missing (sometimes there are just 1, no 0). And I want an easy way to impute in Orange, just fill all the missing values with 0. This is why I asked about discrete and this is why I tested with 0. I was told it is difficult to provide a reasonable solution for discrete.

[ajdapretnar]

This data set still breaks impute.
missing1.xlsx

Well, I should probably say it breaks Data Table badly and other widgets less badly. Seems like TimeVariable is not formatted properly?

[janezd]

I believe this to be fixed now. (It worked at some point, but now it was broken for any file. Shame on me - I apparently changed something and haven't tested it again.)

It can be merged now, but has to wait for #5117 (fix for xlsx), because it' s based on it.

[ajdapretnar]

I managed to crash something while testing.

---------------------------- IndexError Exception -----------------------------
Traceback (most recent call last):
  File "/Users/ajda/orange/orange3/Orange/widgets/data/owtable.py", line 83, in data
    return super().data(index, role)
  File "/Users/ajda/orange/orange3/Orange/widgets/utils/itemmodels.py", line 978, in data
    return coldesc.format(instance)
  File "/Users/ajda/orange/orange3/Orange/widgets/utils/itemmodels.py", line 828, in format_dense
    return str(instance[var])
  File "/Users/ajda/orange/orange3/Orange/data/variable.py", line 176, in __str__
    return self.variable.str_val(self)
  File "/Users/ajda/orange/orange3/Orange/data/variable.py", line 813, in repr_val
    return '{}'.format(self.values[int(val)])
IndexError: tuple index out of range

But for the life of me I can't reproduce now, so...