[FIX] Concatenate: Fix wrong merging of categorical features

[janezd]

Issue

Fixes #4406. Also fixes part of #4382.

The problem was that when merging domains, attributes from the first domain are used and other attributes values' that don't appear in the first table are turned into nan's.

This fix applies only when "primary table" is not present. When it is present, #4422 suffices.

Description of changes

When filtering for unique attributes, two (or more) attributes with different values are replaced by a new attribute.

Includes

• Code changes
• Tests

[codecov]

Codecov Report

Merging #4425 into master will decrease coverage by 4.31%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4425      +/-   ##
==========================================
- Coverage   87.46%   83.15%   -4.32%     
==========================================
  Files         405      268     -137     
  Lines       74135    53962   -20173     
==========================================
- Hits        64841    44870   -19971     
+ Misses       9294     9092     -202

[VesnaT]

This enables creating combinations of self._number_of_decimals, self.adjust_decimals and self._format_str that are forbidden by number_of_decimals setter.

Speaking of self.adjust_decimals. Why do we need that?

[janezd]

You're right. This should go through setter. I changed it to:

    def copy(self, compute_value=None, *, name=None, **kwargs):
        number_of_decimals = kwargs.pop("number_of_decimals", None)
        var = super().copy(compute_value=compute_value, name=name, **kwargs)
        if number_of_decimals is not None:
            var.number_of_decimals = number_of_decimals
        else:
            var._number_of_decimals = self._number_of_decimals
            var.adjust_decimals = self.adjust_decimals
            var.format_str = self._format_str
        return var

adjust_decimals is a flag that tells whether the number of decimals was fixed (e.g. through setter) or is it being adjusted to the largest number of decimals (while reading the file). See function val_from_str_add_cont.

[VesnaT]

I'd rather make self.values a property and check self._value_index - self._value consistency in its setter. It's too easy to obtain inconsistent state with this api.

[janezd]

True. But this commit is from #4422, on which this PR was based. #4422 was already merged with this code.

I agree that values should be a property and it should also be a tuple, not a list. But if I remember correctly, I already tried to change this once but ran into problems. At any rate, it's not in this PR, but if you'd like to try it, I support it.

[VesnaT]

Since this is a public method, I'd add a check if s not in self._value_index, before inserting a new value.

[janezd]

Same as above. I agree, but this is from #4422, which is already merged.

[janezd]

I now implemented this in #4450.

Plus fixing some lint issues for a good measure.

[VesnaT]

Why passing oper parameter? It can be set inside self.merge_domains.

[janezd]

Inertia. :)

[janezd]

@VesnaT, I added a test that tries different orders of adding signals: 98b0e15#diff-e8327cf214d283cae48e36d934820db9R351.

Can you pull the PR and run the test? If it passes: can you modify it to replicate what you're doing in canvas, so that it will fail?