Confusables - does it reverse punycode?

[johnhtodd]

The "confusables" section - does it reverse punycode? All the strings in our case that are fed in have been punycoded. I am not certain if the code reverses that punycode action before performing a confusables match. If not, it needs to do that on strings bounded by "." and starting with "xn--"

If it already does that: great!

Also: there are two classes of confusables: punycode and non-punycode confusables. Catching both types of confusables may necessitate two runs. Example: netf1ix.com is a non-punycode confusable.

[esensar]

I don't think it does, but should that punycode reversal be a preprocesaor step maybe? I believe punycode would break most of the other rules.

[johnhtodd]

I'm not sure about a preprocessor in all cases. I think, for example, it would not be useful on NYIIS or other phonetic models, would it?

The more I think about it, the more relevant it is in some of the typographic filters. Levenshtein, for instance, would benefit from seeing the text in non-punycode format.

So... do we add a flag in each filter that allows the filter to be run twice, once with punycode (if it is indeed a string that can be un-punycoded) and once without? And then Yet Another JSON flag that says if the match was against the punycode or un-punycoded?

Not sure how to represent this easily.

[esensar]

I'm not sure about a preprocessor in all cases. I think, for example, it would not be useful on NYIIS or other phonetic models, would it?

Right, but these wouldn't be useful in either case, right? Both punycode and non-punycode non-ASCII strings would never match phonetic models (I think).

The more I think about it, the more relevant it is in some of the typographic filters. Levenshtein, for instance, would benefit from seeing the text in non-punycode format.

So... do we add a flag in each filter that allows the filter to be run twice, once with punycode (if it is indeed a string that can be un-punycoded) and once without? And then Yet Another JSON flag that says if the match was against the punycode or un-punycoded?

I am thinking that we can have that filter in both directions (to and from punycode) and both of these could optionally pass both values further into the ruleset. Split target filter adds some metadata to the JSON to tell which position it was in, so we can have these add that punycode metadata too.

Seems doable to me, I will know if I missed something once I start implementing 😄