[clang] Warning for implicit conversion and operations between different `charN_t`

[eisenwave]

Consider the following code:

bool contains_oe(std::u8string_view str) {
    for (char8_t c : str)
        if (c == U'ö') // comparison always fails, but no diagnostic catches this
            return true;
    return false;
}

If str is a correctly encoded UTF-8 string, the comparison always fails because no UTF-8 code unit can be 0x6F, and ö is U+00F6. Comparing charN_t with different N is virtually always a bug, or could have just as well been written using a different type of literal. Comparing these types is not going to give meaningful results except for U+007F and below, and even then, it's unclear why you wouldn't use the proper type.

I've floated the idea of deprecating this behavior in the C++ standard in a number of places, and it was received positively. StackOverflow users also suggested getting rid of it here: https://stackoverflow.com/q/79604433/5740428

In the meantime, it would be useful to have a warning when charN_t is converted to a different Unicode character type. This warning should be triggered for any implicit conversion, not just as part of a comparison because the same bug can be produced like:

bool contains_char(std::u8string_view str, char8_t c);
// ...
contains_char(U'ö');

[frederick-vs-ja]

There is an abandoned WG21 paper P1880 about this, see cplusplus/papers#630.

Also, per P2029R4, it seems intended to allow uN stuffs to contain invalid code units.

[frederick-vs-ja]

Perhaps we want something like [[vendor::assume_utf]] (or even a standardized [[assume_utf]]) as [[assume]] doesn't seem suitable for being specialized for UTF.

[eisenwave]

@frederick-vs-ja Knowing that a string is properly UTF-8 encoded would have caught this bug (with the right warnings) but it doesn't make char8_t and char32_t comparison meaningful in general. I don't think it's the easiest and most reliable direction to go in.

If you did the same search as in the example but searched for U'\N{NON-BREAKING SPACE}' (U+00A0), you would get no warning with [[assume_utf]] because a UTF-8 string can contain a code unit whose value is 0xa0. The comparison is not tautologically false in this case, but it yields (only) false positives, such as Ԡ (U+0520), which is UTF-8 encoded as 0xD4 0xA0

The underlying problem is that char8_t == char32_t is car == banana.

[frederick-vs-ja]

The underlying problem is that char8_t == char32_t is car == banana.

Partially agreed, but AFAIK that there're many uses of comparison between char and wchar_t, while they almost always represent different encodings. Perhaps it would be better to consider char and wchar_t together.

Resolved

Also, the title seems to be incomplete, as it seems desired to warn on both comparison and conversion, but the comparison between different charN_t types doesn't convert them to the wider one.