mb_detect_encoding does not pick first matching candidate

[Lehren]

Description

The following code:

<?php
$encodings = [
            'UTF-8',
            'UTF-16',
            'UTF-32',
            'ISO-8859-1',
            'ISO-8859-2',
            'ISO-8859-6',
            'ISO-8859-15',
            'ASCII'
        ];
        
$a = 'NL';

echo mb_detect_encoding($a, $encodings, true);

Resulted in this output:

'UTF-16'

But I expected this output instead:

'UTF-8'

This worked correctly in 8.0, and was broken from that point onward. Just the string 'NL' is valid UTF-8. It's also valid UTF-16 of course, but UTF-8 is first in the list, so I expect it to be picked first. If the string contained UTF-16 characters, only then would I expect it to return 'UTF-16'

Sandbox: https://onlinephp.io/c/86885

PHP Version

PHP 8.3.12

Operating System

No response

[youkidearitai]

NL is 0x4e4c in ASCII. 4e4c is valid to UTF-16 so U+4E4C is 乌.
mb_detect_encoding since PHP 8.1 also seems to make sense.

I think there are various ways of thinking.

[Lehren]

4e4c is also valid UTF-8 so I don't understand what your point is. I am just dealing with strings that I need to convert to UTF-8 in case it is not UTF-8, what would you have me do?

[youkidearitai]

@Lehren Sorry for my poor English.

@alexdowad I'm sorry, I can't understand behavior of mb_detect_encoding, could you explanation this?

[alexdowad]

@youkidearitai Thanks for commenting on this issue. Your English is fine; don't worry about that.

@Lehren, thanks very much for letting us know about the challenge you are facing. In short, the behavior you have reported is intended and is not a bug. To explain the reason, it would be good to start with the documentation:

Detects the most likely character encoding for [string](https://www.php.net/manual/en/language.types.string.php) string from an ordered list of candidates.

Automatic detection of the intended character encoding can never be entirely reliable; without some additional information, it is similar to decoding an encrypted string without the key. It is always preferable to use an indication of character encoding stored or transmitted with the data, such as a "Content-Type" HTTP header.

This function is most useful with multibyte encodings, where not all sequences of bytes form a valid string. If the input string contains such a sequence, that encoding will be rejected, and the next encoding checked.

In harmony with the docs, while mb_detect_encoding does consider the order of candidate encodings, it primarily attempts to determine what is "the most likely" character encoding for the target string. To estimate the likelihood of any of the candidate encodings being the intended one, it uses several different heuristics, and these heuristics have been adjusted over time to achieve better detection accuracy on realistic input text. Hence, it is not surprising that the results may be different for different PHP versions.

If, as you say, you want to "convert strings to UTF-8 if they are not UTF-8", then you would do best to tell the PHP interpreter to do that:

if (!mb_check_encoding(str, 'UTF-8')) {
    /* Convert `str` to UTF-8; either use mb_detect_encoding to guess its encoding, or check a list of candidates in order using mb_check_encoding */
}

Here is one more piece of advice. Although it depends on the details of your application, it's best for most applications to follow what the documentation says here: "It is always preferable to use an indication of character encoding stored or transmitted with the data, such as a Content-Type HTTP header". Automatic encoding detection is a niche feature, which should not be used in most production code.