[RFC] Add a locale for grapheme case-insensitive functions

[youkidearitai]

Add $locale parameter for grapheme case insensitive functions.
RFC: https://wiki.php.net/rfc/grapheme_add_locale_for_case_insensitive

[heiglandreas]

I have a probably stupid question but I try to understand the use-case:

Do you have an example where the locale is relevant to determine uppercase or lowercase characters?

From what I understand for example the lower-case letter å has a corresponding upper-case letter Å - regardless of the locale.

So what do we need a locale for if we want to find år in Århus?

Or is the idea to also find ar in Århus?

If the later, then IMO that is faaaaar more than "just" adding a locale to grapheme_functions.

[youkidearitai]

@heiglandreas
Thank you very much for response.

From what I understand for example the lower-case letter å has a corresponding upper-case letter Å - regardless of the locale.

Yes, right.

So what do we need a locale for if we want to find år in Århus?
Or is the idea to also find ar in Århus?

Yes, later it is. I think below:

$ sapi/cli/php -r 'var_dump(grapheme_stripos("Aarhus", "å", locale: "da"));'
int(0)
$ sapi/cli/php -r 'var_dump(grapheme_stripos("Aarhus", "å"));'
bool(false)

Thanks for give to me to example.

From https://unicode-org.github.io/icu/userguide/transforms/casemappings.html#full-language-specific-case-mapping , Turkish I of small letter is ‘İ’(`U+0130). This RFC is supports these case.

[youkidearitai]

It's midnight. I'll fix this morning. thanks.

[heiglandreas]

I find this highly problematic as it completely changes the way the i comparison functions work.

This is not to say that it isn't a valuable addition! But I think this should be a separate (set of) functions.

So far the i functions compare on a "binary" level ignoring the 3rd bit. So a (01100001) and A (01000001) are the same.

This is what most people expect when they pass a string to an i function.

This change here though not only compares without considering the case, it also does some character replacements based on the locale.

So it is basically doing the following under the hood:

$string = 'Aarhus';
$comparison = 'år';
$normalizedString = Transliterator::create('dk-lower')->transliterate($string);
$normalizedComparison = Transliterator::create('dk-lower')->transliterate($comparison);
grapheme_strpos($normalizedString, $normalizedComparison);

Where all I would expect is

grapheme_strpos(
	mb_strtolower('Aarhus'),
	mb_strtolower('år')
);

To give you an example why that might become really messy:

In german we have two distinct different words: Busse (plural of a bus) and Buße (penance).

When comparing them with a german locale they would not be identical. Fine.

But when comparing them with a for example US-locale they would suddenly be the same. Why? Because in languages that do not have the ß character this is often converted to ss. Which then gives a totally new meaning to the word.

By nonchalantly doing more than a case-insensitive comparison suddenly bus becomes part of Buße...

Or - for the Octoberfest fans: A Maß suddenly is included in Masse.

This is a hard no from my side!

[youkidearitai]

@heiglandreas
Thank you for your comment.
I understand just a little your problem.

I saw your example code, so I understand this PR is many problem.

But when comparing them with a for example US-locale they would suddenly be the same. Why? Because in languages that do not have the ß character this is often converted to ss. Which then gives a totally new meaning to the word.

I'm going to trying to ICU depends, However, this means many problems.
Because the meaning changes.

Your information is valuable.
Thank you very much. thanks again.

This RFC moved back to "Under Discussion".
I'm not going to force it.

[youkidearitai]

Dear @heiglandreas (or anyone)
Please give me more information.

Are you satisfied with the current behavior of grapheme_stripos(not locale insensitive and case-insensitive grapheme functions?

I think need to locale from your comment. Because changes to behavior of regions.

[heiglandreas]

I do see the issue with especially the turkish alphabet where the Capital letter "I" (U+0049) is associated with the lower case letter "ı" (U+0131) and the Capital letter "İ" (U+0131) is associated with the lower case letter "i" (U+0069) - so essentially matching the "wrong" letters to one another.
Which is not only a problem with turkish but also some other languages from the area like Azerbaijani, Crimean Tatar, Gagauz, Kazakh and Tatar.

TBH I am not sure this is something that can be handled by a standard locale as "en-EN" as that is region-specific and does - in it's general form - not say anything about the used alphabet.

So for example "sr-RS" (Serbian as spoken in Serbia) does not say whether to use the cyrilic or the latin alphabet. One would need to be explicit and use "sr-Latn-RS" to also include the alphabet. Similarily would you need to explicitly specify whether to use the Latin, the Arabic or the Cyrilic alphabet in Azerbaijani like via "az-Latn-AZ" or "az-Cyr-AZ".

The best option IMO with the current tools would be something like this:

$t = Transliterator::create('tr-Lower');

var_dump(grapheme_strpos(
	$t->transliterate('İ'),
	$t->transliterate('i')
));

But this uses an ID from the ICUs Transliterator which has only very little to do with a locale.

So from my point of view a new set of functions (names something like grapheme_strt* for Grapheme-function (grapheme_) for strings (str) based on transliteration (t)) would make sense to not overload the current functions with a slightly different functionality.

So in this case a grapheme_strtipos('İ', 'i', 'tr-Lower', 0) would return 0

Sidenote: mb_strtolower returns i̇ as lower-case version of İ so

grapheme_stripos('İ', 'i̇');

will output 0...

[youkidearitai]

@heiglandreas
Thank you. I see.
That means override to grapheme_stri* is not make sense. So we are going to new functions (grapheme_strt*), right?
I'm interested to that idea.

Just a moment, please.

[youkidearitai]

I add the strength for grapheme functions. Also affect to Ideographic characters. So that means affect to CJK characters(Japanese says Kanji(漢字)).
I feel getting serious matter...

[youkidearitai]

This RFC is accepted. So ready for review.

[youkidearitai]

@php/release-managers-85 Is this need your approvals?

[edorian]

Thank you for the ping. RM wise, no objections. Leaving the implemention review to others.

Feels worth it to add a single line into NEWS as well before merging.

[youkidearitai]

@edorian Thank you your approved. I added this news to NEWS file.
I'll waiting to half day that before merge. @devnexen @derickr Any objections? (sorry for if your time is midnight)

[devnexen]

No objections from me I may have a better look tomorrow.

[edorian]

@youkidearitai please note that my review is not technical, just RM process. So getting a technical approval is also required :)

[youkidearitai]

@devnexen Thank you! I'll waiting few days.
@edorian Thank you for approve. I got it.

[devnexen]

little inconsistency here, maybe locale_len = 0;

[youkidearitai]

Thanks! Added.

[devnexen]

could you add more tests with invalid locales please ?

[youkidearitai]

Sure! Added.

[devnexen]

fine by me. let s wait another approval before merging though.