Add support for gettext plural forms

[Calinou]

Following discussion in #31423, we should eventually implement support for pluralization as provided by gettext. Many languages have more than 2 plural forms, which means the plural forms cannot be hardcoded. Instead, the localization file should be able to define as many plural forms as the language requires.

The main focus would be to make it available in the editor, but making it usable in projects would be welcome as well. For projects, this would likely require using PO files for localization, as CSV is too limited for this.

[remram44]

Is there a particular reason a library such as gettext can't be used directly, instead of using a custom PO file parser?

That would add:

• Less code to maintain here
• Support for MO files (compiled binary message catalogs)
• Support for plurals and context, which is not there yet

Is the problem simply a build concern, it's hard to link it statically on some platforms?

[remram44]

Looking around, this would require multiple changes:

• Update the PO parser to read the plural forms from config (it's a conditional expression that takes n and evaluate to an integer, the index of the form to use, for example (Polish) plural=(n==1 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2);)
• Update the PO parser to read multiple msgstr variants for each string
• Update the Translation class to store the multiple forms (and change Translation::add_message())
• Update the API, Object.tr() should take a number as argument (and for proper xgettext support, it should probably take two forms for the msgid). Those arguments can be added to tr() with proper defaults or it can be a new function (tr_plural/trs/tr_n)

[Calinou]

Is there a particular reason a library such as gettext can't be used directly, instead of using a custom PO file parser?

I believe licensing would be an issue, as gettext libraries appear to be LGPL-licensed, which we can't integrate in Godot. (The reason is that we'd like to link binaries statically without restrictions, especially on iOS where static linking is the only option.)

[akien-mga]

Indeed, LGPL is problematic since we'd have to provide a way for users to change the version of the LGPL'ed work even in the statically compiled version, which is far from trivial.

We used to have a library under LGPLv2.1 (libwebsockets), but it had a static linking exception. We're also better off without its legalese now.

Note: great resource on the LGPLv2.1: https://copyleft.org/guide/comprehensive-gpl-guidech11.html
We bundled LGPL'ed code for cross platform convenience would make things even tricker as we may be seen as a "work based on the library".

[blockspacer]

"provide a way for users to change the version of the LGPL'ed work even in the statically compiled version," - What is version of the LGPL'ed work?

What do you think about linking gettext as shared lib in community provided module? Anyone tested that approach before?

[blockspacer]

BTW, that do you think about gettext-compatible library, under BSD license?

http://wiki.netbsd.org/projects/project/libintl/
http://ftpmirror.your.org/pub/NetBSD/NetBSD-release-8/src/lib/libintl/

attempt at modernizing and updating NetBSD's BSD-licensed libintl
implementation:
https://gitlab.com/worr/libintl

[Calinou]

What do you think about linking gettext as shared lib in community provided module? Anyone tested that approach before?

Nothing prevents you from doing this, but we don't want to do it officially 🙂

Keep in mind some platforms such as iOS don't support dynamic linking at all.

BTW, that do you think about gettext-compatible library, under BSD license?

http://wiki.netbsd.org/projects/project/libintl

I didn't know there were permissively-licensed, feature-complete libraries for handling PO files. Someone needs to give it a try, I guess.

[blockspacer]

BTW I can see gettext switched to LGPLv2.1, with static linking exception . Anyone can confirm it?

Based on comment from code https://github.com/autotools-mirror/gettext/blob/3c82d1a522c0b4e79168ff479d90eeadc109d9f6/gettext-runtime/intl/xsize.h

"GNU Lesser General Public License as published by
the Free Software Foundation; either version 2.1, or (at your option)
any later version."

UPD:
it gives the right to use libintl as a static library or to incorporate libintl into another library only to free software.
https://www.gnu.org/software/gettext/manual/html_node/Discussions.html

[Calinou]

@blockspacer I don't see a static linking exception in that license notice.

[blockspacer]

What do you think about ICU Message syntax
https://phrase.com/blog/posts/guide-to-the-icu-message-format/#CC
https://support.crowdin.com/icu-message-syntax/#plural
https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/classMessageFormat.html

See icu::Formattable, icu::PluralRules and base/i18n from chromium (BSD license)

https://github.com/chromium/chromium/blob/2ca8c5037021c9d2ecc00b787d58a31ed8fc8bcb/base/i18n/message_formatter.cc#L64

https://github.com/chromium/chromium/blob/2ca8c5037021c9d2ecc00b787d58a31ed8fc8bcb/ui/base/l10n/formatter.cc#L166

https://codereview.chromium.org/2736543002/

http://userguide.icu-project.org/formatparse/messages

[blockspacer]

BTW tinygettext has expression parser for plural forms, zlib license.
tinygettext/tinygettext#23
https://github.com/tinygettext/tinygettext/blob/master/src/plural_forms.cpp