Support for internationalization of help pages

[jeroen]

At rOpenSci there is interest to provide translations for package help pages for some of our packages, initially for Spanish speaking users. Currently R has support for translating messages, but to the best of my knowledge, not for help pages. We are interested to know if prior work has been done in this area, and where it stalled, and if this is something that maybe we can contribute.

Ideally, the package author could write man pages such a way that the user experience would be the same as for messages: when a user does ?topic a manual page in the local language would be opened, and if not available, fall back on the English manual page.

One way would be to allow for e.g. a man-es directory in the source package (in addition to the regular man directory), with Spanish versions of the manuals. Though a drawback may be that this implies a full copy of the manual pages, which might complicated R CMD check. Alternatively maybe the Rd format can be extended to allow for particular multilingual sections.

@eliocamp mentioned:

It'd be interesting to talk to r core about it. We've exchanged emails with Hadley about this some years ago and it lead to this proof of concept: https://github.com/eliocamp/translated
It uses some weird feature of help that allows you to type es?function to get to an .Rd with a function-es alias. I don't think this is the indented use, but it's what we currently have. But before running with it it might be good to talk with r core.

@maelle @yabellini

[mmaechler]

I think we (R-core) would welcome translation efforts for help pages.
However to make this feasible, it would have to be a very modular concept, let's call it "translation module" (to not call it "package"; it may be possible but not necessary to use a kind of well known R package to implement a translation module).
E.g. ((just thinking loud, all to be discussed, but probably not too far from a general R core opinion)):

• R's source would only contain a few "hooks" and functionality say in tools (or in package translations which would become a more regular package with NAMESPACE, R/ and man/).
• For each language there's a separate "translation module" which can be installed separately.
• A Spanish module may be a bundle of separate package specific modules (base, stats,.. MASS, Matrix, .., dplyr,...); so it may make sense of thinking {language} x {R_package} as basic unit.
• CRAN may host the installable translation modules (or not).
• translation modules have maintainers, in 99 to 100% not from R-core.
• Basic R releases are still made independently from and without containing translation modules.
• The R Foundation or also other interested parties may of course provide R+spanish or similar releases (combining download of R plus some translation modules)

[aitap]

Quite a few different ways to implement this on the lower level too.

1. The translated help page is just an Rd document prepared separately, with or without the use of Rd preprocessors. The help system needs just a bit of markup to (1) mark it as a translation of a given help page into a given language and probably (2) note the version of the translation in order to be able to detect outdated translations, and a way to look up the translated help pages, preferably in a similar way to how Gettext looks up translations (e.g. a hypothetical nls-lang=fr_CA translation preferred to nls-lang=fr on a system with fr_CA locale set.)

   Pros:

   • Minimal changes needed to help() and friends

   Cons:

   • A lot of new infrastructure needed to support the translations

2. The translations are Uniforum-style .po files. Before rendering the document, the help system extracts translatable strings from the Rd syntax tree and feeds them to gettext() (probably with a dedicated translation domain, separate from R-level and C-level messages). The splitting algorithm should be automagical enough to avoid making every help page a soup of \gettext{...} macros, but let the author customise the result. Gettext recommends translating full sentences to paragraphs and avoiding "unusual markup", so something will have to be done about cases like an \eqn{} in the middle of a sentence.

   Pros:

   • Lots of existing infrastructure for Gettext-based translations, from Poedit to Weblate
   • An external translation module would only need to provide the *.po/*.mo files

   Cons:

   • More complicated changes to help system
   • The algorithm that makes translation templates out of Rd files needs to be very good. Any change in it may invalidate existing templates and therefore existing translations.

Someone will have to extract the strings Arguments, Author, References, See Also, Value, and so on from tools:::sectionTitles and Rd.sty and make them translatable. Speaking of Rd output formats... Plain text is probably easy because the active locale must be able to represent the user's desired translation of the help page. HTML is also mostly fine because it's just UTF-8, but maybe we should be concerned about right-to-left text direction marks in both cases. For HTML, the lang attribute must be set correctly due to Han unification meaning that many CJK code points are ambiguous and will be rendered differently in the context of different languages. (We'll need someone well-versed in one of these languages to comment further.)

LaTeX is where the real fun starts. While TeX has been developed with internationalisation in mind, Knuth didn't predict Unicode, and so every TeX engine needs slightly different hacks for every language, especially once non-Latin scripts enter the picture. LuaLaTeX and XeLaTeX both support UTF-8 and OpenType fonts, but they don't perform font substitution and need to be told the exact fonts to use; otherwise the code points missing in the default font come out empty. (Perhaps LuaLaTeX could be taught to interface to fontconfig, but I'm not a real TeXnician and haven't been able to find any such scheme that already exists. The exact desired fonts may also be absent on the system where the LaTeX document is eventually compiled.) The R help system currently relies on the pdfTeX engine, and at the very least, Rd.sty will need updating if we are to switch the TeX engine.

CRAN used to host contributed documentation, including a few translations of R's Texinfo manuals (e.g.), but it's frozen now. Presumably, once base R help pages are translated, there will be a need for translated R manuals, too.

[eliocamp]

Just an aside regarding LaTeX and non-Latin scripts. Isn't this a problem even without translations? Right now a package could declare its language to Chinese and have Chinese documentation, couldn't it? How does that documentation render?

[aitap]

I think it doesn't. Rd2pdf defaults to pdflatex, for which you need special packages and to mark the CJK characters in a special way. Even if you switch to a Unicode TeX engine, you still need to set the fonts in the preamble, for which there's currently no API (if you don't count a LaTeX injection "vulnerability" from an undocumented environment variable).

There's an old API that enables Cyrillic characters if a special environment variable is set, but there's no API for a package to request special environment variables for its help pages. (\Sexpr[stage=render] will affect the whole R process, which I guess is relatively fine for a non-CRAN package.)

[eliocamp]

Oh, OK. Maybe this needs to be looked into independently of internalisation or not, since it sounds to be incompatible with today's policies of allowing non-English documentation without explicitly limiting non-Latin scripts.

[eliocamp]

I think having translation pages as extra .Rd files is the best idea. R would only need to handle finding the correct help page and linking the different translations. Then how these pages are translated and maintained would be up to the package maintainers.

[mmaechler]

yes. Still, at the sprint one could start thinking about which parts need to be added to base R itself ... after having decided about the principle translation-module format (see my first remark).