Tracking: Internationalization and localization

[kevinramharak]

This issue serves as a central hub for tracking all requests and discussions about support for internationalization and localization (I18n & L10n).

Goals

• Track progress on implementing internationalization and localization.

Reported issues

• Support more languages and locales #27
• After vscode installs the Chinese language pack, the syntax highlighting is incorrect #83
• In Vue and TS file, error no format， after i installed plugin #108
• Not work with Chinese language packs #128

Related PR's

• add support for chinese lang #33
• Supporting korean (and boilerplates for i18n) #49
• Partial support for other languages #110

Notable Fork's

• https://github.com/ShiGongZhi/pretty-ts-errors (Chinese)

Resources

• VSCode API on Localization
• VSCode example repo on Localization
• TypeScript all diagnostic messages in JSON format
• TypeScript localization files for those diagnostic messages
• VSCode Language Packs Repo

[kevinramharak]

After some digging it seems TypeScript uses these scripts to use the diagnostics.json and the locale files to generate two big files, one typescript file and one json file:

• https://github.com/microsoft/TypeScript/blob/d3be7e171bf3149fe93c3ce5a85280f1eba3ef8d/scripts/generateLocalizedDiagnosticMessages.mjs
• https://github.com/microsoft/TypeScript/blob/d3be7e171bf3149fe93c3ce5a85280f1eba3ef8d/src/compiler/utilitiesPublic.ts#L687-L704
• https://github.com/microsoft/TypeScript/blob/d3be7e171bf3149fe93c3ce5a85280f1eba3ef8d/scripts/processDiagnosticMessages.mjs

Those are then used in the code to support diagnostics for each language.
I think we can safely use those files as a reference for how to map our own transforms correctly.
Maybe some (semi) automation to pull those files, and use error codes as indexes will keep it clean and prevent us from duplicating all of their diagnostics.

More reading material if you really like l10n:

• https://datatracker.ietf.org/doc/html/rfc4646
• https://learn.microsoft.com/en-us/windows/win32/intl/locale-names?redirectedfrom=MSDN

[kevinramharak]

I added a start of a possible implementation in the feature/diagnostics-l10n branch.
The README.md explains the proposal.
I wrote a script that uses the generated locale files from typescript, and its source file diagnosticMessages.json to map into a structure like:

// `@pretty-ts-errors/diagnostics-l10n` provides either an export or function to fetch this data structure
declare const diagnosticMessageLocalesMap: DiagnosticMessageLocalesMap;

// using the spanish locale
const spanishLocale = "es";
const errorCode = 2322;
const diagnosticMessageTemplateInSpanish =
  diagnosticMessageLocalesMap[spanishLocale][errorCode];
// -> El tipo '{0}' no se puede asignar al tipo '{1}'.

// with the english locale (also the default locale of the TypeScript compiler)
const englishLocale = "en";
const diagnosticMessageTemplateInEnglish =
  diagnosticMessageLocalesMap[englishLocale][errorCode];
// -> Type '{0}' is not assignable to type '{1}'

This could allow us to split a received diagnostics based on its template, which would also remove the need for a lot of the (dark) regex magic we currently use to parse messages. It might also help us to avoid a lot of the headaches that locale support brings, since we just use whatever the typescript compiler spits out.
We only care about specific parts in the type parts, and those are mostly just plain English ascii.

[kevinramharak]

@yoavbls I have been doing some analysis on the current use of regexes in formatDiagnosticMessages to see what it actually searches for and how it translates to the approach above and concluded this is probably a good direction to go, not just for localization, but for maintainability, performance and easier implementation of other features.

Summary

I have found that some regexes do duplicate work and/or are order dependent, so if the order would be changed, we would get a different outcome. A few have minor redos vulnerabilities (#139 (comment)) and most of them are hard to understand (as mentioned in Tasks - pretty-ts-errors 🎀 (view).

Because we can fetch these diagnostic message templates (as in feature/diagnostics-l10n) we can reduce a lot of runtime complexity to simple ahead of time static analysis. This would improve both performance and maintainability.

This would also make it very easy to implement a more abstract representation of a formatted diagnostic like mentioned in #152 (comment), which makes it easier to render the same formatted message correctly to the hover tooltip, editor panel and menu panel.

This is a pretty different approach to the current implementation so I'm wondering what your thoughts are.

Explanation

A lot of regexes use 'anchors' to locate the types in the diagnostic to be able to format them:

/(module|file|file name|imported via) ['"“](.*?)['"“](?=[\s(.|,]|$)/

With the template based approach, we don't need these regexes as we know exactly where typescript is going to inject the types based on the parameters in the template:

Imported via {0} from file '{1}'

All the transformations are now run in a 'pipeline' on the input string, but with template parts we know ahead of time what parts of the diagnostic is static text and what parts are types originating from the cod of the user. This means we only have to run these relevant transformations on each part based on its type (type or text), instead of running a line of transforms on the same string.

There are also a few regexes that handle very specific error codes:

  jsonKeyValuePart: [1480, 1481, 5101, 5107],
  declareModule: [7035, 7058],
  isMissingTheFollowingPropertiesFromType: [2739, 2740],
  typePair: [2202, 2203, 2204, 2205, 2319, 2320, 2321, 2365, 2367, 2859],
  typeAnnotationMustBeAnyOrUnknown: [1196],

Since we already know their codes, we can keep a map of additional transform functions that are added to the default formatting pipelines, only when the code is relevant.