Merge branch 'main' into define-resolved-values

Author: eemeli

.github/workflows/validate_tests.yml

@@ -0,0 +1,28 @@
+name: Validate test data
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - test/**
+  pull_request:
+    branches: '**'
+    paths:
+      - test/** 
+
+jobs:
+  run_all:
+    name: Validate tests using schema
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+      - name: Install CLI tool for JSON Schema validation
+        run: npm install --global ajv-cli
+      - name: Validate tests using the latest schema version
+        run: >
+            ajv validate --spec=draft2020
+            -s $(ls -1v schemas/*/*schema.json | tail -1)
+            -d 'tests/**/*.json'
+        working-directory: ./test

LICENSE

@@ -37,3 +37,5 @@ Except as contained in this notice, the name of a copyright holder shall
 not be used in advertising or otherwise to promote the sale, use or other
 dealings in these Data Files or Software without prior written
 authorization of the copyright holder.
+
+SPDX-License-Identifier: Unicode-3.0

README.md

@@ -54,7 +54,7 @@ user interfaces that can appear in any language and support the needs of diverse
 
 The current specification starts [here](spec/README.md) and may have changed since the publication
 of the Tech Preview version.
-The Tech Preview specification is [here](tr35-messageformat.md) (link to follow).
+The Tech Preview specification is [here](https://www.unicode.org/reports/tr35/tr35-72/tr35-messageFormat.html)
 
 The current draft syntax for defining messages can be found in [spec/syntax.md](./spec/syntax.md).
 The syntax is formally described in [ABNF](spec/message.abnf).
@@ -102,16 +102,32 @@ The Working Group continues to address feedback
 and develop portions of the specification not completed for the LDML45 Tech Preview release.
 The `main` branch of this repository contains changes implemented since the technical preview.
 
-Implementers should be aware of the following normative changes during the tech preview period:
-- _(list to be updated during tech preview)_
+Implementers should be aware of the following normative changes during the tech preview period.
+See the [commit history](https://github.com/unicode-org/message-format-wg/commits) 
+after 2024-04-13 for a list of all commits (including non-normative changes).
+- [#771](https://github.com/unicode-org/message-format-wg/issues/771) Remove inappropriate normative statement from errors.md
+- [#767](https://github.com/unicode-org/message-format-wg/issues/767) Add a test schema and
+  [#778](https://github.com/unicode-org/message-format-wg/issues/778) validate tests against it
+- [#775](https://github.com/unicode-org/message-format-wg/issues/775) Add a definition for `variable`
+- [#774](https://github.com/unicode-org/message-format-wg/issues/774) Refactor error types, adding a _Message Function Error_ type (and subtypes)
+- [#769](https://github.com/unicode-org/message-format-wg/issues/769) Add `:test:function`,
+  `:test:select` and `:test:format` functions for implementation testing
+- [#743](https://github.com/unicode-org/message-format-wg/issues/743) Collapse all escape sequence rules into one (affects the ABNF)
+- _more to be added as they are merged_
 
 ## Implementations
 
-(The working group expects that ICU75 will include both Java and C/C++ implementations of the tech preview specification)
-
-- Java: [`com.ibm.icu.message2`](https://unicode-org.github.io/icu-docs/apidoc/dev/icu4j/index.html?com/ibm/icu/message2/package-summary.html), part of ICU 72 released in October 2022, is a _tech preview_ implementation of the MessageFormat 2 syntax, together with a formatting API. See the [ICU User Guide](https://unicode-org.github.io/icu/userguide/format_parse/messages/mf2.html) for examples and a quickstart guide.
+- Java: [`com.ibm.icu.message2`](https://unicode-org.github.io/icu-docs/apidoc/dev/icu4j/index.html?com/ibm/icu/message2/package-summary.html), part of ICU 75, is a _tech preview_ implementation of the MessageFormat 2 syntax, together with a formatting API. See the [ICU User Guide](https://unicode-org.github.io/icu/userguide/format_parse/messages/mf2.html) for examples and a quickstart guide.
+- C/C++: [`icu::message2::MessageFormatter`](https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/classicu_1_1message2_1_1MessageFormatter.html), part of ICU 75, is a _tech preview_ implementation of MessageFormat 2.
 - JavaScript: [`messageformat`](https://github.com/messageformat/messageformat/tree/master/packages/mf2-messageformat) 4.0 implements the MessageFormat 2 syntax, together with a polyfill of the runtime API proposed for ECMA-402.
 
+The working group is also aware of these implementations in progress or released, but has not evaluated them:
+- [i18next](https://www.npmjs.com/package/i18next-mf2) i18nFormat plugin to use mf2 format with i18next, version 0.1.1
+
+> [!NOTE]
+> Tell us about your MessageFormat 2 implementation!
+> Submit a [PR on this page](https://github.com/unicode-org/message-format-wg/edit/main/README.md), file an issue, or send email to have your implementation appear here.
+
 ## Sharing Feedback
 
 Technical Preview Feedback: [file an issue here](https://github.com/unicode-org/message-format-wg/issues/new?labels=Preview-Feedback&projects=&template=tech-preview-feedback.md&title=%5BFEEDBACK%5D+)

docs/tech-preview-blog-post.md

@@ -0,0 +1,128 @@
+# Blog Post for Technical Preview
+
+Today, Unicode announced the Technical Preview of MessageFormat 2,
+a new standard for creating and managing user interface strings.
+These messages can dynamically include data values formatted
+(using information in the Common Locale Data Repository [CLDR])
+according to the needs of the language and culture of the end user.
+Such messages can be adjusted to meet the linguistic needs of each
+language and are designed to be translated easily and efficiently.
+
+Previously, software developers had to choose between many different
+APIs and templating languages to build user interface strings.
+These solutions did not always provide for the features of different
+human languages. Support was limited to specific platforms
+and these formats were not widely supported by translation tools,
+making translation and adaptation to specific cultures costly
+and time consuming.
+Most significantly, message formatting was limited to a small
+number of built-in formats.
+
+One of the challenges in adapting software to work for
+users with different languages and cultures is the need for **_dynamic messages_**.
+Whenever a user interface needs to present data as part of a larger message,
+that data needs to be formatted. 
+In many languages, including English, the message itself needs to be altered
+to make it grammatically correct.
+
+For example, if a message in English might read:
+
+> Your item had **1,023** views on **April 8, 2024**.
+
+The equivalent message in French might read:
+
+> Votre article a eu **1 023** vues le **8 avril 2024**.
+
+Or Japanese:
+
+> あなたのアイテムは **2024 年 4 月 8 日**に **1,023** 回閲覧されました。
+
+But even in English, there are grammatical variations required:
+
+> Your item had _no views_...
+> 
+> Your item had 1 _view_...
+> 
+> Your item had 1,043 _views_...
+
+Once messages have been created, they need to be translated into the various
+languages and adapted for the various cultures around the world.
+Previously, there was no widely adopted standard,
+and existing formats provided only rudimentary support for managing
+the variations needed by other languages.
+Thus, it could be difficult for translators to do their work effectively.
+
+For example, the same message shown above needs a different set of variations
+in order to support Polish:
+
+> Twój przedmiot nie _ma_ żadnych _wyświetleń_.
+> 
+> Twój przedmiot _miał_ 1 _wyświetlenie_.
+> 
+> Twój przedmiot _miał_ 2 _wyświetlenia_.
+> 
+> Twój przedmiot _ma_ 5 _wyświetleń_.
+
+
+MessageFormat 2 makes it easy to write messages like this
+without developers needing to know about such language variation.
+In fact, developers don't need to learn about any of the language
+and formatting variations needed by languages other than their own
+nor write code that manipulates formatting.
+
+MessageFormat 2 messages can be simple strings:
+```
+    Hello, world!
+```
+
+A message can also include _placeholders_ that are replaced by user-provided values:
+```
+    Hello {$user}!
+```
+
+The user-provided values can be transformed or formatted using functions:
+```
+    Today is {$date :date}
+    Today is {$date :datetime weekday=long}.
+```
+
+Messages can use a function (called a _selector_) to choose between
+different versions of a message.
+These allow messages to be tailored to the grammatical (or other) requirements of 
+a given language:
+```
+    .match {$count :integer}
+    0   {{You have no views.}}
+    one {{You have {$count} view.}}
+    *   {{You have {$count} views.}}
+```
+
+Unlike the previous version of MessageFormat, MessageFormat 2 is designed for
+extension by implementers and even end users.
+This means that new functionality can be added to messages without modifying
+either existing messages or, in some cases, even the core library containing the 
+MessageFormat 2 code.
+
+MessageFormat 2 provides a rich and extensible set of functionality
+to permit the creation of natural-sounding, grammatically-correct, 
+messages, while enabling rapid, accurate translation
+and extension using new and improved internationalization functionality
+in any computing system.
+
+The Technical Preview is available for comment.
+The stable version of this specification is expected to be part of the 
+Fall 2024 release of CLDR (v46).
+Implementations are available in ICU4J (Java) and ICU4C (C/C++)
+as well as JavaScript.
+Feedback about implementation experience,
+syntax,
+functionality,
+or other parts of the specification is welcome!
+See the end of this article for details on participation and how to comment on this work.
+
+MessageFormat 2 consists of multiple parts: 
+a syntax, including a formal grammar, for writing messages;
+a data model for representing messages (including those ported from other APIs);
+a registry of required functions;
+a function description mechanism for use by implementations and tools;
+and a test suite.

docs/tools/linkify.js

@@ -0,0 +1,50 @@
+// Work in progress: tooling to linkify the HTML produced from
+// the MessageFormat 2 markdown.
+// this has been tested on the tr35-messageformat.html file
+// but not implemented in LDML45
+function linkify() {
+  const terms = findTerms();
+  const missing = new Set();
+  const links = document.querySelectorAll("em");
+  links.forEach((item) => {
+    const target = generateId(item.textContent);
+    if (terms.has(target)) {
+      const el = item.lastElementChild ?? item;
+      el.innerHTML = `<a href="#${target}">${item.textContent}</a>`;
+    } else {
+      missing.add(target);
+    }
+  });
+  // report missing terms
+  // (leave out sort if you want it in file order)
+  Array.from(missing).sort().forEach((item)=> {
+      console.log(item);
+  });
+}
+
+function findTerms() {
+  const terms = new Set();
+  document.querySelectorAll("dfn").forEach((item) => {
+    // console.log(index + ": " + item.textContent);
+    const term = generateId(item.textContent);
+    // guard against duplicates
+    if (terms.has(term)) {
+        console.log("Duplicate term: " + term);
+    }
+    terms.add(term);
+    item.setAttribute("id", term);
+  });
+  return terms;
+}
+
+function generateId(term) {
+  const id = term.toLowerCase().replaceAll(" ", "-");
+  if (id.endsWith("rategies")) {
+    // found in the bidi isolation strategies
+    return id.slice(0, -3) + "y";
+  } else if (id.endsWith("s") && id !== "status") {
+    // regular English plurals
+    return id.slice(0, -1);
+  }
+  return id;
+}

docs/why_mf_next.md

@@ -7,18 +7,15 @@ why MessageFormat is important and why MessageFormat 2.0 is needed.
 
 ## Intro
 
-The `MessageFormat` API and syntax have been around for a long time.
-
-Intro
+The `MessageFormat` API and syntax have been around for a long time:
 
 - `MessageFormat` is the Unicode API for software localization
-- It is 20 years old, well designed, proven solution
-  Its design was optimized for the software development model
-  of twenty years ago.
-  Implementers, developers, and translators struggle with its shortcomings.
+- It is 20 years old and is a well-designed, proven solution
 
-The current wave of software development uses dynamic languages, modern UI
-frameworks and new forms of user interactions (voice, VR etc.).
+However, its design was optimized for the software development model of twenty
+years ago. Implementers, developers, and translators struggle with its
+shortcomings. The current wave of software development uses dynamic languages,
+modern UI frameworks, and new forms of user interactions (voice, VR etc.).
 
 Considering these new challenges, combined with the lessons learned from using
 `MessageFormat`, we aim to design the next iteration of `MessageFormat`