Add contextual options to expression-attributes design doc (#780)

* Add design doc for u:options

* Apply changes to expression attributes design doc

* Update exploration/u-options.md

Co-authored-by: Eemeli Aro <[email protected]>

* Apply suggestions from code review

Co-authored-by: Addison Phillips <[email protected]>

* Rename exploration/u-options.md -> exploration/contextual-options.md

* Refactor as a change of the expression-attributes proposal

* Apply suggestions from code review

Co-authored-by: Addison Phillips <[email protected]>

* Apply suggestions from code review

Co-authored-by: Tim Chevalier <[email protected]>

* Apply suggestions from code review

Co-authored-by: Addison Phillips <[email protected]>

* Add text explaining markup identifier uses

---------

Co-authored-by: Addison Phillips <[email protected]>
Co-authored-by: Tim Chevalier <[email protected]>

Author: 3 people

exploration/expression-attributes.md

@@ -13,6 +13,7 @@ Status: **Proposed**
 		<dt>Pull Requests</dt>
 		<dd><a href="https://github.com/unicode-org/message-format-wg/pull/458">#458</a></dd>
 		<dd><a href="https://github.com/unicode-org/message-format-wg/pull/772">#772</a></dd>
+		<dd><a href="https://github.com/unicode-org/message-format-wg/pull/780">#780</a></dd>
 		<dd><a href="https://github.com/unicode-org/message-format-wg/pull/792">#792</a></dd>
 	</dl>
 </details>
@@ -27,17 +28,33 @@ Function options may influence the resolution, selection, and formatting of anno
 These provide a great solution for options like `minFractionDigits`, `dateStyle`,
 or other similar factors that influence the formatted result.
 
-However, this single bag of options is not appropriate in all cases,
-in particular for attributes that pertain to the expression as a selector or a placeholder.
-For example, many of the [XLIFF 2 inline element] attributes don't really make sense as function options.
+Such options naturally correspond to function arguments or builder-style function constructors.
+Each option is specific to the associated API.
+Message authors, such as translators or developers, want consistent ways to do common tasks,
+such as providing hints to translation tools or overriding the locale,
+but, unless MessageFormat provides otherwise, cannot rely on implementations
+to consistently implement these.
+
+To reduce the learning curve for users and improve consistency,
+it would be useful to have common options
+(generally those related to the formatting context)
+shared between all functions.
+
+Separately from formatting concerns,
+it is often useful to attach other information to message expressions and markup.
+For example, presenting how an example value could be formatted can be very useful for the message's translation.
+Providing the original source representation of a placeholder may be essential for being able to format a non-MF2 message,
+if it has been transformed to MF2 to provide translators with a unified experience.
+As a specific example, many of the [XLIFF 2 inline element] attributes have no meaning
+to the function that they appear as options or annotations of.
 
 [XLIFF 2 inline element]: http://docs.oasis-open.org/xliff/xliff-core/v2.1/os/xliff-core-v2.1-os.html#inlineelements
 
 ## Use-Cases
 
 ### User Story: Formatting Context Override
 As a message author, I want to override values in the _formatting context_ for a specific _expression_.
-I would like to do this in a consistent, effective manner that does not require a change to the 
+I would like to do this in a consistent, effective manner that does not require a change to the
 _function_ or _markup_ support code in order to be effective.
 As far as the code is concerned, it just reads the value from the _formatting context_ normally.
 
@@ -82,11 +99,11 @@ Some examples include:
   These may include an example, which is best represented in MF2 as an `@example=...` attribute.
 
 - In #772, @eemeli calls out:
-  > While working on [moz.l10n](https://github.com/mozilla/moz-l10n/), 
-  > a new Python localization library that uses the MF2 message and 
-  > [resource data model](https://github.com/eemeli/message-resource-wg/pull/16) to represent messages 
-  > from a number of different current syntaxes, 
-  
+  > While working on [moz.l10n](https://github.com/mozilla/moz-l10n/),
+  > a new Python localization library that uses the MF2 message and
+  > [resource data model](https://github.com/eemeli/message-resource-wg/pull/16) to represent messages
+  > from a number of different current syntaxes,
+
   Apple's Xcode supports localization of plural messages via `.stringsdict` XML files,
   which encode the plural variable's name as a `NSStringLocalizedFormatKey` value,
   where it appears as e.g. `%#@countOfFoo@` or similar.
@@ -98,31 +115,31 @@ At least the following expression attributes should be considered:
 
 - Attributes with a formatting runtime impact:
 
-  - `id` — An identifier for the expression.
+  - `id` — An identifier for the expression or markup.
     This is included in the formatted part,
-    and allows the parts of an expression to be explicitly addressed.
+    and allows each part of a message to be explicitly addressed.
 
     > Example identifying two literal numbers:
     >
     > ```
-    > The first number was {1234 :number @id=first} and the second {56789 :number @id=second}.
+    > The first number was {1234 :number u:id=first} and the second {56789 :number u:id=second}.
     > ```
 
   - `locale` — An override for the locale used to format the expression.
-    Should be expressed as a non-empty sequence of BCP 47 language codes.
 
-    > Example embedding a French literal in an English message:
+    > Example embedding a French date in an English message:
     >
     > ```
-    > In French, “{|bonjour| @locale=fr}” is a greeting
+    > In French, this date would be displayed as {|2024-05-06| :date u:locale=fr}
     > ```
 
   - `dir` — An override for the LTR/RTL/auto directionality of the expression.
 
-    > Example explicitly isolating the directionality of a placeholder:
+    > Example explicitly isolating the directionality of a placeholder
+    > for a custom user-defined function:
     >
     > ```
-    > Welcome, {$username @dir=auto}
+    > Welcome, {$user :x:username u:dir=auto}
     > ```
 
 - Attributes relevant for translators, tools, and other message operations,
@@ -149,10 +166,10 @@ At least the following expression attributes should be considered:
     whether the expression should or should not be localised.
     The values here correspond to those used for this property in HTML and elsewhere.
 
-    > Example embedding a non-translatable French literal in an English message:
+    > Example embedding a non-translatable CLI command in a message:
     >
     > ```
-    > In French, "{|bonjour| @locale=fr @translate=no}" is a greeting
+    > Use {+code @translate=no}git ls-files{-code} to list all files in a repository.
     > ```
 
   - `canCopy`, `canDelete`, `canOverlap`, `canReorder`, etc. — Flags supported by
@@ -176,20 +193,16 @@ including expressions without an annotation.
 
 Attributes are distinct from function options.
 
-Common attributes are defined by the MF2 specification
-and must be supported by all implementations.
+Common options or attributes should work the same way in different functions.
+
+Special options or attributes should not conflict with other option names.
 
 Users may define their own attributes.
 
 Implementations may define their own attributes.
 
-Some attributes may have an effect on the formatting of an expression.
-These cannot be defined within comments either within or outside a message.
-
 Each attribute relates to a specific expression.
 
-An attribute's scope is limited to the expression to which it relates.
-
 Multiple attributes should be assignable to a single expression.
 
 Attributes should be assignable to all expressions, not just placeholders.
@@ -204,38 +217,132 @@ the reserved/private-use rules will need to be adjusted to support attributes.
 
 ## Proposed Design
 
-Add support for option-like `@key=value` attribute pairs at the end of any expression.
+Provide separate solutions for attributes that impact formatting,
+and those which do not,
+so that their namespaces are not comingled.
+
+### Contextual options
+
+Define the expected values and handling for the following options
+wherever they are used:
+
+- `u:id` — A string value that is included as an `id` or other suitable value
+  in the formatted part(s) for the placeholder,
+  or any other structured formatted results.
+  Ignored when formatting to a string, but could show up in error messages.
+- `u:locale` — A comma-delimited list of BCP 47 language tags,
+  or an implementation-defined list of such tags.
+  The tags are parsed, and they replace the _locale_
+  defined in the _formatting context_ for this expression or markup.
+- `u:dir` — One of the string values `ltr`, `rtl`, or `auto`.
+  Replaces the character directionality
+  defined in the _formatting context_ for this expression or markup.
+
+Error handling should be well defined for invalid values.
+
+Additional restrictions could be imposed,
+e.g. requiring that each `u:id` is unique within a formatted message.
+
+### Attributes
+
+Add support for standalone `@key` as well as
+option-like `@key=value` attribute pairs with a literal value
+at the end of any expression or markup.
+
+To distinguish attributes from options,
+require `@` as a prefix for each attribute asignment.
+Examples: `@translate=yes` and `@xliff:canCopy`.
+
+Do not allow expression or markup attributes to influence the formatting context,
+or pass them to function handlers.
+
+Drop variable values from the `attribute` rule:
+
+```diff
+-attribute = "@" identifier [[s] "=" [s] (literal / variable)]
++attribute = "@" identifier [[s] "=" [s] literal]
+```
+
+## Alternatives Considered
+
+### Do nothing
+
+Continue to [caution](https://github.com/unicode-org/message-format-wg/blob/d38ff326d2381b3ef361e996c3431d1b251518d6/spec/syntax.md#attributes)
+function authors and other implementers away from creating function-specific or implementation-specific option values
+for the use cases presented above.
+
+As should be obvious, the current situation is not tenable in the long term, and should be resolved.
+
+### Do not provide any guidance
 
-If the syntax for function options is extended to support flag-like options
-(see <a href="https://github.com/unicode-org/message-format-wg/issues/386">#386</a>),
-also extend expression attribute syntax to match.
+Do not include in the spec rules or guidance for declaring formatted part identifiers,
+or overriding the message locale or directionality.
 
-To distinguish expression attributes from options,
+Do not define a common way to communicate information
+about an expression or markup to translators or tools.
+
+This would mean not defining anything for default registry functions either,
+effectively requiring implementation-specific options like `icu:locale`.
+
+Other functions could use their own definitions and handling for similar options,
+such as `locale` or `x:lang`.
+
+Formatted parts for markup would not be able to directly include an identifier.
+
+If not explicitly defined, less information will be provided to translators.
+
+Function options may be used as a workaround,
+but each implementation and user will end up with different practices.
+
+### Define options for default registry only
+
+Do not define a common way to communicate information
+about an expression or markup to translators or tools.
+Instead, define at least `locale` and `dir` as options for default registry functions,
+with handling internal to each function implementation.
+
+Other functions could use their own definitions and handling for similar options,
+such as `locale` or `x:lang`.
+
+Formatted parts for markup would not be able to directly include an identifier.
+Implementations and users will need to invent their own practices with
+markup option names like `l10n-id`, `mf:id`, or `markupId`
+to refer to specific markup parts.
+
+Do not define a common way to communicate information
+about an expression or markup to translators or tools.
+
+### Use attributes also for contextual options
+
+Add support for standalone `@key` as well as
+option-like `@key=value` attribute pairs with a literal or variable value
+at the end of any expression or markup.
+
+To distinguish attributes from options,
 require `@` as a prefix for each attribute asignment.
 Examples: `@translate=yes` and `@locale=$exprLocale`.
 
-Define the meaning and supported values of some expression attributes in the specification,
+Define the meaning and supported values of some attributes in the specification,
 including at least `@dir` and `@locale`.
 
 To support later extension of the specified set of attributes while allowing user extensibility,
-suggest custom attribute names to include a U+002D Hyphen-Minus `-`.
-Examples: `@can-copy=no`, `@note-link=|https://...|`.
+require custom attribute names to be namespaced.
+Examples: `@xliff:can-copy=no`, `@note:link=|https://...|`.
 
 Allow expression attributes to influence the formatting context,
 but do not directly pass them to user-defined functions.
 
-## Alternatives Considered
+### Use function options, but with some suggested "discard" namespace like `_`
 
-### Do not support expression attributes
+Examples: `_:translate=yes` and `_:example=|World|`.
 
-If not explicitly defined, less information will be provided to translators.
+Requires reserving an additional namespace.
 
-Function options may be used as a workaround,
-but each implementation and user will end up with different practices.
+Requires cooperation from implementers to ignore all options using the namespace.
 
-### Use function options, but with some suggested prefix like `_`
+Makes defining namespaced attributes difficult.
 
-A bit less bad than the previous, but still mixes attributes and options into the same namespace.
+Could be combined with the definition of `u:dir`, `u:locale`, and `u:id` contextual options.
 
 At least a no-op function is required for otherwise unannotated expressions.
 
@@ -249,17 +356,6 @@ esp. if a similar expression is used in multiple variants.
 
 Comments should not influence the runtime behaviour of a formatter.
 
-### Define `@attributes` as above, but explicitly namespace custom attributes
-
-As namespacing may also be required for function names and function option names,
-and because we want to allow at least for custom function options
-to be definable on default formatters,
-the namespace rules for parts of the specification would end up differing.
-
-By suggesting instead of requiring,
-we rely on our stability policy to guide implementations to keep clear of the namespace
-that may be claimed by later versions of the specification.
-
 ### Enable function chaining within a single expression
 
 By allowing for multiple annotation functions on a single expression,