BiDi test cases unclear / implementation specific

[mihnita]

In spec/bidi.json we have these tests:

{
    "description": "complex-message   = o *(declaration o) complex-body o",
    "src": ".local $x = {1} \u200F {{ {$x}}}",
    "exp": " \u20681\u2069"
},
...
{
    "description": "function-expression = \"{\" o function *(s attribute) o \"}\"",
    "src": "{1 \u200E :number \u200F}",
    "exp": "1"
},

They look like the same test to me, but they expect different results.

Why I think that they are technically the same test:

1. The bidi control characters in between parsed elements (sprinkled with the spaces for proper rendering)
   have no effect on functionality.

So the two sources become:

    "src": ".local $x = {1} {{ {$x}}}",
    "src": "{1 :number}",

Both format the value 1 from a number literal.

2. We did allow implementations to use the proper function for a type.

So if I pass a date, the implied function will be :dateformat.

For a number literal, the most intuitive implied function would be :number

So formatting {1} and {1 :number} is the same.

---

Since the input of the test is virtually the same, we should expect the same result.

One might say "no, {1} is not the same as {1 :number}, because 1 in is a string, so the associated method is :string.

---

Also, rendering {1} as if the function is :string has no useful functionality.
It means we know the string to format in advance, at compile time.
At we render it "as is".

So there is no point to put it in a placeholder to begin with.

I know we don't have in the spec anything about what how to determine default functions.

But if everything is a string, and everything goes to :string, the whole expression with string literal without function becomes a lot less useful.

And having such a test in official test suite, even if not specified anywhere, prevents implementations from doing better.

[eemeli]

So formatting {1} and {1 :number} is the same.

This is not allowed. We only allow for an implementation to choose to apply additional processing (such as implicitly considering the placehodler to have a :number or :string function) for variables:

Else, if the expression consists of a variable, its resolved value is defined by variable resolution. An implementation MAY perform additional processing when resolving the value of an expression that consists only of a variable.

[mihnita]

This is not allowed. We only allow for an implementation to choose to apply additional processing (such as implicitly considering the placehodler to have a :number or :string function) for variables:

These are existing tests in bidi.json:

    {
        "description": "complex-message   = o *(declaration o) complex-body o",
        "src": "\u200E .local $x = {1} {{ {$x}}}",
        "exp": " \u20681\u2069"
    },
    {
        "description": "complex-message   = o *(declaration o) complex-body o",
        "src": ".local $x = {1} \u200F {{ {$x}}}",
        "exp": " \u20681\u2069"
    },
    {
        "description": "complex-message   = o *(declaration o) complex-body o",
        "src": ".local $x = {1} {{ {$x}}} \u2066",
        "exp": " \u20681\u2069"
    },

If they are not allowed then we should either remove them, or they should have "expErrors"

[eemeli]

Those messages are entirely valid though? What's not allowed is for an implementation to introspect a literal value and based on that change how that literal is formatted.

[mihnita]

What's not allowed is for an implementation to introspect a literal value and based on that change how that literal is formatted.

Yes, that is the problem, with everything becoming a string (literal ~ string now)
But 1 was not a mere literal value, is a number literal.

At some point we shifted to demoting that idea and make everything strings.
In doing that we lost some functionality.

OK, was discussed already, there is no appetite to change that. I know.

But let's clarify these test cases.

You say:

This is not allowed.
...
Those messages are entirely valid though?

What is it?

---

If we say that the tests are valid, then what it means is that a string literal is always formatted with :string if a function is not specified.

The only utility for that is to make things read-only.
Protecting a piece of the message from translation.

Do you think that should be explicitly written somewhere (probably in formatting.md?)

[aphillips]

Yes, that is the problem, with everything becoming a string (literal ~ string now)

I think this might be overstating it a bit. To be sure: literals are literals (== strings).

At some point we shifted to demoting that idea and make everything strings.
In doing that we lost some functionality.

Actually, what changed is that we require the function annotation in order to do selection. We do explicitly allow variables to have their type introspected by an implementation and thus be defaulted to a different function:

An implementation MAY perform additional processing when resolving the value of an expression that consists only of a variable.

For example, it could apply function resolution using a function and a set of options chosen based on the value or type of the variable. So, given a message like this:

   Today is {$date}

If the value passed in the variable were a date object, such as a JavaScript Date or a Java java.util.Date or java.time.Temporal, the implementation could interpret the placeholder {$date} as if the pattern included the function :datetime with some set of default options.

We don't say whether this is allowed (or not) for literals. In the presence of a function, the function handler deals with interpreting the literal. In the absence of a function annotation, I think we presume only :string is available. If you want a different function, you are obligated to name it:

You have {1} and only {1 :number} wildebeest.

If we say that the tests are valid, then what it means is that a string literal is always formatted with :string if a function is not specified.

My above example treats {1} as a :string and the other, obviously, is a number. Allowing introspection of the literal in the first placeholder might produce wildly different results across implementations. It's unclear if that is a Good Thing. We should probably say something in the section on Literal Resolution about this.

Do you think that should be explicitly written somewhere (probably in formatting.md?)

Yes.

[aphillips]

Actually, we already do say this:

Note

This means that a literal value with no function
is always treated as a string.
To represent values that are not strings as a literal,
a function needs to be provided:

[eemeli]

This is not allowed.
...
Those messages are entirely valid though?

What is it?

The quote above is missing the context for the first. It is specifically responding to

So formatting {1} and {1 :number} is the same.

In order for an implementation to format {1} and {1 :number} as the same, introspection of the literal value is required, and that is not allowed. This is also reflected in the fuller context of the second quote:

Those messages are entirely valid though? What's not allowed is for an implementation to introspect a literal value and based on that change how that literal is formatted.

Please note that I try to be specific about the language I use, and "valid" has a very well-defined meaning in MF2. When I say "not allowed", that's not a synonym for "not valid", but an attestation that some interpretation of the spec is not permissible for an implementation or other processor thereof.

[aphillips]

@mihnita I missed asking you about this one earlier. Okay to move to 48? Or is this a 47 blocker for you?

[mihnita]

@mihnita I missed asking you about this one earlier. Okay to move to 48? Or is this a 47 blocker for you?

Not a blocker.
It is just a useless appendage. Similar to the attributes.

[aphillips]

Thanks @mihnita

I added this and the other issues to the agenda issues table for discussion in an upcoming call. I don't think we'll get to them Monday, but we will get to them.