[Suggestion]: Allow all unclosed inline markups

[boltlessengineer]

Norg spec's Layer 1 is for use-cases like general messaging apps.

And for messaging apps, it's really weird to have "syntax error" on message text.
v1 spec tried fallback to punctuations in most cases when any inline markups are incomplete or invalid.

But that made parser implementation really difficult.
Instead, I'm suggesting to make every incomplete markups valid and never throw syntax error on paragraph level.

Example

*unclosed bold is valid bold
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ bold

*last /markup should _close/ first.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ bold
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ italic
                     ^^^^^^^^^^^^^^ underline
                           ^ punctuation (not italic closing modifier)

[unclosed link is also valid
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ anchor for "unclosed link is also valid"

We are still forcing user to close last markup like #34 suggested.
But we are allowing all incomplete markups as valid instead.

Implementation

Parser can automatically close any markups with zero-width token when it encounters paragraph break (or any other tokens that make paragraph end.)
This is fairly easy in tree-sitter parser too. Why v3 couldn't do this before was because I tried to make /*bold/ not bold not /*bold/ still italic&bold.

Pros

More suitable for layer 1 use-cases

As I mentioned above, it is extremely weird to have syntax error in simple messaging app.

Better editor UX

In mobile applications, user might use keyboard toolbar buttons to toggle markups instead of manually inserting special characters.
If they turn on "bold" button, app should insert bold closing modifier like auto-pair to make it valid bold forcing user to move the cursor when they want to end the bold text. But moving cursor in mobile applications is quite hard.

If we allow unclosed markups, the editor UX feels more like WYSIWYG editor. When user turns on "bold" button, every following text will be bold unless user turns off "bold". If user wants to end the bold text, they can simply turns off the "bold" from keyboard toolbar. No need to manually move the cursor.

Edge-cases

Only edge-case I can think of right now is \. When norg content ends with \ character without EOL or EOF.
Though I think this edge-case is rare enough to just ignore on parsing.

[vhyrro]

I agree with the general sentiment, but I don't think I can say yes to this proposal sadly. Software is moving in the direction of catching mistakes early. What's really happening when you don't close markup is you haven't specified an expected region for the markup. An expected region is made up of a start and end marker. If the user hasn't specified an end marker, we can't assume the region for them. This is basically like saying the end marker is null, and we know that nulls aren't good hah.

On a large scale (say e.g. a massive note) this sort of erroring is really important. It may be slightly more annoying to see that your paragraph has an "error" because the markup is unclosed, but that's still much better than the format pretending that everything is okay, only for you to find out that on page 632 of your book you forgot to close a link and half the entire page is malformed.

As for layer 1 use cases, all parsers have error recovery. Even treesitter does a good job by inserting (MISSING "*") nodes in places where the markup should be. Such parsers could insert the missing nodes themselves, or could instead choose to go to each node and precede the unclosed markup with a \. I would expect they'd have hand rolled parsers of some kind anyway, so they could handle this edge cases themselves or patch the parser to work a little different for their specific use case.

[boltlessengineer]

An expected region is made up of a start and end marker.

In parser perspective yes, but I don't think that's what's happening in general user's mind. When a user type * right before a non-whitespace character, the user must going to do something with bold. They haven't specified an end marker, which means it should span endlessly.

Just like how GUI applications work. When user toggles the bold style, it will be preserved until they explicitly ends the bold mode.

On a large scale (say e.g. a massive note) this sort of erroring is really important

We can still force explicit markup close rule by linter rule. That's not that hard. I think it's actually more easy than collecting undefined parser error.
TreeSitter is designed to handle error recovery, but as you know, it's designed for many other aspects and it is clearly not a best parser framework for markup languages.

they could handle this edge cases themselves

I'm heavily against with this idea. Leaving undefined behavior and let every single applications to implement by their own preference.
Personally this was most annoying part of markdown ecosystem. All applications have different behavior in certain scenarios. I've seen some applications even have different behavior in same window (editor and preview.)

[boltlessengineer]

@vhyrro Let me rephrase what I'm suggesting.

• all incomplete inline markups should have a defined fallback. (no undefined behavior)
• paragraph break still has highest precedence than any other inline syntax. So When paragraph ends, it ends. Have clear separtion between inline syntax and block syntax.

Reasons

Layer 1 should never be broken

Layer 1's main use-cases are SNS applications. So they have to render something ragardless of what user wrote. Getting "ERROR: please fix the error on column 54" on messaging app will be really weird experience. As we have to render something, we should define what should be rendered on each cases and we should not leave several /undefined behaviors/ in our markup syntax.

all parsers have error recovery. Even treesitter does a good job by inserting (MISSING "*") nodes in places where the markup should be. Such parsers could insert the missing nodes themselves, or could instead choose to go to each node and precede the unclosed markup with a \. I would expect they'd have hand rolled parsers of some kind anyway, so they could handle this edge cases themselves or patch the parser to work a little different for their specific use case.

xkcd-927 problem here. We can't just say "There will be parsers not following the spec eventually" and let them do what they want. That's exactly what happened to Markdown (every parser claims to support CommonMark & GFM. Which means they don't) and I hope Norg to be better than that.
Yes, there will be parsers not following the rules. But if they don't follow the rule, they should be defined as non-compliant parsers.

I'm saying that we should never have undefined behavior and provide strict rules for every single edge cases we can imagine.
This is a hard job. But we can at least try.

Give better user experience

Above was reason for why we should not have undefined behaviors, and this one is about why that undefined behavior for unclosed markups should be "valid markup".

tl;dr

• Better than "fallback to normal text" because we can indicate users about the opened markups.
• Actually it is only way to define behavior for unclosed markups.

Imagine this: (| as cursor position)

word` word `word word word word|

What should user see? They should at least know something is going on after column 12. The context has been changed. If they insert ` here, it will have different meaning from ` at column 5.

*word
word|

* heading

word
word*

This is also a common broken markup example and it is clear that user don't want to break the heading below. Even tree-sitter has some error recovery features, it can't handle this kind of situations well (e.g. unclosed string).

If we know we don't want to go futher than paragraph break like this example, it would be better to make parsers do exactly that.

It may be slightly more annoying to see that your paragraph has an "error" because the markup is unclosed, but that's still much better than the format pretending that everything is okay, only for you to find out that on page 632 of your book you forgot to close a link and half the entire page is malformed.

Why won't we have both? Give better experience while writting and provide linter to force user to explicitly close them. Parsable errors are even better than errors literally breaking the parser. For example, rust-analyzer parses many imaginary non-existing syntax to provide better error messages.

We can just put (unclosed_bold) alongside with (bold) and let applications to choose how to render (unclosed_bold). In some use-cases, having any kind of errors can be nonsense, or they might even not have ability to indicate the error. But there are also some cases where user needs the error. Application behavior on error may differ by applications. So having parser that can parse all will satisfy both.