What new features will antlr5 support ?

[ericvergnaud]

• support for templates
• clarified imports/includes
• built-in macros for common lexer patterns: string, ws, id, integer, decimal…
• improved grammar checking
• token categories (keyword, control, literal…) fo facilitate generation of simple editor support
• contextual tokens (for example, treating 'value' differently in C# depending on the context: Property setter or not ?)
• delegated grammars (to embed js and css in html)
• token value comparators
• incremental parsing
• auto-completion
• reflective-apis
• predictable-cache
• parser fragments

[ericvergnaud]

Adding Ken's wish list:

Wish List for Antlr5.pdf

[KvanTTT]

Does it make sense to answer here or it's better if @kaby76 extract topics from pdf to GitHub discussions? I have thoughts on some ideas.

[ericvergnaud]

It's better to create separate discussions, and keep this discussion as the inventory of all feature-related discussions

[ericvergnaud]

This discussion is bout the list of features, not the features themselves.
The discussion for each feature is managed in its dedicated GH discussion (see 1st conversation item above).
We will discuss prioritization here.

[KvanTTT]

If Token value comparison operator is implemented it will significantly simplifies SQL and other grammars with many contextual keywords and make them less error-prone (there have been a lot of cases when new users introduced a new keyword without adding it to identifier list).

UPDATE: I see you've already added the similar item:

contextual tokens (for example, treating 'value' differently in C# depending on the context: Property setter or not ?)

[ericvergnaud]

We can dive into details in the dedicated discussion

[KvanTTT]

token categories (keyword, control, literal…) fo facilitate generation of simple editor support

It's a good idea. But I've also added a splitter token (typically it's new line '\n`). It can help to improve lexer/parser errors reporting, make parse-tree full fidelity (when whitespaces and comments are linked with respective normal tokens).

[ericvergnaud]

Not sure what that means exactly, can you create the corresponding discussion and elaborate ?

[KvanTTT]

Yes, it's better creating a new discussion with description. I'll do that.

[ericvergnaud]

As of writing, I'm seeing some level of clarity and consensus with the following:

• clarified imports/includes
• annotations (renamed)
• token value comparators

Some ideas have not reached consensus yet:

• support for templates

The following ideas have not matured:

• improved grammar checking
• incremental parsing
• auto-completion
• reflective-apis
• predictable-cache

The following ideas are abandoned:

• built-in macros for common lexer patterns: superseded by includes
• contextual tokens superseded by token value comparators
• delegated grammars closed

I might be wrong but I don't foresee any 'immature' feature impacting our design.
If that's correct, we need to reach consensus on templates, then we could start implementing things ?

[ftomassetti]

I agree. I am pretty sure that grammar checking, auto-completion, reflective-APIs, and predictable caches are something we can add later and would not impact the design. The only one that could be more impactful could be incremental parsing. Perhaps we could understand if the feature is worth the effort or not