Rants

[ericvergnaud]

This place is for respectfully criticizing antrl4, throwing out immature suggestions, any stuff that doesn't yet deserve a dedicated discussion.

[ftomassetti]

First of all, thank you for accepting me in this amazing group!

I would like to share the use-cases that we have for ANTLR 4 at Strumenta, as the starting point discussing potential improvements in ANTLR 5.

We use ANTLR on a daily basis, across different projects to build two main kinds of projects:

• Legacy modernization projects: in this case we build parsers that we use to analyze large codebases and then to transpile to another language (e.g., from RPG to Java, from SAS to Python, from Visual Basic to Javascript, etc.)
• Domain Specific Language projects: in this case we develop parsers then we later use in interpreters, code-generators, and editors

Do you also use ANTLR for similar purposes? Or do you build different things with it?

Error tolerance and API

I think there is a main difference in how we use the parser in the two contexts I described above:

• When building parsers for supporting transpilation, interpretation, or code analysis we deal with code that is correct. Therefore we appreciate an API that take advantage of the correctness of the tree
• When building editors we instead need API that a robust and let us work with trees which are incorrect. However, they are more cumbersome to work with

For example, let's suppose that we are examining a sum operation, When working with parsers with correct code, we may want the node class to look like this:

class SumOperation {
    var left: Expression // this in Kotlin means the Expression is guaranteed not to be null
    var right: Expression
}

So that we can access the left and right operand and be sure to get valid values back. This means skipping a lot of ifs.

When using the parser within the editor, we instead need to have this API:

class SumOperation {
    var left: Expression? // this in Kotlin means the Expression may or may not be null
    var right: Expression?
}

This is the case because the expression could be malformed (as the user is typing the code when we process it). Because of that the tree may be incomplete. We still need to process it for things like autocompletion.

Multiple platforms

Often we need to build a parser for the same language and be able to use it from multiple platforms.

These is the case for two different reasons:

• We may want to build some code processing logic to run on the JVM or CLR or Python, and then an editor running in the browser
• We may build a parser for a certain language targeting the JVM, and then have a second client ask for a parser for the same language, but running on Python

While we can write a grammar without semantic actions or predicates and generate parsers using multiple targets, we typically also need to share some code around the parsers, to process some nodes. For example, we could want to share some semantic error-checking logic or symbol resolution logic across the different platforms. Right we use some form of inter-process communication, with one component doing the work, for example on the JVM, and another component in Typescript or Python contacting the JVM component through web sockets and getting some result serialized in JSON, protobuf or something else. We have our own tools to generate stubs and handle serialization and de-serialization on the two sides, but still deployment is a little bit cumbersone.

We also explored the idea of solving this problem by using Kotlin multi-platform, and this is why we started working on the ANTLR Kotlin target. The idea was to generate Kotlin multiplatform parser code, depending on the Kotlin multiplatform runtime, and build libraries on top of that in Kotlin multiplatform, to be later integrated in JVM applications, browser applications, but also Python by generating native libraries and Python wrappers.

I understand that the goal is to go for WASM and perhaps that will solve this problem for us.

Tree reorganization + APIs

Currently the way in which we use ANTLR is by processing its results and wrapping them with one of our libraries (Kolasu, Tylasu, Pylasu, or SharpLasu, they are basically porting of the same library to Kotlin, TypeScript, Python, and C#, where Kolasu is by far the most mature one).

We do that for two reasons:

• When translating from the ANTLR tree to our own tree we can do some tree reorganization, producing a nicer tree
• Our trees have richer APIs

Tree re-organization

For tree reorganization, we do that in two ways:

• Adding special comments in the grammar and then use tools that process those annotations to guide the generation of our tree classes
• Through code, for all tree transformations that are not trivial or common (for those we have our special comments)

Special comments in the grammar can look like this:

While the mapping in code could look like:

        registerNodeFactory(CobolParser.DecLiteralExprContext::class) { pt: CobolParser.DecLiteralExprContext,
            t: ASTTransformer ->
            DecLiteralExpr(BigDecimal(pt.DEC().text.replace(",", ".")))
        }

APIs

We have a lot of APIs to do things such as:

• Exporting to JSON and XML (I think @kaby76 mentioned this in his wish list)
• Providing exporting/importing capabilities to EMF and LionWeb (a "standard" for modeling that is currently being developed)
• Homogeneous/Reflective APIs
• APIs to traverse the tree
• APIs to transform the tree

I could go more in detail about this in separate issues, as we have a lot of different APIs to process the trees in different ways, but here there are a few considerations.

The homogeneous/reflective APIs is the most important one. It consists in the possibility of asking each node for its type, and then ask the type for its structure. This is useful to implement generic algorithms like XML serialization.

For example, something like:

fun serializeNode(node: Node, tagName: String) XMLNode{
    val xmlNode = XMLNode(tagName)
    xmlNode.addAttribute("nodeType", node.concept.name)
    node.concept.attributes.foreach { attr ->
        xmlNode.addAttribute(attr.name, attr.valueFor(node))
    }
    node.concept.ruleReferences.foreach { ruleRef ->
        xmlNode.addChild(serializeNode(ruleRef.valueFor(node), ruleRef.name))
    }
    return xmlNode
}

Regarding APIs to traverse the tree, we like to use tree iterators and ways to find in a subtree nodes of a given type. We never use listeners or visitors.

I hope this is not too long, and if there is any question I am happy to go more in detail

[kaby76]

Amen on a tree reorganization! I use an XPath-based system to do semantic analysis and some simple transformations. To see some of things I do with Antlr grammars, here are a few of the tools: https://github.com/kaby76/g4-scripts

I really can't do tree editing with a token stream because the tokens are indexed by integer into the stream. Adding and deleting tokens is very slow, and basically impractical. If I break up a transpiler into its component refactorings, I cannot just reparse the partially rewritten frontier after editing. The model presented with TokenStreamRewrite doesn't work for my use, which involves scraping, refactoring, testing, and translating EBNF.

Visitors and listeners are ok, but as I've pointed out many times, it's like programming in assembly language. It's great that it's available, but as a first choice, who needs to solve real-world problems using assembly code?? We need a better way of working with trees. XQuery and XPath is a proven tech and they work really, really well with parse trees. We need a wrapper or export to a DOM so we can use better tools. The existing XPath that's implemented in some of the Antlr runtime should be discarded, and a real-world solution used in its place.

[KvanTTT]

Error tolerance

It looks like this section correlates with antlr/antlr4#4040 (originally antlr/antlr4#1972)

I have an idea how to handle optional expression in ANTLR grammar to distinguish between absence and erroneous expressions. Also, I think it makese sense to create classes for every alternative for better corresponding with target-language type system. I'll try to implement it in a new Antlr grammar parser written by hand (lexer is already ready as I wrote).

[ericvergnaud]

User rant: antlr/antlr4#455

[jimidle]

I think that user request was rightly closed. The IDEs that support Maven knew about the generated java classes and they coudl easily be debugged. It was an XY question I think.

Output location coudl certainly be modified according to the target, but then we will get reports that:

• It doesn't work like ANTLR4 and so we have to rejig our build tools!
• The output directory is not what we want
• Why does it change for different targets
• and so on

The tool should not try to make such decisions and just do what it is told to do.