CHANGES.md

• v2

v2

v2 was released in November 2020. It contains the following changes, some of which are backwards-incompatible:

• Added optional LexString() and LexBytes() methods that lexer definitions can implement to fast-path lexing of bytes and strings.

• A new stateful lexer has been added.

• A filename must now be passed to all Parse*() and Lex*() methods.

• The text/scanner lexer no longer automatically unquotes strings or supports arbitary length single quoted strings. The tokens it produces are identical to that of the text/scanner package. Use Unquote() to remove quotes.

• Tok and EndTok will no longer be populated.

• If a field named Token []lexer.Token exists it will be populated with the raw tokens that the node parsed from the lexer.

• Support capturing directly into lexer.Token fields. eg.

  type ast struct {
      Head lexer.Token   `@Ident`
      Tail []lexer.Token `@(Ident*)`
  }
  

• Add an experimental/codegen for stateful lexers. This provides ~10x performance improvement with zero garbage when lexing strings.

• The regex lexer has been removed.

• The ebnf lexer has been removed.

• All future work on lexing will be put into the stateful lexer.

• The need for DropToken has been removed.