The latest rego grammar is required

[gitchong]

When I use the old grammar, the rego input file cannot be recognized correctly. Since I am a rookie and cannot modify the lexer and parser very well, I need the latest rego grammar. I am going to use java to check whether the rego file is grammatically correct with the help of antlr v4. Please help me if there is an easier way, thanks a lot

[anderseknert]

Hi @gitchong! While it's not 100% up to data, you could take a look at the BNF used by the IntelliJ OPA plugin: https://github.com/open-policy-agent/opa-idea-plugin/blob/master/src/main/grammar/Rego.bnf

It's also used in a JVM project, so you should be able to pull some of the code from there as well if that would help.

[gitchong]

First of all, thank you for your help. I have been trying to use the bnf you mentioned for a long time. Due to the strong dependence of opa-idea-plugin and idea itself, I need to call it separately. When creating PsiBuilder, the environment project is always missing. After repeated attempts, I gave up I changed to antlr v4 to solve my problem, but the RegoLexer.g4 I wrote is always not so perfect, I really need help

[anderseknert]

Perhaps @vgramer who maintains the IDEA plugin knows how to best create the PsiBuiilder without that happening. If you write a custom one yourself i'll be difficult for others to help with that.

[vgramer]

@gitchong grammar used in the plugin is based on the OPA documentation. If antlr v4 supports left recursion, it should be pretty straightforward to convert it from the documentation.

I haven't opened PR on opa-idea-plugin yet, because the generated AST is not correct, but if you just want to check file is syntaxly correct, it should not be a problem.

Here is my work in progress:

  tokens = [
        PACKAGE_TOKEN       = 'package'
        IMPORT_TOKEN        = 'import'
        AS                  = 'as'
        DEFAULT             = 'default'
        VAR_ASSIGN          = ':='
        UNIFICATION         = '='
        TRUE                = 'true'
        FALSE               = 'false'
        LPAREN              = '('
        RPAREN              = ')'
        LBRACK              = '['
        RBRACK              = ']'
        ELSE                = 'else'
        LBRACE              = '{'
        RBRACE              = '}'
        SEMICOLON           = ';'
        SOME                = 'some'
        COMMA               = ','
        DOT                 = '.'
        NOT                 = 'not'
        WITH                = 'with'
        IN                  = 'in'
        EVERY               = 'every'
        IF                  = 'if'
        CONTAINS            = 'contains'
        COLLON              = ':'
        EQ                  = '=='
        NOT_EQ              = '!='
        LESS_OR_EQUAL       = '<='
        GREATER_OR_EQUAL    = '>='
        GREATHER            = '>'
        LESS                = '<'
        BIT_OR              = '|'
        BIT_AND             = '&'
        PLUS                = '+'
        MINUS               = '-'
        MUL                 = '*'
        QUOTIENT            = '/'
        REMAINDER           = '%'
        SET_OPEN            = 'set('
        NULL                = 'null'
        ASCII_LETTER        = 'regexp:[A-Za-z_][A-Za-z_0-9]*'
        NUMBER              = 'regexp:-?(0|[1-9]\d*)(\.\d+)?([eE][+-]?\d*)?'
        WHITE_SPACE         = 'regexp:\s+'
        STRING_TOKEN        = 'regexp:"([^\\"\r\n]|\\[^\r\n])*"?'
        RAW_STRING          = 'regexp:`[^`]*(`)?'
        COMMENT             = 'regexp:[ \t]*#[^\r\n]*'
    ]
}


module              ::= package (import|  rule)*
package             ::= "package" ref
import              ::= "import" ref ( "as" var )?
rule                ::= "default"?  rule-head  rule-body*
rule-head       ::= ref ( rule-head-set | rule-head-obj | rule-head-func | rule-head-comp )
rule-head-comp  ::= ( assign-operator expr )?  "if"? // term has been changed to expr. otherwise a := a + 2  raise an error on +
rule-head-obj   ::= "[" term "]" ( assign-operator expr )? "if"?
rule-head-func  ::= "(" rule-args ")" ( assign-operator expr )? "if"?
rule-head-set   ::= "contains" term "if"? | "[" term "]"
rule-args       ::= term { "," term }
rule-body       ::= else-expr | "{" query "}" | literal

// TODO explain fix
else-expr       ::=      else-expr-A | else-expr-B
else-expr-A       ::= "else" assign-operator expr ("if" (expr | "{" query "}"))?
else-expr-B      ::= "else" if? "{" query "}"



query               ::= ( literal |';' )+
literal             ::=( some-decl | expr | "not" expr ) with-modifier*
with-modifier       ::= "with" term "as" expr // term has been changed to expr. allow with input.yes as true == true
some-decl           ::= "some" term ( "," term )* ( "in" expr)*
expr-call           ::= var ref-arg-dot* "(" ( expr ( "," expr )* ","? )? ")"

//TODO explain membership + remove left recursion
expr                ::= base-expr (exp-membership)*
exp-membership      ::= ("," base-expr)? "in" base-expr
base-expr            ::= (expr-every | expr-parens | unary-expr | (expr-call ref-arg*) | term ) expr-rest
expr-rest      ::= (infix-operator base-expr)*


expr-parens         ::= "(" expr ")"
unary-expr          ::= "-" expr
expr-every          ::= "every" var ( "," var )? "in" ( expr | expr-call| term ) "{" query "}" // TODO explain why expr-infix has been changed to expr

// todo explain why membership  has been removed  has it should be on expr level
term                ::= ref | var | scalar | array | object | set | array-compr | object-compr | set-compr
array-compr         ::= "[" term "|" query "]"
set-compr           ::= "{" term "|" query "}"
object-compr        ::= "{" term-pair "|" query "}"
term-pair           ::= term  ":"  term
infix-operator      ::= assign-operator | bool-operator | arith-operator | bin-operator
bool-operator       ::= "==" | "!=" | "<" | ">" | ">=" | "<="
arith-operator      ::= "+" | "-" | "*" | "/" | "%"
bin-operator        ::= "&" | "|"
assign-operator     ::= ":=" | "="
ref                 ::= ( expr-call  | array | object | set | array-compr | object-compr | set-compr |var  )  ref-arg*
ref-arg             ::= ref-arg-dot | ref-arg-brack
ref-arg-brack       ::= "[" ( expr | "_" ) "]"
ref-arg-dot         ::= "." var

// var must also matches keywords "in", "every" otherwise import can not be parsed (<var> expected, got 'every')
var                 ::= ASCII_LETTER | IN | EVERY | IF |CONTAINS
scalar              ::= string | NUMBER | TRUE | FALSE | NULL
string              ::= STRING_TOKEN| RAW_STRING
array               ::= '[' expr? ( ',' expr )* ','? ']'
object              ::= '{' object-item? ( ',' object-item )* ','? '}'
object-item         ::= ( scalar | ref | var ) ":" expr
set                 ::= empty-set | non-empty-set
non-empty-set       ::= "{" term ( "," term )* ','?  "}"
empty-set           ::= "set(" ")"

[gitchong]

Thanks, I'll try it