Need a Little Grammar Help

[ballcoach12]

I'm working on a grammar that needs to support specification of HTTP paths in a similar manner to OpenAPI. I almost have it working, but I can't get the parser to recognize the SEGMENT rule as shown in this playground session

I don't know why 'abc' or 'B' are not recognized here as SEGMENT.

Grammar:

grammar Paths

entry PathModel:
    paths+=PathAttribute*;

PathAttribute:
    '[path' ':' path=URL ',' 'method'':' method=Method']';

Section:
'/' (items+=Item)* ('/' items+=Item)? ;

Item:
Param_Segment | Path_Segment;

Param_Segment:
'{' segment=PARAM '}';

Path_Segment infers PathSegment:
  segment=SEGMENT;

URL:
    sections+= Section+;

    Method returns string:
    ('get' | 'put' | 'post' | 'delete');

terminal PARAM: ID ;

hidden terminal WS: /\s+/;

terminal ID: /[_a-zA-Z][\w_]*/;

terminal SEGMENT:
    (LETTER | DIGIT | SPECIAL)+
;

terminal LETTER:
    ('a'..'z' | 'A'..'Z')
;

terminal DIGIT:
    ('0'..'9')
;

terminal SPECIAL:
    ('-' | '_' | '.' | '~' | '%' HEX HEX)
;

terminal HEX:
    ('0'..'9' | 'a'..'f' | 'A'..'F')
;

hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

[msujew]

@ballcoach12 Your terminals are clashing quite heavily. In /abc/B both abc and B get identified as PARAM terminals, since that's the first terminal in your terminal list that is able to tokenize these characters. Note that the tokenization and parsing phase are split and the tokenization (lexing) is performed without context (whereas the parsing phase uses context).

In your case, PARAM, ID, SEGMENT and LETTER are ambiguous during the tokenization phase. The tokenizer will never read ID, since the first-fit algorithm employed there will always read PARAM first. A similar thing happens with SEGMENT since there's quite an overlap in these token types.

[ballcoach12]

Thank you, @msujew! Sometimes I can't see the forest for the trees!

I revised my grammar to clear up the clutter, and it now works. Here is the new grammar:

grammar Paths

entry PathModel:
    paths+=PathAttribute*;

PathAttribute:
    '[path' ':' path=URL ',' 'method'':' method=Method']';

Section:
'/' (items+=Item)* ('/' items+=Item)? ;

Item:
Param_Segment | Path_Segment;

Param_Segment:
'{' segment=ID '}';

Path_Segment infers PathSegment:
  segment=ID;

URL:
    sections+= Section+;

    Method returns string:
    ('get' | 'put' | 'post' | 'delete');

hidden terminal WS: /\s+/;

terminal ID: /[_a-zA-Z][\w_]*/;

hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;