exclude text when match until newline

[Yokozuna59]

If I want to match names in this format:

name: <anything until newline>

But I don't want to match name: itself, just everything after it until newline. But this code do:

grammar Persons

entry Model:
    (names+=FULL_NAME)*;

terminal FULL_NAME: "name:" /[^\n].*/;

I tried using this code, but it will be an infinite loop:

grammar Persons

entry Model:
    ("name:" names+=(FULL_NAME))*;

terminal FULL_NAME: /[^\n].*/;

So how to exclude name:?

Note: STRING parse rule is reserved for string wrapped between double quote.

terminal STRING: /"[^"]*"/;

[msujew]

@Yokozuna59 The language you're building doesn't seem to be tokenizable by just using Regexp. Chevrotain doesn't support positive lookbehind as of now, so you're best approach would be to override the TokenBuilder service and change the pattern property of your FULL_NAME token to a custom matcher function.

[Yokozuna59]

@Yokozuna59 The language you're building doesn't seem to be tokenizable by just using Regexp. Chevrotain doesn't support positive lookbehind as of now, so you're best approach would be to override the TokenBuilder service and change the pattern property of your FULL_NAME token to a custom matcher function.

Oh, has anyone done it before, overriding the TokenBuilder services and reintegrate Langium again? I want to see an example of whatever the case was so I can come up with my own use case.

[msujew]

Please note that service overrides are an integral and relatively simple part of Langium. You don't need to clone Langium to do that. Our documentation on that process can be found here.

[Yokozuna59]

I have created an example for what I wanted: link.

1. I only want one name to be declared, but if there is another name, then override the previous one (don't want array of names).
2. empty names should be marked as undefined, e.g., name:

It's working fine for parsing the extracting data from the code, but I keep getting parser errors. That's why I've commented them here.

Is there a way to resolve this using Langium API? If not, how can I use Chevrotain API and preform CONSUME to stop getting those error.

[Yokozuna59]

never mind, I figured it out. I didn't know it was possible to add Cardinalities to a single parse rule.

Anyway for anyone who might ask for it here is the code (because I'll delete the repo):

Langium grammer:

grammar Example

entry Example:
    name=NAME*;

// placeholder terminal rule
terminal NAME: /name/;

Override TokenBuilder:

import {
  CustomPatternMatcherFunc,
  CustomPatternMatcherReturn,
  TokenType,
} from "chevrotain";
import { DefaultTokenBuilder } from "langium";
import { TerminalRule } from "langium/lib/grammar/generated/ast";

const nameRegex = /(?:^|[\t ]+)name[\t ]*:[\t ]*([^\n]+)?/;
const matchName: CustomPatternMatcherFunc = (text: string) => {
  let result: CustomPatternMatcherReturn | null = null;
  let match = nameRegex.exec(text);
  while (match !== null) {
    result = match[1] === undefined ? null : [match[1].trim()];
    text = text.replace(match[0], "").trim();
    match = nameRegex.exec(text);
  }
  return result;
};

export class ExampleTokenBuilder extends DefaultTokenBuilder {
  override buildTerminalToken(terminal: TerminalRule): TokenType {
    const tokenType = super.buildTerminalToken(terminal);
    if (tokenType.name === "NAME") {
      tokenType.PATTERN = matchName;
      tokenType.LINE_BREAKS = false;
    }
    return tokenType;
  }
}

Thanks for you help!