Referencing issues

[idiosyncrasy00]

Hi, I am writing the grammar for the DSL language and making a langium-client to generate from my DSL into text. The part of grammar is expressed as below:

UsecaseCS:
         'usecase' name=UnrestrictedName
         ('description' '=' description=STRING)?
	'primaryActor' '=' primaryActor=GetActorCSName
  	('secondaryActors' '=' '{' secondaryActors+=GetActorCSName (',' secondaryActors+=GetActorCSName)* '}')?
	(precondition=UsecasePreconditionCS)?
	(postcondition=UsecasePostconditionCS)?
	firstStep=StepCS
         (extPoints+=ExtensionPointCS)*
	'end'
;

StepCS:
	(stepCS1=UCStepCS | stepCS2=RejoinStepCS | stepCS3=ActStepCS)
	(altFlows+=AltFlowCS)*
	(nextStep=StepCSTemp)?
;

StepCSTemp:
	(stepCS1=UCStepCS | stepCS2=RejoinStepCS | stepCS3=ActStepCS)
	(altFlows+=AltFlowCS)*
	(nextStep=StepCS)?
;

ActStepCS:
	ActorStepCS | SystemStepCS
;

ActorStepCS:
	(isActorStep?='actStep') (name=UnrestrictedName)?
	('description' '=' description=STRING)?
	'from'
		preSnapshot=SnapshotPatternCS
	'to'
		postSnapshot=SnapshotPatternCS
	( 'actions' (actions+=ActorActionCS)* )?
	'end'
;

SystemStepCS:
	'sysStep' (name=UnrestrictedName)?
	('description' '=' description=STRING)?
	'from'
		preSnapshot=SnapshotPatternCS
	'to'
		postSnapshot=SnapshotPatternCS
	( 'actions'
		(actions+=SystemActionCS ';')*
	)?
	'end'
;

RejoinStepCS:
	'rejoinStep' rejoinTo=UnrestrictedName
	('description' '=' description=STRING)?
	('when'
		condition=SnapshotPatternCS)?
	'end'
;

UCStepCS:
	'inclStep' (name=UnrestrictedName)?
	('description' '=' description=STRING)?
	includedUC=[UsecaseCS:QualifiedName]
	'end'
;

AltFlowCS:
	'altStep' (name=UnrestrictedName)? 'at' baseStep=UnrestrictedName
		('description' '=' description=STRING)?
		'when'
			condition=SnapshotPatternCS
		'end'
		altStep=StepCS
	'end'
;

The relevant function, built up from AST tree in langium-cli:

export function generateUsecaseCS(usecaseCS: UsecaseCS, fileNode: CompositeGeneratorNode): void {
    //other code
    let stepCS1 = usecaseCS.firstStep.stepCS1?.includedUC.$refText;
    let stepCS2 = usecaseCS.firstStep.stepCS2?.rejoinTo;
    let stepCS3 = usecaseCS.firstStep.stepCS3?.name;

    console.log("ucStep is ", stepCS1); //result is always undefined 
    console.log("rejoinStep is ", stepCS2); //result is always undefined
    console.log("actStep is ", stepCS3); //result is printed if the relevant coding block in DSL file is added
}

The issue is likely lies in here.

(stepCS1=UCStepCS | stepCS2=RejoinStepCS | stepCS3=ActStepCS)

I've read the langium documentation about the referencing issues [/discussions/829] and I wonder if my code falls in that scenario. I wonder if the issue is fixed now and there is a solution for this. The textual DSL file works fine without errors, but the code generator does not.

[idiosyncrasy00]

P/s: The AST for given above piece of code is generated as follow:

export interface UsecaseCS extends AstNode {
    readonly $container: FrslModel | PackageCS;
    readonly $type: 'UsecaseCS';
    firstStep: StepCS
}

export interface StepCS extends AstNode {
    readonly $container: AltFlowCS | StepCSTemp | UsecaseCS;
    readonly $type: 'StepCS';
    altFlows: Array<AltFlowCS>
    nextStep?: StepCSTemp
    stepCS1?: UCStepCS
    stepCS2?: RejoinStepCS
    stepCS3?: ActStepCS
}

export interface StepCSTemp extends AstNode {
    readonly $container: StepCS;
    readonly $type: 'StepCSTemp';
    altFlows: Array<AltFlowCS>
    nextStep?: StepCS
    stepCS1?: UCStepCS
    stepCS2?: RejoinStepCS
    stepCS3?: ActStepCS
}

export type ActStepCS = ActorStepCS | SystemStepCS;

export interface ActorStepCS extends AstNode {
    readonly $container: StepCS | StepCSTemp;
    readonly $type: 'ActorStepCS';
    actions: Array<ActorActionCS>
    description?: string
    isActorStep: boolean
    name?: UnrestrictedName
    postSnapshot: SnapshotPatternCS
    preSnapshot: SnapshotPatternCS
}

export interface SystemStepCS extends AstNode {
    readonly $container: StepCS | StepCSTemp;
    readonly $type: 'SystemStepCS';
    actions: Array<SystemActionCS>
    description?: string
    name?: UnrestrictedName
    postSnapshot: SnapshotPatternCS
    preSnapshot: SnapshotPatternCS
}

export interface RejoinStepCS extends AstNode {
    readonly $container: StepCS | StepCSTemp;
    readonly $type: 'RejoinStepCS';
    condition?: SnapshotPatternCS
    description?: string
    rejoinTo: UnrestrictedName
}

export interface UCStepCS extends AstNode {
    readonly $container: StepCS | StepCSTemp;
    readonly $type: 'UCStepCS';
    description?: string
    includedUC: Reference<UsecaseCS>
    name?: UnrestrictedName
}

export interface AltFlowCS extends AstNode {
    readonly $container: StepCS | StepCSTemp;
    readonly $type: 'AltFlowCS';
    altStep: StepCS
    baseStep: UnrestrictedName
    condition: SnapshotPatternCS
    description?: string
    name?: UnrestrictedName
}

[msujew]

Hey @LockedAwya,

at least with the grammar you have supplied us, I can't reproduce the issue (see this playground link). Langium seems to parse the stepCS2=RejoinStepCS assignment as expected:

The issue is likely somewhere else. Can you take a look at the output window of your language server in vscode? It might print a warning about grammar ambiguity, which could explain the behavior you're seeing. If anything, this is related to the parser part of Langium, not the generated code, which are just interface definitions that get erased at runtime anyway.

[idiosyncrasy00]

Yes, the grammar ambiguity exists:

Ambiguous Alternatives Detected: <0, 1> in <OPTION1> inside <UsecasePreconditionCS​> Rule,
<description, =, STRING, IDENTIFIER, :> may appears as a prefix path in all these alternatives.
See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#AMBIGUOUS_ALTERNATIVES
For Further details.
Ambiguous Alternatives Detected: <0, 1> in <OPTION3> inside <ObjVarCS​> Rule,
<IDENTIFIER, ;, IDENTIFIER> may appears as a prefix path in all these alternatives.
See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#AMBIGUOUS_ALTERNATIVES
For Further details.
Ambiguous Alternatives Detected: <0, 1> in <OPTION1> inside <UsecasePostconditionCS​> Rule,
<description, =, STRING, IDENTIFIER, :> may appears as a prefix path in all these alternatives.
See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#AMBIGUOUS_ALTERNATIVES
For Further details.
Ambiguous Alternatives Detected: <0, 1> in <OPTION3> inside <ObjVarCS​> Rule,
<Real, =, STRING> may appears as a prefix path in all these alternatives.
See: https://chevrotain.io/docs/guide/resolving_grammar_errors.html#AMBIGUOUS_ALTERNATIVES

Update and add relevant parts, fyi:

ObjVarCS:
	(isMatched?='$')?name=UnrestrictedName ':'
	(anytype=UnrestrictedName ('('anytype1=UnrestrictedName')')?)?
	(type=[ClassCS:QualifiedName])?
;

UsecasePreconditionCS:
	'ucPrecondition'
		('description' '=' description=STRING)?
		snapshot=SnapshotPatternCS
	'end'
;

UsecasePostconditionCS:
	'ucPostcondition'
		('description' '=' description=STRING)?
		snapshot=SnapshotPatternCS
	'end'
;

SnapshotPatternCS:
	('snapshotID' '=' name=UnrestrictedName)?
	('description' '=' description=STRING)?
	//objects+=ObjVarCSWithClassType | objects+=ObjVarCSWithPrimitiveTypes |
	( (objects+=ObjVarCS) ';')*
	(links+=VarLinkCS ';')*
		(constraints+=ConstraintCS)*
;

But the given warnings seem to have nothing to do with the aforementioned issue.
P/s: I also add more information in UsecaseCS definition in the first post, you can take a look on it.

[msujew]

Can you just link a reproducible example somewhere? It's kind of hard to make a whole grammar using the snippets you provided with missing terminals and so on. I can't think of a reason why this is happening, given that the grammar that you posted initially works as expected.

[idiosyncrasy00]

Sure thing. The editor works fine but I don't know where are the ambiguity rules from and if it is a referencing issue. This could lead to the adversity of extracting textual information from the content.

Link to the example

P/s: link was updated. I cannot see the syntax tree in the playground.

[msujew]

You're missing an entry rule, but otherwise it seems to work as expected. Link

[idiosyncrasy00]

@msujew I just notice a reference complexity in the code thanks to the syntax tree in your playground. The syntax tree does not treat stepCS1, stepCS2 and stepCS3 as the same level, instead each latter declared component is treated as a nested component inside the previous declaration.

The context grammar is:

(stepCS1=UCStepCS | stepCS2=RejoinStepCS | stepCS3=ActStepCS)
(altFlows+=AltFlowCS)*
(nextStep=StepCSTemp)?

That is for example in the content:

usecase ProcessSale
	inclStep HandlePayment
	end
	actStep s1 from
	to
	end
	rejoinStep s1 when
	end
	actStep s4 from
	to
	end
end

The following syntax tree for firstStep is:

{
    firstStep: {
        $type: "StepCS",
        stepCS1: {
            $type: "UCStepCS",
            includedUC: Reference('/elements@1')
        },
        nextStep: {
            $type: "StepCSTemp",
            stepCS3: {
                $type: "ActorStepCS",
                isActorStep: true,
                name: "s1"
            },
            nextStep: {
                $type: "StepCS",
                stepCS2: {
                    $type: "RejoinStepCS",
                    rejoinTo: "s1"
                },
                nextStep: {
                    $type: "StepCSTemp",
                    stepCS3: {
                        $type: "ActorStepCS",
                        isActorStep: true,
                        name: "s4"
                    }
                }
            }
        }
    }
}

If I reorder the content a little bit, that is:

usecase ProcessSale
	description = '...'		
	primaryActor = Customer
	actStep s1 from
	to
	end
	rejoinStep s1 when
	end
	inclStep HandlePayment
	end
	actStep s4 from
	to
	end
end

The following syntax tree for firstStep is modified as well:

{
    firstStep: {
        $type: "StepCS",
        stepCS3: {
            $type: "ActorStepCS",
            isActorStep: true,
            name: "s1"
        },
        nextStep: {
            $type: "StepCSTemp",
            stepCS2: {
                $type: "RejoinStepCS",
                rejoinTo: "s1"
            },
            nextStep: {
                $type: "StepCS",
                stepCS1: {
                    $type: "UCStepCS",
                    includedUC: Reference('/elements@1')
                },
                nextStep: {
                    $type: "StepCSTemp",
                    stepCS3: {
                        $type: "ActorStepCS",
                        isActorStep: true,
                        name: "s4"
                    },
                }
            }
        }
    }
}

I wonder if there is a way to resolve the "nesting problem"?

[msujew]

Right, that is kind of nesting in your AST expected, given that there's no indicator for nesting in your language. Usually, DSL/programming languages define some sort of delimiter to identify when a nested element is supposed to start/stop. Usually something like parenthesis, braces or brackets ((), {} or []) works, although indentation can also work (Python/YAML).

Langium discards all whitespace tokens by default, but it's still possible to build a language that uses whitespace sensitive indentation. For that you will need to override the TokenBuilder service and implement something like the Chevrotain Python Tokenizer example. You need to use that special indent/dedent token in your grammar to signal to the grammar that it should only nest elements if the parser reads the indent token, and that it should end the nesting once it reads the dedent token.

I'm wondering right now whether we should have some additional functionality in the framework that automatically creates INDENT/DEDENT tokens based on decorator information. I've created #1016 to track this feature.

[idiosyncrasy00]

@msujew I figured out to my own references issue. The recursive statement at (nextStep=StepCSTemp)? which made the AST at firstStep nested. Simply deleting it and modifying several relating parts solved the problem. The ambiguity have nothing to do with this tho.

Changing from this:

(stepCS1=UCStepCS | stepCS2=RejoinStepCS | stepCS3=ActStepCS)
(altFlows+=AltFlowCS)*
(nextStep=StepCSTemp)?

AltFlowCS:
	'altStep' (name=UnrestrictedName)? 'at' baseStep=UnrestrictedName
		('description' '=' description=STRING)?
		'when'
			condition=SnapshotPatternCS
		'end'
		altStep=StepCS
	'end'
;

To this:

(stepCS1+=UCStepCS | stepCS2+=RejoinStepCS | stepCS3+=ActStepCS)* //(* or + is acceptable)
(altFlows+=AltFlowCS)*

AltFlowCS:
	'altStep' (name=UnrestrictedName)? 'at' baseStep=UnrestrictedName
		('description' '=' description=STRING)?
		'when'
			condition=SnapshotPatternCS
		'end'
		altStep+=StepCS
	'end'
;

Note that AltFlowCS with statement altStep+=StepCS still gets the recursion and if digging deeper, there are still more repeated errors but it is not a big deal for me. Thank you for your cooperation. Much appreciated.

Link to the fixed code