Concise and elegant ParserTests in Langium ???

[thomasbaar]

Dear all,

in Xtext, writing compact parser tests is easy.
For example, for the default langium grammar (Person-Greeting), the tests could look like:

	@Inject extension ParseHelper<Model>
	@Inject extension ValidationTestHelper

	@Test
	def void testSuccess() {
		'person Alice Hello Alice!'
		.parse.assertNoErrors
	}

	@Test
	def void testFailure() {
		'person Bob Hello Alice!'
		.parse
		.assertError(L03HwDSLPackage::eINSTANCE.greeting, Diagnostic.LINKING_DIAGNOSTIC)
	}

Could somebody give me a hint how I can write similar tests in Langium?
Is there something comparable to ParseHelper and ValidationTestHelper?

Thanks in advance,
Thomas

[msujew]

Hey @thomasbaar,

there is actually a very similar concept in Langium. We export a parseHelper and validationHelper function from langium/test. You can see that here.

[thomasbaar]

Dear Mark,

thanks for your fast reply. In the meantime, I managed successfully to make some parser tests run, but there is one open question left.

So, what to do to have parser tests for a standard hello-world project? First, you have to extend package.json slightly:

{
  ...
  "scripts": { ...
     "test": "vitest",
    ...}
  ...
  "devDependencies": {...
    "vitest": "~0.32.0",
    ...}

Furthermore make sure that your dependencies "langium" and "langium-cli" are the most recent (1.2.0), since I had problems with 1.1.0.

As a second step, I added the file PRJHOME/vite.config.ts, but I am unsure whether this step is really necessary:

import { defineConfig } from 'vitest/config';

export default defineConfig({
    test: {
        coverage: {
            provider: 'c8',
            reporter: ['text', 'html'],
            include: ['src'],
            exclude: ['**/generated'],
        },
        deps: {
            interopDefault: true
        }
    }
});

The main step is to create a folder PRJHOME/test and to add a file parser.test.ts to it:

import { clearDocuments, expectNoIssues, expectWarning, parseHelper, validationHelper } from 'langium/test';
import { createHelloWorldServices } from '../src/language-server/hello-world-module';
import { Model, Person } from '../src/language-server/generated/ast';
import { EmptyFileSystem } from 'langium/lib/workspace';
import { afterEach, describe, test} from 'vitest';



const services = createHelloWorldServices(EmptyFileSystem);
const parse = parseHelper(services.HelloWorld);
const locator = services.HelloWorld.workspace.AstNodeLocator;
const validate = validationHelper<Model>(services.HelloWorld);


describe('Successful parser tests', () => {
  // necessary, otherwise test runs can interfere!!
  afterEach(() => {
    clearDocuments(services.HelloWorld);
  });


  test('Single declaration', async () => {
    const validation = await validate(`
       person Anna 
      `);
    expectNoIssues(validation);
  });

  test('Single declaration and single reference', async () => {
    const validation = await validate(`
     person Anna Hello Anna!
    `);
    expectNoIssues(validation);
  });

  test('Multiple declarations and multiple references', async () => {
    const validation = await validate(`
   person Anna 
   Hello Peter!
   person Bob 
   person Peter 
   Hello Anna!
   Hello Anna!
  `);
    expectNoIssues(validation);
  });
});





describe('Unsuccessful parser tests', () => {
  // necessary!!
  afterEach(() => {
    clearDocuments(services.HelloWorld);
  });

  test('Declaration with lowercase name', async () => {
    const validation = await validate(`
     person anna person Peter
    `);
    // walk through parsed model to determine culprit of warning
    const person = (validation.document.parseResult.value.persons[0] as Person);
    expectWarning(validation, 'Person name should start with a capital.', {
      node: person,
      property: 'name'
  });
  });


  // TODO: Here we should get a cross-reference error!
  // How to express this expectation in the test?
  test('Wrong Reference', async () => {
    const validation = await validate(`
     person Anna 
     Hello Peter!
    `);
    expectNoIssues(validation); // TBC
  });
});

Note that you can cluster your tests with the describe() function.
The test of my first cluster are all successful. The second cluster should check,
whether validators are run as expected or that other errors are caught.

Here I come to my last open problem: In the very last test, the cross-reference in
Hello Peter! fails, since person Peter has not been declared. How do express
this expectation in the test?

A last point:
Is there anybody out there who can formulate the three successful tests more compact?
It is not really a big issue, since these tests are already quite readable. But in Xtext I could
express the content of each test by just one line: input.parse.assertNoIssues ...

Best
Thomas

[msujew]

Here I come to my last open problem: In the very last test, the cross-reference in Hello Peter! fails, since person Peter has not been declared. How do express this expectation in the test?

You can access the reference to the person on the Hello object and confirm that the error property is set, or that the ref property is undefined.

Is there anybody out there who can formulate the three successful tests more compact?

You can wrap everything in a function, something like the following that you can use in your test function:

expectSuccessfulParse(text: string): Promise<void> {
  const validation = await validate(text);
  expectNoIssues(validation);
}

Or even more succinct:

testParse(name: string, text: string): void {
  test(name, async () => {
    const validation = await validate(text);
    expectNoIssues(validation);
  }
}

// used like this:
describe('...', () => {
  testParse('Single declaration', 'person Anna');
}

Compared to the very static nature of Java/JUnit, you can be very "functional" in how you define your test cases.

[thomasbaar]

Here I come to my last open problem: In the very last test, the cross-reference in Hello Peter! fails, since person Peter has not been declared. How do express this expectation in the test?

You can access the reference to the person on the Hello object and confirm that the error property is set, or that the ref property is undefined.

Ok, thanks! The following code works for me:

test('Wrong Reference', async () => {
    const validation = await validate(`
     person Anna 
     Hello Peter!
    `);
    const greeting = (validation.document.parseResult.value.greetings[0] as Greeting);
    // reference starting at greeting object is not defined and has error-property
    expect(greeting.person).toBeUndefined
    expect(greeting.person.error).toBeTruthy
  });

[msujew]

Uhm, I'm quite sure that this test doesn't do what you think it does. You're missing the function call parenthesis for toBeUndefined. Also, person is supposed to be defined, while person.ref is supposed to be undefined. ref is the actually referenced object, while person is the reference itself.

[thomasbaar]

You're missing the function call parenthesis for toBeUndefined.

You are right, another habit survived from working with xtend. In xtend, a function call x.m() can be abbreviated as x.m since m() does not have any parameter. Here, x.toBeUndefined() means the function call where x.toBeUndefined represents the function itself.
If one forgets the parenthesis, the test always succeeds.

Also, person is supposed to be defined, while person.ref is supposed to be undefined. ref is the actually referenced object, while person is the reference itself.

Actually, this is something that takes time to get used to :-( The traditional navigation through the model (as known from xtext, OCL, etc.) does not work any longer. For example, in order to access from a Greeting-object to the name of the referenced person, I would traditionally write o.person.name, while now I have to write instead o.person.ref?.name
The next test illustrates this:

  test('Reference object has correct name', async () => {
    const validation = await validate(`
     person Anna 
     Hello Anna!
    `);
    const greeting = (validation.document.parseResult.value.greetings[0] as Greeting);
    expect(greeting.person.ref?.name).toEqual("Anna") // note person.ref refers to Person-object
  });

Uhm, I'm quite sure that this test doesn't do what you think it does.

Finally, I have fixed the above test.

test('Wrong Reference', async () => {
    const validation = await validate(`
     person Anna 
     Hello Peter!
    `);
    const greeting = (validation.document.parseResult.value.greetings[0] as Greeting);
    expect(greeting.person.ref).toBeUndefined()
    expect(greeting.person.error).toBeDefined()
    expect(greeting.person.error?.message).toEqual("Could not resolve reference to Person named 'Peter'.")
  });