Replies: 9 comments 34 replies
-
UAN Logical Instruction SetModel: UAN FormatDescription
Partial GrammarUANBlock:
category+=UANCategory*
;
UANCategory:
category=
(
'CATEGORY01' | 'CATEGORY02' |
'CATEGORY03' | 'CATEGORY04' |
'CATEGORY05' | 'CATEGORY06'
) user_names+=UANNames*
;
UANNames:
name=ID
;Partial Validatorexport class LoraxDslValidator {
...
checkUANNamesIsLowerCase(category: UANNames, accept: ValidationAcceptor): void {
if (category.name) {
const name = category.name;
if (name.toLocaleLowerCase() !== name) {
accept('warning', 'UAN names should be lowercase.', { node: category, property: 'name' });
}
}
}
...
}Tag: @msujew This block seems to behave as expected. Mostly verifying the overall approach is sound because this general approach will be used for the rest of the implementation. |
Beta Was this translation helpful? Give feedback.
-
Scope NarrowingTag: @msujew I'm wondering if I'm missing a more idiomatic implementation here ... basically, the question is "what's the correct way to implement scope narrowing?" The scope computation I implemented is purposefully simple but that comes at the expense of broadening the scoped visibility. Fundamentally, I'm asking where is the best place to include the scoping complexity: within the grammar (more explicit), within the scope provider (language developer burden), or something else I'm missing completely (e.g. It's easier to explain with a simple example (toy repo commit). In this case, I narrowed the scope using a Example TextSourceEXECUTE
UAN-DEFINITION
TOOLS
hammer // var declarations
screwdriver
pliers
BOOKS
book1 // var declarations
book2
CARDS
ace // var declarations
queen
END UAN
SDB
TOOL-USAGE:
DESC: pliers QTY 3 // var references
DESC: hammer QTY 7
BOOK-USAGE:
DESC: book1 QTY 2 // var references
DESC: book2 QTY 1
CARD-USAGE:
DESC: ace QTY 1 // var references
DESC: queen QTY 0
END SCENARIO
END-INSTRUCTIONSScope Provider SnippetSourceoverride async computeExports(document: LangiumDocument): Promise<AstNodeDescription[]> {
const exportedDescriptions: AstNodeDescription[] = [];
for (const node of streamAllContents(document.parseResult.value)) {
if (isUANItem(node)) {
exportedDescriptions.push(this.descriptions.createDescription(node, node.name, document));
}
}
return exportedDescriptions;
}Original Grammar SnippetSourceUANCategory:
{infer Books} name='BOOKS' (items+=UANItem ('ref:' ref=[UANItem])?)+
| {infer Cards} name='CARDS' (items+=UANItem ('ref:' ref=[UANItem])?)+
| {infer Tools} name='TOOLS' (items+=UANItem ('ref:' ref=[UANItem])?)+
;
UANItem:
name=ID
;
SDBBlock:
name=SDBBlockHeader
'RANDOM-NUMBER-SEED:' seed=INT
toolUsage=ToolUsageBlock?
'END' 'SCENARIO'
;
ToolUsageBlock:
name='TOOL-USAGE:'
( 'DESC:' tool=[UANItem] 'QTY' qty=INT)*
;Narrowed Grammar SnippetSourceUANCategory:
{infer Books} name='BOOKS' (books+=UANBook)+
| {infer Cards} name='CARDS' (cards+=UANCard)+
| {infer Tools} name='TOOLS' (tools+=UANTool)+
;
type UANItem = UANBook | UANCard | UANTool;
UANBook:
name=ID
;
UANCard:
name=ID
;
UANTool:
name=ID
;
SDBBlock:
name=SDBBlockHeader
'RANDOM-NUMBER-SEED:' seed=INT
toolUsage=ToolUsageBlock?
bookUsage=BookUsageBlock?
cardUsage=CardUsageBlock?
'END' 'SCENARIO'
;
ToolUsageBlock:
name='TOOL-USAGE:'
('DESC:' tool=[UANTool] 'QTY' qty=INT)*
;
BookUsageBlock:
name='BOOK-USAGE:'
('DESC:' book=[UANBook] 'QTY' qty=INT)*
;
CardUsageBlock:
name='CARD-USAGE:'
('DESC:' card=[UANCard] 'QTY' qty=INT)*
; |
Beta Was this translation helpful? Give feedback.
-
The question is always about what you want to scope. If you say that you just want a single special element type of your language as a possible cross reference type, then you should go with the typical cross reference grammar approach. If you're trying to limit the amount of elements available scoped elements by some other metric, you will need to implement this as a specialized scope provider. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @msujew . I just wanted to make sure there wasn't an easy button that I was missing somehow. |
Beta Was this translation helpful? Give feedback.
-
TDB Placeholder |
Beta Was this translation helpful? Give feedback.
-
System Design Block |
Beta Was this translation helpful? Give feedback.
-
|
I think I largely answered my own questions - but I'll capture my thoughts here for posterity (and reference for anyone playing along).
Testing parsing (or anything for that matter) is probably not wise against a known or possibly invalid grammar. Ask me how I know ...
I'm still not sure how this is done in practice. I would guess it's language specific. Regardless, I elected to go 'top-down' and use copious amounts of scaffolded print structures (pre-validated). It's verbose, but it works - there's probably something more clever one could do, but I'm certainly unaware.
My grammar is still incomplete, and I'm already at nearly 1k lines. I'm not sure I could minimize any appreciable amount of validation by tightening up the grammar, and it's already getting pretty unwieldy. |
Beta Was this translation helpful? Give feedback.
-
I believe we usually also go for top-down testing. For example, when performing scoping tests, we don't just test the scope provider by itself, but usually set up a test file which is basically a whole integration test.
That's a lot. I've certainly seen some grammars that large, but there was a ton of domain information embedded in that language. Is it similar for your language? |
Beta Was this translation helpful? Give feedback.
-
I suspect that is the case, although admittedly I don't have a large sample size from which to draw comparisons. However, I stumbled across a text section buried in the bowels of the binary consisting of ~1500+ lexical codes which correspond to all of the tokens (keywords). This was the motivation behind #601 since there is very little consistency in the composition of the keywords (i.e. hyphenated vs. white space).
This makes sense and is essentially the conclusion that I arrived at. By the time I get to the 'bottom' I'll be doing full integration tests. However, I will say that scoping is the one riddle I haven't solved yet. That warrants a separate question though. Perhaps this implementation will provide a good test case for scaling which hopefully can be used for optimization to the extent possible within a node.js framework. |
Beta Was this translation helpful? Give feedback.
-
Ok, that's an almost ridiculous amount of keywords/token types. In comparison, languages such as JavaScript have roughly ~50 different token types (keywords/strings/numbers/IDs). |
Beta Was this translation helpful? Give feedback.
-
Agreed. I'd like to say that the original intent was to have an oddly worded novella, but I think this is more likely what happens when engineers are left unsupervised. In the end, it is a lot of boiler plate and hopelessly overengineered. However, I don't have the luxury (or resources) to refactor the DSL itself. Perhaps a meta-DSL? :) |
Beta Was this translation helpful? Give feedback.
-
|
Question: Nested Multi-Line Comments As shown above, multi-line comments are pretty straightforward. Accordingly, the terminal rule is: hidden terminal ML_COMMENT: /\$\>[\s\S]*?\<\$/;Occasionally, there are nested comments (very annoying - but happens) and that is allowed/supported by the DSL syntax. I figured out the hard way that JavaScript doesn't support recursion |
Beta Was this translation helpful? Give feedback.
-
|
Interesting. I believe there is a clever solution using lexer modes. Be aware that they are theoretically supported, but only work as of a minor fix made in First off, you can separate the Then you'll need to slightly modify the export class CustomTokenBuilder extends DefaultTokenBuilder {
buildTokens(grammar: Grammar): TokenVocabulary {
const tokens = super.buildTokens(grammar);
const commentStart = tokens.find(token => token.name === 'ML_COMMENT_START')!;
const commentEnd = tokens.find(token => token.name === 'ML_COMMENT_END')!;
const commentContent = tokens.find(token => token.name === 'ML_COMMENT_CONTENT')!;
commentStart.PUSH_MODE = 'comment';
commentEnd.POP_MODE = true;
const nonCommentTokens = tokens.filter(token => token !== commentEnd && token !== commentContent);
const commentTokens = [commentEnd, commentContent];
return {
modes: {
comment: commentTokens,
nonComment: nonCommentTokens
},
defaultMode: 'nonComment'
};
}
}That way you'll basically build up a stack of comments which are popped, as the lexer encounters a |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @msujew. We'll definitely wait until support is part of the release ... this is just a corner case that I knew I'd have to address at some point. The proposed solution matches my intuition for how it's handled by the DSL compiler so I'm glad I'm not too far off base. |
Beta Was this translation helpful? Give feedback.
-
|
Question: Development Strategy As I think through how best to tackle this, do you have any recommendations on how to incrementally build out the capability? Ideally, I'd be able to get early prototypes that are somewhat useable into the hands of a larger audience for user feedback and testing. Given the size and scope of the DSL, the breadth of available features within VSCode, and the current capabilities of Langium do you have any recommendations on how best to iteratively and incrementally build up the LSP? There's probably a better way to ask this question, so let me know if it makes sense. |
Beta Was this translation helpful? Give feedback.
-
|
Perfect - that's exactly what I was looking for. Thanks! Regarding the cross reference resolution, using your approach of Not sure if this is the right way of inspecting it, but using a Not sure I'm reading it correctly, but |
Beta Was this translation helpful? Give feedback.
-
|
Ah, yes I forgot that we fixed that bug only after the |
Beta Was this translation helpful? Give feedback.
-
|
RE: Validation Unit Testing (Hello World) Ok, so this works ... but I'll be damned if I can explain why. What I came up with is very similar to the process that you guys use in your core library. Care to explain the magic here? If nothing else, while trivial, it's a decent example of a validator unit test I suppose. :) Grammar Validation (hello-world-validator.ts):import { ValidationAcceptor, ValidationChecks, ValidationRegistry } from 'langium';
import { HelloWorldAstType, Person } from './generated/ast';
import type { HelloWorldServices } from './hello-world-module';
/**
* Registry for validation checks.
*/
export class HelloWorldValidationRegistry extends ValidationRegistry {
constructor(services: HelloWorldServices) {
super(services);
const validator = services.validation.HelloWorldValidator;
const checks: ValidationChecks<HelloWorldAstType> = {
Person: validator.checkPersonStartsWithCapital
};
this.register(checks, validator);
}
}
/**
* Implementation of custom validations.
*/
export class HelloWorldValidator {
checkPersonStartsWithCapital(person: Person, accept: ValidationAcceptor): void {
if (person.name) {
const firstChar = person.name.substring(0, 1);
if (firstChar.toUpperCase() !== firstChar) {
accept('warning', 'Person name should start with a capital.', { node: person, property: 'name' });
}
}
}
}Grammar Validator Test (hello-world-validator-test.test.ts)import { expectWarning, validationHelper, ValidationResult } from 'langium/test';
import { Model } from '../language-server/generated/ast';
import { createHelloWorldServices } from '../language-server/hello-world-module';
const services = createHelloWorldServices().HelloWorld;
const validate = validationHelper<Model>(services);
describe('Check Names', () => {
const input = `
person Bob
person alice
person Ralph
`;
let validationResult: ValidationResult<Model>;
beforeAll(async () => {
validationResult = await validate(input);
});
test('Names Capitalized Warning', () => {
const rule = validationResult.document.parseResult.value.persons[1];
expectWarning(validationResult, 'Person name should start with a capital.',
{
node: rule,
property: {
name: 'name'
}
});
});
}); |
Beta Was this translation helpful? Give feedback.
-
With pleasure, though you might need to be a bit more specific there :) In general, we just store a list of functions (i.e. validations) that can be applied to a certain type of AST node. Once we reach the validation phase, we just run through all AST nodes in a given document and execute all validations that are registered for that type. The test utility function simulates this behavior and returns a list of all diagnostics (i.e. validation errors, warnings, etc.). Additional utilities such as the |
Beta Was this translation helpful? Give feedback.
-
|
Tag: @msujew Thanks! That fills in some of the mental gaps ... I'm slowly wrapping my head around this (I think). Understanding the subtleties of 'what and when' per the lifecycle gets tricky. I paused a bit on implementing the LORAX DSL to learn how to use somewhat properly use your framework using Implementing Domain-Specific Languages with Xtext and Xtend - Second Edition. I got about 2/3 of the way through and decided that learning Xtext/Xtend was silly, so I created a repo to track the progress using Langium instead - maybe some decent documentation will fall out of it. It's been a fruitful exercise thus far, but a few questions have come up. Questions:1) Validators
2) Quickfix
3) Outline: Symbology & Structure (UI)
Let me know if you'd like me to move this to a separate thread. Thanks! |
Beta Was this translation helpful? Give feedback.
-
As of now, the latter one. It might be interesting to implement this as a feature though.
Maybe the langium validator might help you. There are a few validations which are related to uniqueness of names:
Right now, we simply add the
None so far.
You're right. You can override the existing |
Beta Was this translation helpful? Give feedback.
-
|
Question: DocumentLinkProvider One of the great features of this DSL is its excellent documentation consisting of well-linked PDFs. I see that the protocol provides an interface for document links and I was wondering if there are any plans to implement an Given that Langium itself can take advantage of TSDoc, I'm entirely not sure there would ever be a dog food requirement to implement it. I'm pretty sure that solo'ing this is beyond my nascent skills, but I'd be more than happy to help. Here's an example from the Lean project: Linkimport {DocumentLinkProvider, TextDocument, DocumentLink, ProviderResult, Range, workspace, window} from 'vscode';
const seeNoteRegex = /note \[([^\]]+)\]/gi;
const libraryNoteRegex = /library_note "(.*)"/g;
export class LibraryNoteLinkProvider implements DocumentLinkProvider {
provideDocumentLinks(document: TextDocument): ProviderResult<DocumentLink[]> {
const links: DocumentLink[] = [];
for (const m of document.getText().matchAll(seeNoteRegex)) {
const link = new DocumentLink(new Range(
document.positionAt(m.index), document.positionAt(m.index + m[0].length)));
link.tooltip = m[1];
links.push(link);
}
return links;
}
async resolveDocumentLink(link: DocumentLink): Promise<DocumentLink> {
const noteName = link.tooltip;
for (const leanFile of await workspace.findFiles('**/*.lean')) {
const content = (await workspace.fs.readFile(leanFile)).toString();
for (const m of content.matchAll(libraryNoteRegex)) {
console.log(m[1]);
if (m[1] === noteName) {
const lineNo = content.substr(0, m.index).split(/\r\n|\r|\n/).length;
link.target = leanFile.with({ fragment: `L${lineNo}` });
return link;
}
}
}
await window.showErrorMessage(`Library note "${noteName}" not found.`);
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Yes I think eventually we want at least base interfaces for all LSP request types. I don't know whether an abstract class makes sense here, the actual implementation may be very different depending on what kind of links are in the document. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome! It's not at the top of the wish list, but it would definitely be nice to have. If I happen to figure something out in the meantime, I'd be happy to submit a PR against one of the example languages if I'm able to. |
Beta Was this translation helpful? Give feedback.
-
|
@snarkipus I implemented #688 to have this feature in 0.5. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @msujew! That will be incredibly helpful! |
Beta Was this translation helpful? Give feedback.
-
Disambiguate Parser Rules:Tags: @msujew Typically I can work my way out of these, but this one is a little more complicated. I get the dreaded Essentially, I need a
What am I missing? Thanks in advance for the help! Desired Grammar...
HorizontalDefinition: // TODO: needs a pretty complicated validator
CircularOption | PointsOption | SpehricalOption
;
CircularOption infers HorizontalDefinition:
'MIN/MAX RNG:' minRng=REAL maxRng=REAL rngUOM=XYUnits
'COUNTERCLOCKWISE' 'FROM:' from=REAL fromUOM=HDGUnits
'TO:' to=REAL toUOM=HDGUnits
;
PointsOption infers HorizontalDefinition:
('X,Y:' x=REAL y=REAL xyUOM=XYUnits)*
;
SpehricalOption infers HorizontalDefinition:
('L/L:' lat=COORDS lon=COORDS)*
;
...Passing Grammar (untested)...
HorizontalDefinition: // TODO: needs a pretty complicated validator
CircularOption | PointsOption | SpehricalOption
;
CircularOption infers HorizontalDefinition:
'MIN/MAX RNG:' minRng=REAL maxRng=REAL rngUOM=XYUnits
'COUNTERCLOCKWISE' 'FROM:' from=REAL fromUOM=HDGUnits
'TO:' to=REAL toUOM=HDGUnits
;
PointsOption infers HorizontalDefinition: // TODO: figure out a better way to structure this
'X,Y:' x=REAL y=REAL xyUOM=XYUnits
'X,Y:' x=REAL y=REAL xyUOM=XYUnits
('X,Y:' x=REAL y=REAL xyUOM=XYUnits)*
;
SpehricalOption infers HorizontalDefinition: // TODO: figure out a better way to structure this
'L/L:' lat=COORDS lon=COORDS
'L/L:' lat=COORDS lon=COORDS
('L/L:' lat=COORDS lon=COORDS)*
;
... |
Beta Was this translation helpful? Give feedback.
-
|
Hey, so basically the issue is that both |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @msujew! That resolved it (and also makes perfect sense). As an added bonus, I had the opportunity to fix the 75 other places I had misused cardinality . |
Beta Was this translation helpful? Give feedback.
-
Strange Issue - Possible Validation Idea@msujew Here's a notional case: someRule:
'foo' name=STRING
('bar' barName=STRING barID=INT)*
(bazRule)*
;
bazRule:
'bar' barName=STRING barID=INT
;Not sure there's any way to cleanly keep you from doing this ... I'll give some thought to what a validation would look like. Since |
Beta Was this translation helpful? Give feedback.
-
DSL Project Workspace - Configuration Management (off-topic)This question is probably somewhat out of scope, but it's relevant since I'm trying to resolve which files to include for scope computation purposes. Essentially, I need some way of managing the project/workspace state (notionally using a config file) since a naive approach of including all files in a particular directory won't work for my use case. On the Langium side, it seems reasonable to initialize the workspace by filtering against a config file (see diagram below). Not sure if there are any preexisting examples ... I know the Domainmodel has example code for specifying the project root. Secondarily, what is a practical way of managing this file or one that's often used in practice? The VSCode Extension API turns into a veritable rabbit hole with numerous options/contribution points (TreeView API, WebView API, Configuration API, etc.). Obviously, this is heavily use-case dependent, but I'm wondering what you guys use as a standard approach?
|
Beta Was this translation helpful? Give feedback.
-
|
So, during the workspace initialization, you can perform some filtering by overriding this method: Note that you'll likely need some more complex file handling in the language server. While the code above works for the initial indexing of the workspace, opening a file that doesn't belong to the workspace will still be sent to the language server. You'll need to handle that in some way. For vscode: We just contribute a json schema for such cases, see our package.json. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Disclaimer
Our team consists of engineers and analysts and as such we are not software or language developers; that is, collectively we have very limited practical TypeScript/Javascript experience and cursory, academic (if any) exposure to language design.
The overall intent of this thread is two-fold:
Previous Questions/Comments:
Use-Case
Create developer tooling support for an existing in-house DSL. While current developer/analyst tooling exists, it is limited to generic text editors and a legacy (unused) Emacs plug-in.
(original desc)
DSL Overview & Description
For the purpose of this exercise, we'll refer to this language/DSL as the LORAX (Langium Obvious Required Adoption eXpectations) DSL.
LORAX Language Description:
From the docs (not particularly helpful or insightful) ...
LORAX Model Structure
The first phase of LORAX modeling entails reading and checking the various types of user-generated, model input data (instructions). Specifically, these model inputs/instructions are listed as:
Global Instructions
Each file includes 'global options' at the file header/footer described as:
<logical-instruction-set>: occurs one or more times, referring to one of the following composite data items:<data-option>:SAVE-DATAorDO-NOT-SAVE-DATAPartial Grammar:
Comments - Terminal Rules
Numeration - TERMINAL Rules?
12.30..00.67-6.8E40.3e-2Partial Grammar (Terminal Rules)
Units of Measure (UOM) - Keywords?
Examples:
(SEC)(MIN)(HR)(M)(KM)(FT)(MILES)Logical Instruction Sets (Composite Data Items)
UAN, TDB, and SDB will be captured in separate threads.
LORAX LSP: Objectives & Features
Document the existing DSL grammar definition/rules/syntax and develop validation methods native to VSCode to support real-time developer/analyst feedback during analysis environment development.
NOTE: It's understood that the Langium framework may not provide all of the above features. The intent is to capture the overall list of features available within the VSCode environment and determine what is or is not something to implement within Langium (or possibly features not yet complete).
Tag: @msujew - Please feel free to comment or provide feedback at your convenience. I'll tag you in any specific questions.
Beta Was this translation helpful? Give feedback.
All reactions