Formatting SL_COMMENT nodes

[shutterfreak]

I'm writing a formatter which extends AbstractFormatter and I am able to format most nodes. The integration in the vscode extension is straightforward.

Where I'm stuck, is in formatting the placement of hidden SL_COMMENT CST nodes.

For instance, I have the following simplified grammar:

grammar Graph

entry Model:
    (styles+=Style)*
;


fragment StyleRef:
    ':' styleref=[Style:ID]
;

Style:
    'style'
    StyleRef?
	name=ID
	definition=StyleBlock
;

StyleBlock:
    (
        '{}'
    ) | (
        '{'
        ';'* items+=StyleDefinition?
        (';'+ items+=StyleDefinition)* ';'*
        '}'
    )
;

StyleDefinition:
    topic = ID
    ':'
    value = ( STRING | ID )
;

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal STRING: /"(\\.|[^"\\])*"|'(\\.|[^'\\])*'/;
hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

Here's an input file written according to this grammar:

style yesno { // Generic yes/no
lineWindth: "1pt"
}
style :yesno yes { // Yes
lineColor: green;
labelText: "yes";
}
style :yesno no { // No
lineColor: red;
labelText: "no";
}

I want the formatter to place the SL_COMMENT nodes that follow the opening brace '{' keyword token to a new line, and indent it according to the nesting level, as in the following "ideal" output example:

style yesno {
    // Generic yes/no
    lineWindth : "1pt" ;
}
style:yesno yes {
    // Yes
    lineColor : green ;
    labelText : "yes" ;
}
style:yesno no {
    // No
    lineColor : red ;
    labelText : "no" ;
}

I know how to do it for the StyleDefinition AST nodes, but I can't get the SL_COMMENT nodes to be formatted, even though I can detect when a SL_COMMENT node is preceded by a '{' token node.

Here's my overly verbose attempt at writing a formatComments() method which is called in the format() method for specific AST node types:

private formatComments(node: AstNode): void {
    // Log the start of comment formatting for the current node.
    console.log(`formatComments(${node.$type} ${isNamed(node) ? `"${node.name}"` : '(unnamed)'})`);

    // Check if the node has a Concrete Syntax Tree (CST) node.
    if (node.$cstNode) {
        // Log the container information of the CST node.
        console.log(
            `formatComments(${node.$type} ${isNamed(node) ? `"${node.name}"` : '(unnamed)'}) -- container: ${
                node.$cstNode.container
                    ? `length: ${node.$cstNode.container.content.length}, Grammar: [ $containerProperty: ${node.$cstNode.container.grammarSource?.$containerProperty ?? '(not defined)'}, $containerIndex: ${node.$cstNode.container.grammarSource?.$containerIndex ?? '(not defined)'} ]`
                    : '(not defined)'
            }`,
        );

        // Stream all CST nodes from the current node's CST.
        const cstNodes = CstUtils.streamCst(node.$cstNode).toArray();

        // Filter out single-line comment (SL_COMMENT) nodes (they are hidden) which belong to the current container.
        const commentNodes = cstNodes.filter(
            (cstNode) =>
                isLeafCstNode(cstNode) &&
                cstNode.hidden &&
                cstNode.tokenType.name === 'SL_COMMENT' &&
                cstNode.container === node.$container,
        );

        // Log the number of comment nodes found for the current node.
        console.log(
            `formatComments(${node.$type} ${isNamed(node) ? `"${node.name}"` : '(unnamed)'}) -- formatting comments for ${node.$type} ${
                isNamed(node) ? `"${node.name}"` : '(unnamed)'
            }, count: ${commentNodes.length}`,
        );

        // Iterate through each comment node.
        commentNodes.forEach((commentNode) => {
            // Get the parent AST node of the comment.
            const parent = commentNode.container?.astNode;

            // Log the comment text and its parent AST node type and name.
            console.log(
                `formatComments(${node.$type} ${isNamed(node) ? `"${node.name}"` : '(unnamed)'}) -- comment: [[${commentNode.text.replaceAll('\r', '\\r').replaceAll('\n', '\\n')}]] parent: ${
                    parent?.$type
                } ${parent?.$cstNode?.astNode && isNamed(parent.$cstNode.astNode) ? `"${parent.$cstNode.astNode.name}"` : '(unnamed)'}`,
            );

            // Get the parent CST node of the comment.
            const parentCstNode = commentNode.container;

            // Find the CST node immediately before the comment node.
            const previousNode = parentCstNode?.content.find(
                (child) => child.end === commentNode.offset,
            );

            // Check if the previous node is a '{' token.
            if (previousNode && isLeafCstNode(previousNode) && previousNode.tokenType.name === '{') {
                // Log that the comment is immediately after '{' and will be moved to a new line.
                console.log(
                    `formatComments(${node.$type} ${isNamed(node) ? `"${node.name}"` : '(unnamed)'}) -- Moving inline comment after { to a new line: ${commentNode.text}`,
                );

                // Log the CST content before formatting.
                console.log(
                    render_text(
                        inspect(
                            parentCstNode?.content.map((c) => ({
                                text: c.text,
                                range: c.range,
                                offset: c.offset,
                            })),
                        ),
                        `formatComments(${node.$type} ${isNamed(node) ? `"${node.name}"` : '(unnamed)'}) -- Before formatting`,
                    ),
                );

                // Add a newline and indent before the comment node.
                this.getNodeFormatter(node)
                    .cst([commentNode])
                    .prepend(Formatting.newLine())
                    .prepend(Formatting.indent());

                // Log the CST content after formatting.
                console.log(
                    render_text(
                        inspect(
                            parentCstNode?.content.map((c) => ({
                                text: c.text,
                                range: c.range,
                                offset: c.offset,
                            })),
                        ),
                        `formatComments(${node.$type} ${isNamed(node) ? `"${node.name}"` : '(unnamed)'}) -- After formatting`,
                    ),
                );
            }
            // Add an indent before the comment node.
            this.getNodeFormatter(node).cst([commentNode]).prepend(Formatting.indent());
        });
    }
}

Is this possible with the Langium Formatter?

[msujew]

Hey @shutterfreak,

You might want to take a look at the createHiddenTextEdits method that performs the formatting for all comment-like cst nodes.

[shutterfreak]

Thanks! I created a custom override of createHiddenTextEdits() which runs (debug log statements) but the edits are not implemented in the formatted document. Is there a specific call I need to add to have these edits executed? Do I need to register another service to my module?

[msujew]

The edits returned from createHiddenTextEdits should be applied, unless they are in conflict with other edits and filtered out during avoidOverlappingEdits. Can you check whether your edits are removed in there?

[shutterfreak]

Thanks again!

I created a debug version of avoidOverlappingEdits() and I now understand what is happening.

1. The Langium formatter generates an array of edits after running the format() method for every AST node, which in turn typically calls custom format methods for different AST node types.

2. After complete AST traversal, createHiddenTextEdits() is called to format hidden tokens (custom implementation required)

3. Lastly, avoidOverlappingEdits() will:

   1. review all edits and check for editing conflicts.
   2. filter the reviewed edits with isNecessary() to keep only the necessary edits

The problem is that the edits refer to ranges in the unmodified document, whereas the document is being modified through the edits. The Range objects eventually have stale references due to earlier modifications.

I suppose that a way to fix this problem in a generic way, would be to keep track of changes in ranges resulting from earlier edits.

How are TextEdit edits applied?

[msujew]

The problem is that the edits refer to ranges in the unmodified document, whereas the document is being modified through the edits. The Range objects eventually have stale references due to earlier modifications.
I suppose that a way to fix this problem in a generic way, would be to keep track of changes in ranges resulting from earlier edits.

That should not be necessary. The formatter should return the edits as independent edits, which are then applied collectively on the client. I.e. you don't need to adjust the text edits based on previous edits yourself in the language server. Doing so will definitely result in unexpected results from the formatting.

[shutterfreak]

This is the debug log heavy implementation I used:

  protected override avoidOverlappingEdits(
    textDocument: TextDocument,
    textEdits: TextEdit[],
  ): TextEdit[] {
    console.log(`avoidOverlappingEdits() - textEdits: ${textEdits.length}`);

    const edits: TextEdit[] = [];
    let i = 0;
    for (const edit of textEdits) {
      console.log(
        `avoidOverlappingEdits() - processing textEdit[${i}]: edits: ${edits.length}, 🔎 edit: range: ${rangeToString(edit.range)}, newText: ${JSON.stringify(edit.newText)}`,
      );
      let last = edits[edits.length - 1];
      while (edits.length > 0) {
        const currentStart = textDocument.offsetAt(edit.range.start);
        const lastEnd = textDocument.offsetAt(last.range.end);
        if (currentStart < lastEnd) {
          console.log(
            `avoidOverlappingEdits() - textEdit[${i}]: edits: ${edits.length}, edit currentStart: ${currentStart}, lastEnd: ${lastEnd} -- currentStart < lastEnd: ❌ popping last edit`,
          );
          edits.pop();
          last = edits[edits.length - 1];
        } else {
          console.log(
            `avoidOverlappingEdits() - textEdit[${i}]: edits: ${edits.length}, edit currentStart: ${currentStart}, lastEnd: ${lastEnd} -- currentStart >= lastEnd: ✅ we can push the current edit`,
          );
          break;
        }
      }
      edits.push(edit);
      console.log(
        `avoidOverlappingEdits() - textEdit[${i}]: 👍 edit pushed to edits, edits: ${edits.length}`,
      );
      i++;
    }
    console.log(`avoidOverlappingEdits() - textEdit[${i}]: edits: ${edits.length}`);
    const necessaryEdits = edits.filter((edit) => this.isNecessary(edit, textDocument));
    console.log(
      `avoidOverlappingEdits() - textEdit[${i}]: necessary edits: ${necessaryEdits.length} -- will be returned`,
    );
    necessaryEdits.forEach((edit, index) =>
      console.log(
        `avoidOverlappingEdits() :: Edit ${`    ${index}`.slice(-4)}: ${rangeToString(edit.range)} : ${JSON.stringify(edit.newText)}`,
      ),
    );
    return necessaryEdits;
  }
``

[shutterfreak]

Here's my implementation which takes care of placing a SL_COMMENT that is right after a '{' node on a new line, properly indented (note: the code should also be applicable to multi-line ML_COMMENT nodes (untested)):

  /**
   * Creates hidden text edits for single-line comments after an opening brace.
   * Adjusts indentation and adds newlines as needed.
   *
   * @param previous The previous CST node.
   * @param hidden The hidden CST node (single-line comment).
   * @param formatting The formatting action.
   * @param context The formatting context.
   * @returns An array of TextEdit objects.
   */
  protected override createHiddenTextEdits(
    previous: CstNode | undefined,
    hidden: CstNode,
    formatting: FormattingAction | undefined,
    context: FormattingContext,
  ): TextEdit[] {
    console.log(`createHiddenTextEdits("${hidden.text}")`);

    if (
      isLeafCstNode(hidden) &&
      hidden.tokenType.name === 'SL_COMMENT' &&
      isLeafCstNode(previous) &&
      previous.tokenType.name === '{'
    ) {
      // Custom logic for SL_COMMENT comments after '{'
      const startLine = hidden.range.start.line;

      // Calculate the start range for the new text edit
      let startRange: Range | undefined = undefined;
      if (isLeafCstNode(previous) && previous.range.end.line === startLine) {
        // Hidden node is on the same line as its previous node

        startRange = {
          start: previous.range.end,
          end: hidden.range.start,
        };
      } else {
        // Not on same line
        startRange = {
          start: {
            character: 0,
            line: startLine,
          },
          end: hidden.range.start,
        };
      }

      const edits: TextEdit[] = [];

      // Calculate the expected indentation
      const hiddenStartText = context.document.getText(startRange);
      const move = this.findFittingMove(startRange, formatting?.moves ?? [], context);
      const hiddenStartChar = this.getExistingIndentationCharacterCount(hiddenStartText, context);
      const expectedStartChar = this.getIndentationCharacterCount(context, move);

      const characterIncrease = expectedStartChar - hiddenStartChar;

      if (characterIncrease === 0) {
        return [];
      }

      let newText = '\n';
      if (characterIncrease > 0) {
        newText = newText + (context.options.insertSpaces ? ' ' : '\t').repeat(characterIncrease);
      }

      const lines = hidden.text.split('\n');
      lines[0] = hiddenStartText + lines[0];
      for (let i = 0; i < lines.length; i++) {
        const currentLine = startLine + i;
        if (characterIncrease > 0) {
          const textEdit: TextEdit = {
            newText,
            range: {
              start: startRange.start,
              end: startRange.end,
            },
          };
          console.log(
            `createHiddenTextEdits("${hidden.text}") -- SL_COMMENT after '{' token -- characterIncrease: ${characterIncrease} > 0 -- textEdit: range: ${rangeToString(textEdit.range)}, newText: ${JSON.stringify(textEdit.newText)}`,
          );

          edits.push(textEdit);
        } else {
          const currentText = lines[i];
          let j = 0;
          for (; j < currentText.length; j++) {
            const char = currentText.charAt(j);
            if (char !== ' ' && char !== '\t') {
              break;
            }
          }

          const textEdit: TextEdit = {
            newText: '\n',
            range: {
              start: {
                character: startRange.start.character, // Was 0
                line: currentLine,
              },
              end: {
                line: currentLine,
                // Remove as much whitespace characters as necessary
                // In some cases `characterIncrease` is actually larger than the amount of whitespace available
                // So we simply remove all whitespace characters `j`
                character: startRange.start.character + Math.min(j, Math.abs(characterIncrease)),
              },
            },
          };
          console.log(
            `createHiddenTextEdits("${hidden.text}") -- SL_COMMENT after '{' token -- characterIncrease: ${characterIncrease} < 0 -- textEdit: range: ${rangeToString(textEdit.range)}, newText: ${JSON.stringify(textEdit.newText)}`,
          );
          edits.push(textEdit);
        }
      }

      return edits;
    }
    console.log(
      `createHiddenTextEdits("${hidden.text}") -- NOT [ SL_COMMENT after '{' token ] -- will call super.createHiddenTextEdits()`,
    );

    // Call the default implementation for other cases
    return super.createHiddenTextEdits(previous, hidden, formatting, context);
  }

[goto40]

Thank you for sharing this code snippet.
We still have problems understanding the logic behind - we have also some issues formatting comments. Are there any plans for an enhanced formatting API, making this easier?

Thank you.