RawString versus Interpreted String

[Zannith]

Does anyone have a suggestion for creating a RawString and InterpretedString rule?
I did have some terminals

terminal DOUBLE_STRING returns string: '"' /(\\.|[^"\\\r\n])*/ '"';
terminal SINGLE_STRING returns string: "'" /(\\.|[^'\\\r\n])*/ "'";
terminal RAW_STRING returns string: '`' /([^`])*/ '`';

But I found that when the rule returns string. That content gets automatically interpreted which made it so that I could not differentiate the literal newline characters in raw_string from actual newlines. The value returned would always be '\n'.
I was thinking trying to implement strings through an array or characters or something but i'm coming up empty on how to implement this. Does anyone have a solution or run across this?

[msujew]

Hey @Zannith,

we have some value converter code here that might be the source of the confusion. You can override the service and let it return the input directly without any manipulation.

However, I'm a bit confused. Using your RAW_STRING terminal, everything works as expected for me. See this playground example. It still retains the \n that exists in the string.

[Zannith]

So here's a typescript example of what i'm trying to do.

    const john = String.raw`Jo\nhn`
    const john2 = String.raw`Jo
hn`
    const john3 = "Jo\nhn"
    console.log(john) // Value Jo\\nhn
    console.log(john2) // Value Jo\nhn
    console.log(john3) // Value Jo\nhn

I want to be able to perform a similar behavior, however by the time that string value enters the value converter it has already been evaluated to text input of "Jo\nhn" for all of the inputs, so I cannot differentiate the characters.
Console output:

Jo\nhn
Jo
hn
Jo
hn

Here's an about equivalent playground example.

[msujew]

I'm sorry, but I fail to follow what is wrong - the playground model looks fine to me (see the tree model view):

The second value is a true newline, which is why there's a whitespace in the model value. Both of the others have the \n as a string embedded within them.

[Zannith]

Hm that does appear close to what I want. I appreciate your help with this.
I'm not able to replicate that in my implementation though. I setup a new project to replicate with devcontainer. Check the parsing.test.ts. At least seeing that tree view shows there should be some way to differentiate the values. (There is a value converter added, but it's not really doing anything)

Here is my test running

And here is the test repository

[msujew]

Uhm, you're using \n in the test file, which results in a newline being inserted. If you want to insert \n as a string, use \\n. Note that there is a difference between something like parse(`...`) and String.raw` ` (the parenthesis, which results in different TypeScript compiler behavior).

[Zannith]

Yea you're right the test file was flawed, I think that had thrown me off. I did figure out my solution through the value converter though.

protected runConverter(rule: GrammarAST.AbstractRule, input: string, cstNode: CstNode): ValueType {
        switch (rule.name.toUpperCase()) {
            case 'RAW_STRING':
                // Remove backticks, return as-is
                return input.slice(1, -1)
            case 'SINGLE_STRING':
                // Remove single quotes and interpret escapes
                return convertString(input);
            case 'DOUBLE_STRING':
                // Remove double quotes and interpret escapes
                return convertString(input);
        }
        return super.runConverter(rule, input, cstNode);
    }

Just modified the convertString to handle unicode and hexcode escapes as well. Turns out I was overcomplicating