feat: add variables support

[daiyam]

References:

• How about named regex? #126

{
    "variables": {
        "BeginComment1": "\\/\\*",
        "EndComment1": "\\*\\/",
    },
},
{
    "beginRegex": "{{!BeginComment1}}",
    "endRegex": "{{!EndComment1}}",
},

• {{:varname}}: the replacement is escaped
• {{!varname}}: the replacement is raw (only available for regex properties)

[daiyam]

@TonyGravagno What do you think?

[TonyGravagno]

I will merge and try this within the next couple days.

value = value.slice(0, match.index) + replacement + value.slice(match.index + length);
🚀 This is what I wanted!!!! 🚀 I'm going to have some fun with this!

🙏❤️😁

[TonyGravagno]

I see from the value replacement that a new regex should be valid
value = value.slice(0, match.index) + replacement + value.slice(match.index + length);
But when using a new variable, the resulting regex is not valid.

Good regex

        "beginRegex": "\\/\\*(?![^\\\"]*\\\"[^\\\"]*(?:\\*\\/))",
        "endRegex": "\\*\\/"

With variables

        "beginRegex": "{{:BeginCommentBlock}}(?![^\\\"]*\\\"[^\\\"]*(?:\\*\\/))",
        "endRegex": "{{:EndCommentBlock}}"

When variables aren't defined, new code correctly parses variable names.

can't find the variable: 'BeginCommentBlock'

And when the variables are defined:

        "variables": {
          "BeginCommentBlock": "\\/\\*",
          "EndCommentBlock": "\\*\\/"
        },

[document] fileName: settings.json, route: main
[document] lang: jsonc, fileName: settings.json
[main] regex: /a^/
[document] foldings: []

We can't see the new pattern (#145) but we know it's resolved.
No error is reported when the pattern is compiled.

Maybe I have the syntax wrong? I'm not referring to the code right now, just looking at the notes above where the ":" and "!" are specified as a prefix to the variable name.

[TonyGravagno]

I think it would be much better for the "variables" declaration to be higher in the settings hierarchy.

As currently implemented, "variables" must be defined within a rule at the same level as begin/end props. To re-use the variables we need to copy them to other rules. 😢
Of course very-specific regex patterns are only valid within the scope of a single rule. But many patterns are re-used in many rules.

I'm hoping for a change for this enhancement, to raise variables into the "explicitFolding.rules" scope as a reserved-word rule (like "*"). All patterns within that one rule can then be used by any other rule, including perFiles rules:

  "explicitFolding.rules": {
    // "lang1,lang2": []
    "*": [],
    "variables": {
      "BeginCommentBlock": "\\/\\*",
      "EndCommentBlock": "\\*\\/",
      "AtoZ1": "[A-Za-z0-9]+",
      "AtoZ2": "[A-Za-z_$]"
    },
    "#comments": [
      {
        "name": "Comments",
        // Beginning of the line, followed by some spaces, then "/*"
        "beginRegex": "{{:BeginCommentBlock}}(?![^\\\"]*\\\"[^\\\"]*(?:\\*\\/))",
        // Anywhere on a line with "*/"
        "endRegex": "{{:EndCommentBlock}}"
      }
    ],

[daiyam]

As currently implemented, "variables" must be defined within a rule at the same level as begin/end props. To re-use the variables we need to copy them to other rules. 😢

try:

"explicitFolding.rules": {
  // "lang1,lang2": []
  "*": [],
  "#vars": {
    "variables": {
      "BeginCommentBlock": "\\/\\*",
      "EndCommentBlock": "\\*\\/",
      "AtoZ1": "[A-Za-z0-9]+",
      "AtoZ2": "[A-Za-z_$]"
    },
  },
  "#comments": [
    { "include": "#vars" },
    {
      "name": "Comments",
      // Beginning of the line, followed by some spaces, then "/*"
      "beginRegex": "{{:BeginCommentBlock}}(?![^\\\"]*\\\"[^\\\"]*(?:\\*\\/))",
      // Anywhere on a line with "*/"
      "endRegex": "{{:EndCommentBlock}}"
    },
  ],
},

I'm hoping for a change for this enhancement, to raise variables into the "explicitFolding.rules" scope as a reserved-word rule (like "*").

I've pondered about that but if I didn't do that it because it wouldn't be able to do that:

"[typescript]": 
  "explicitFolding.rules": [
    {
      "variables": {
        "BeginCommentBlock": "\\/\\*",
        "EndCommentBlock": "\\*\\/",
        "AtoZ1": "[A-Za-z0-9]+",
        "AtoZ2": "[A-Za-z_$]"
      },
    },
    {
      "name": "Comments",
      // Beginning of the line, followed by some spaces, then "/*"
      "beginRegex": "{{:BeginCommentBlock}}(?![^\\\"]*\\\"[^\\\"]*(?:\\*\\/))",
      // Anywhere on a line with "*/"
      "endRegex": "{{:EndCommentBlock}}"
    },
  ],
},

[daiyam]

Also there might a lunatic who will create a language called variables...

[TonyGravagno]

Since the "variables" block defines patterns and literal string values, I don't think it's required to specify the ":" "!" escaped/raw code in each usage. It can be specified once in the variable block:

"BeginCommentRegex": ":\\/\\*", // ":" prefix here
"BeginCommentText": "!/*",
...
"beginRegex": "{{BeginCommentRegex}}(?![^\\\"]*\\\"[^\\\"]*(?:\\*\\/))", // no ":" prefix

Further... It's obvious in the context of regex that we're using regex (within the "beginRegex"). Maybe we can assume that a variable matches the type of the current prop value "begin" or "beginRegex", and only prefix the variable if it's going to be used in a different context.

To be clear, in the above, "BeginCommentRegex" is used within "beginRegex". We don't need to add a colon on the variable value because we know it's only used in a beginRegex prop. I doubt someone is going to use the same variable in different contexts:

name: "rule1",
begin: "{{!BeginComment}"
...
name: "rule23",
beginRegex: "{{:BeginComment}}"

It seems more reasonable to define variables for their purpose:

name: "rule1",
begin: "{{BeginComment}"
...
name: "rule23",
beginRegex: "{{BeginCommentRegex}}"

The prefix actually seems unnecessary - a rare option that may or may not ever be used and might only serve to confuse more than help.

Maybe a contradiction, but I do think unescaped variables are helpful:

"BeginCommentRegex": "\\/\\*",
"BeginCommentText": "/*",
}

I prefer to see "/*" as a human-friendly pattern when possible than to be burdened with the regex. With a ":" prefix that says "escape this" or with some convention on variable names, it would be nice if a variable can be auto-converted from raw to escaped when injected into the final regex for compilation. That is, we write "/*" and it's transparently changed to "\\/\\*".
Honestly I think the value of this is as rare as the prefix itself but maybe with some thinking on this we can establish a better pattern for distinguishing escaped/raw text that does present some value.

[TonyGravagno]

try:

"explicitFolding.rules": {
  // "lang1,lang2": []
  "*": [],
  "#vars": {
    "variables": {
      "BeginCommentBlock": "\\/\\*",
      "EndCommentBlock": "\\*\\/",

OK, but htat syntax is icky. 🤮

SyntaxError: Expected "$", "(", ".", "\\", "\\0", "\\B", "\\D", "\\P{", "\\S", "\\W", "\\b", "\\c", "\\d", "\\f", "\\k<", "\\n", "\\p{", "\\r", "\\s", "\\t", "\\u", "\\u{", "\\v", "\\w", "\\x", "|", CharacterSet, Literal, Quantifier, or end of input but "/" found.
[document] fileName: settings.json, route: main
[document] lang: jsonc, fileName: settings.json
[main] regex: /a^/

Without being able to see what it's processing we don't really know which "/" it's tripping on. (#145).

(Another issue but here for now...)
The only thing useful about that SyntaxError is that at the very end it shows but "/" found. Unless a developer knows to look at the end fo rthat helpful information, it gets lost in the long stream of SyntaxError messages that display when the rules are loaded.

[daiyam]

Currently, it's

property	{{:varname}} (escape)	{{!varname}} (raw)
non-regex	✅	☠️
regex	✅	✅

I was thinking that maybe an escaped string for regex could be useful in some case.

Yes, "endRegex": "{{:EndCommentBlock}}" needs to be "endRegex": "{{!EndCommentBlock}}" since the : is escaping the value.

[daiyam]

OK, but htat syntax is icky. 🤮

Yes, but recommend to use explicitFolding.rules inside a language block like [typescript]. It easier to manage. The global explicitFolding.rules is there only for * and list of languages.

And #vars make a group of variables so you can have more than one and select one based on the languages.

[daiyam]

For the syntax, it has its origin from https://olado.github.io/doT/index.html
I know there is other syntax with only {{varname}} but I like to be explicit on when to use the raw value or not.
It's just make clear that you are pasting another regex and not a random string.
But, maybe, in our case, it's useless or a bit too much...

What do you think?