Source generation support

[sebastienros]

@lahma @jbevain just to show you how it's done.
The logic is not part of the source generator and is driven by user code execution, not code syntax.
The source generator has to run the code first to get the object describing the grammar. Then evaluates the generated tree, not the code itself. That requires the code to be in a static method but that's totally fine. If the grammar needs options then one can generate different parser for each option, and reuse grammar portions across.

[sebastienros]

@Genteure I fixed all the possible compilation issues, at least it works on NCalc which relies on compiled regexes, logging source gen and PolySharp. Just saying that Parlot's source generator will also invoke other source generator so it can build and run the host assembly so it can generate new code for this same host assembly. Not sure how to feel about it, but it seems to work. I am pushing a new version right now.

@Bykiev Made some progress and hitting a problem with the outer LeftAssociative parser that throws an exception, I don't understand what it does, neither does the source gen ;)

[Genteure]

I was watching the repository so this is already in my inbox, but appreciate the update 🙂
Perhaps you meant to ping @gpetrou

[gpetrou]

Should the new version work with System.Text.RegularExpressions.Generator?
There is a warning PARLOT012: Could not find generator assembly 'System.Text.RegularExpressions.Generator' specified in [IncludeGenerators]. Make sure the package is referenced in your project.
Since this is part of the framework, I assume that there is no need for a package.

[sebastienros]

@gpetrou ignore this one. Parlot stubs this one, and logger generator too. It should just work, my benchmark project had one too and it works.

[gpetrou]

Oh, I see. OK thanks.
I found one more issue.
If I have something like the following (just for demonstration purposes):

private abstract class NodeBase
{
}

private class BasicNode<T> : NodeBase
{
    public BasicNode(T value)
    {
        Value = value;
    }

    public T Value { get; }
}

private static readonly Parser<long> Long = Terms.Number<long>(NumberOptions.Integer);
private static readonly Parser<string> Equal = Terms.Text("==");

private static Parser<NodeBase> CreatePropertyParser<TComparand>(
   string name,
   Parser<string> @operator,
   Parser<TComparand> comparand)
{
    return
        comparand
            .AndSkip(@operator)
            .AndSkip(Terms.Text(name, caseInsensitive: true))
            .And(comparand)
            .Then<NodeBase>(
                    items => new BasicNode<TComparand>(
                        items.Item2));
}

[GenerateParser]
private static Parser<NodeBase> Parser()
{
    return CreatePropertyParser(
        "long",
        Equal,
        Long);
}

It seems to be missing the <TComparand> in the generated code.

[sebastienros]

@gpetrou I added a test with you sample, it's working fine.

NCalc is also working correctly without crazy changes. All the things that needed to be altered were described with the analyzer so it was smooth. The most copmlicated was to make the grammar change based on external options, since this can't be done when the parser is constructed anymore, but it all works out with the available parsers combinators (Select). ncalc/ncalc#533

[gpetrou]

I added a test with you sample, it's working fine.

Hmm, I still observe an error with the latest preview 8.
In the generated file, I see:

#line 36 "D:\Projects\ParlotWithSourceGenerator\ParlotWithSourceGenerator\MyParser.cs"
        private static global::ParlotWithSourceGenerator.MyParser.NodeBase _Parser_lambda0(global::System.ValueTuple<global::System.Int64, global::System.Int64> items) => new BasicNode<TComparand>(
                            items.Item2);
#line default

I guess the issue is that we have new BasicNode<TComparand> instead of new BasicNode<long>.

[slang25]

@sebastienros this looks awesome 🤩 what's the current state of this?

[sebastienros]

@slang25 this only thing to add, if anything, is to improve the fact that an hint is displayed in VS or VS Code on the method with the attribute, but couldn't find a way to detect it without impacting the local build (detecting when msbuild is used). It's a niche scenario, and not breaking at all, but I'd prefer to fix it before. This PR is already testable using the published preview.