Please provide us more ways or easier ways to horizontal code reuse

[Nefcanto]

I have discussed codes similar to this before:

• [Proposal]: Add the composition capability to C# #7406
• Can we have default properties, and base web interfaces too inside the interfaces? #8075
• Stuck because of the lack of composition support. How can I combine read, write, read-tree together? #8157

From those discussions, I ended up deciding that the only way I have is to use source generation. Yet source generation is extremely difficult as a way for horizontal code reuse. Instead of a simple include statement, we should do what not to add a bunch of methods to a given class. It works but with the current complexity, it's not truly affordable. The learning curve, debugging pain, separate tech, wiring headaches, and other problems reduce its true feasibility.

Can you simplify horizontal code reuse or add new ways for it?

[HaloFour]

Source generators are definitely the solution for automatically pasting a bunch of templated source into your code. If you're having specific issues with source generators they can be considered and addressed through improvements or documentation, but the language isn't going to start adding templating features because of that.

[Nefcanto]

@HaloFour. Let's consider the partial class as you told me here. I worked with it and it actually solved my problem.

I have a public partial class GraphBusiness in the GraphBusiness.cs file, And then a public partial class GraphBusiness in the GraphBusinessGenerated.cs file and com-pose different snippets of C# code alongside each other there.

And as I told you here I created a simple com+poser in Python to compose my C# partial class from snippet files.

Well, it worked great. I even shipped the code to the production and it is under load right now and works great. Yet there is a problem with that approach. It's blind. I don't have access to the AST or CST anymore. I can't make smart decisions when com+posing the code.

So I have to either come back to the source generation, or another method. I would appreciate it if you could improve the DX of source generation.

Can I tell you one by one?

[dlebansais]

Sorry, I wasn't clear. I'm talking about bugs specific to the Roslyn technology. Obviously bugs related to my domain are on me.
Let me give you an example: as of today, I still don't know if in AnonymousMethodExpressionSyntax the ExpressionBody property can be non-null, I could not find the syntax to test it. Granted I did not spend days on it, and I'm probably just lacking expertise in C#. I just wanted to be specific about the difficulties I'm encountering.

[iam3yal]

@dlebansais I read it but honestly I don't know what properly, bug-free and test of time really means in the context of software. Fact is it's an iterative process and there is a maintenance cost to everything and sometimes it ends up being trial and error until you get things right... unless you're shipping the firmware to an unknown star and you can't fail or sth. :)

I guess it all comes down to the world being made of two kinds of programmers, the talented and the non-talented, me being in the later category.

Mate, I think you're too hard on yourself... you don't need to be talented to be smart but more importantly it's easier to reach out and ask for help, there is always a person smarter than us.

[Nefcanto]

@CyrusNajmabadi I totally disagree. ASP.NET has more audience than SG, yet it's now much simpler to work with.

It got simpler because the team wanted to. As long as you stand on your position, nothing would change.

Yes it's difficult. But what makes it more difficult than AST and CST in JS code?

My personal DX experience was awesome there. I wrote my first static code analysis tool using the same node.js and npm and everything in less than an hour.

C# is not, and should not claim that it is, more complex. The JS world is just as complex as C# and any other ecosystem.

[CyrusNajmabadi]

I still don't know if in AnonymousMethodExpressionSyntax the ExpressionBody property can be non-null

The docs say explicitly:

    /// <summary>
    /// Inherited from AnonymousFunctionExpressionSyntax, but not used for
    /// AnonymousMethodExpressionSyntax.  This will always be null.
    /// </summary>

I'm not sure what to do to help out with this.

[CyrusNajmabadi]

My personal DX experience was awesome there. I wrote my first static code analysis tool using the same node.js and npm and everything in less than an hour.

Javascript is an ecosystem that supports loading javascript directly on demand (as it is a dynamic hosting environment, where js itself is the loadable artifact).

Doing this in .Net is not supported as: 1. we're a common language runtime where you can use any language. 2. our loadable artifact is IL/metadata.

Asking for .Net to change that is entirely unrelated to SGs and entirely out of the scope of what they provide. If you want .net to be able to load and execute C# on demand, that's a request outside of SGs/analzyers and fundamentally alters major design principles of the platform as a whole.

Your language request for the OP feature is already several orders of magnitude more expensive than just using .net as it is designed and using SGs within that. Altering .Net in the way you are describing is orders of magnitude more expensive THAN THAT.

You've now made the cost easily 4-5 orders of magnitude over just using an SG. Just compiling it. And just loading it normally. LIke how all of .net works for all other cases. Just to solve a problem you could solve in a few hours if you just made a project and compiled it.

The JS world is just as complex as C# and any other ecosystem.

ANd the JS world is much worse in many respects because of it. We do not want to go down an astronomically expensive path to get worse outcomes in many areas, just to make this scenario marginally less expensive for you. And by "marginally less expensive" i mean: making it so you can avoid saying "new project". These are not sensible ecosystem investments when the existing user costs are so small.

[Nefcanto]

@HaloFour, can I start from here: Why can't I create source generators, as part of my current source code? Inside a directory. Let's start from there.

What prevents the .NET team to distinguish source generators from the remaining code, and run it in the pipeline?

[Nefcanto]

@CyrusNajmabadi in this case it's a list of requests. One is "same directory". The other is ".net9 target framework".

The other request is simplifying debugging DX. At best we have to create a test to debug it. Can we just call a source generator in Program.cs, passing a directory of source files to it, and debug that way?

Debugging like a "hello world" console app is undeniably much simpler than debugging using a task or Debugger.Break and attach to the process.

Something like:

  new GraphWriteGenerator().Generate(pathOfCodeFiles);

[CyrusNajmabadi]

The other is ".net9 target framework".

I have responded on this. You haven't given info on what is needed from that that is beneficial to the Analyzer/Generator space. For context, no other parties (1st, second or 3rd) have needed this. So if we wanted to supply this to you, it would be necessary to make a case as to why we'd be doing that work, especially as it would lead to analyzers/generators that much of the ecosystem coudl not use.

[CyrusNajmabadi]

The other request is simplifying debugging DX. At best we have to create a test to debug it.

That is not true. You can debug an SG just like you would any other piece of code. It's just normal .net code. Attaching a debugger to it will debug it.

[CyrusNajmabadi]

Can we just call a source generator in Program.cs, [...] and debug that way?

Yes. That is supported. We have MSBuildWorkspace for this experience. So you can load a particular solution/project and analyze the source in it.

passing a directory of source files to it,

This is not supported. But that's because this doesn't have meaning in .net. We have to know what your references are. We have to know what your options are. Or basic things will not work. For example, any reference to any type outside of that source will fail to compile.

[iam3yal]

@Nefcanto Where this code lives? and when it's executed? but I don't even know how this simplifies anything or what it should simplify can you explain?

[Nefcanto]

The next thing that I have in mind for simplification is merging the ISyntaxReceiver into the ISourceGenerator.

I want to add ReadBusiness capability to a couple of classes like HierarchicalBusiness, GraphBusiness, HeteroBusiness, etc.
I need to find them and inject a bunch of read methods like GetList and GetById and GetByFilter, etc. to them.

Why should I create a syntax receiver and add them to a collection separately, then register that receiver, then check the type of the receiver and then loop over them?

Can't I just register a node visitor and on each visit, generate the code?

In other words, instead of this:

public class ReadBusinessSyntaxReceiver : ISyntaxReceiver
{
    public List<ClassDeclarationSyntax> BusinessClasses { get; } = new List<ClassDeclarationSyntax>();

    private static List<string> filter =
    [
        "HierarchicalBusiness",
        "GraphBusiness",
        "HeteroBusiness",
        "AugmenterBusiness",
        "CsvBusiness",
        "FakeDataInsertionBusiness",
        "GranularBusiness",
    ];

    public void OnVisitSyntaxNode(SyntaxNode syntaxNode)
    {

        if (syntaxNode is ClassDeclarationSyntax classDeclaration)
        {
            if (filter.Any(i => i == classDeclaration.Identifier.Text))
            {
                BusinessClasses.Add(classDeclaration);
            }
        }
    }    
}

public class ReadBusinessGenerator : ISourceGenerator
{
    public void Initialize(GeneratorInitializationContext context)
    {
        context.RegisterForSyntaxNotifications(() => new ReadBusinessSyntaxReceiver());
    }

    public void Execute(GeneratorExecutionContext context)
    {
        if (context.SyntaxReceiver is not ReadBusinessSyntaxReceiver receiver)
        {
            return;        
        }

        var sourceCode = @"// finding the read methods and adding them to the target business";
        context.AddSource("GeneratedClass.g.cs", SourceText.From(sourceCode, Encoding.UTF8));
    }
}

Why can't I just have:

public class ReadBusinessGenerator : ISourceGenerator
{
    private static List<string> filter =
    [
        "HierarchicalBusiness",
        "GraphBusiness",
        "HeteroBusiness",
        "AugmenterBusiness",
        "CsvBusiness",
        "FakeDataInsertionBusiness",
        "GranularBusiness",
    ];

    public void Execute(GeneratorExecutionContext context)
    {
        if (context.Node is ClassDeclarationSyntax classDeclaration)
        {
            if (filter.Any(i => i == classDeclaration.Identifier.Text))
            {
                var sourceCode = @"// finding the read methods and adding them to the target business";
                context.AddSource("GeneratedClass.g.cs", SourceText.From(sourceCode, Encoding.UTF8));
            }
        }

    }
}

All I'm saying is that what I want is to find out nodes and act based on them. Why can't I just do that in one class? Even if you prefer or have to keep them separately, give us a combined experience in a super interface or orchestrator interface. An interface called ICombinedSourceGenerator where we can simply filter out nodes and act at the same time. That's so weird to first collect nodes and then act upon nodes.

[CyrusNajmabadi]

Another problem is that it has a method to be implemented called Initialize and we generate code inside that method.

No. You don't generate code inside that method. You initialize your generator, setting up the context object you are passed the relevant incremental generator steps.

Source generation happens later, entirely determined by the incremental engine.

[CyrusNajmabadi]

And I should be able to create a simple class like this:

This would tank performance. Again, we have experience with this. Systems like "code model" and the v1 generator API were attempted. They are not viable. Such an approach does not scale as is enormously expensive in terms of performance.

We don't avoid apis like this you make your life hard. We avoid apis like this because consumers of the generator end up suffering. Again, this is not hypothetical. This was thousands upon thousands of reports from companies about perf tanking because of generators.

This is precisely what I was talking about when I said that each user has an idea about what they want, but we have to design solutions that work for the ecosystem as a whole.

[CyrusNajmabadi]

Instead of IncrementalValueProvider why not a simple Filter?

Because it's not a filter. It's a provider of incremental values.

Those values may be transformed. Those values may be combined. Those values may be collected. Etc. etc.

A simple IGenerator can be much simpler than IIncrementalGenerator.

The core principle here, which people do need to understand if they want to create scalable generators is the incremental part.

When the world outside has nothing to do with the incremental

All of compilation, building, and analysis is incremental. That's what ensures it is fast enough to be usable. If we didn't have those all be incremental, then every edit you made would take MINUTES (or worse), until you got to do anything with the edited code. Msbuild incrementally builds your code. The compiler server incrementally builds your code. The language server incrementally analyzes your code.

Our original generator system was not incremental, and it was designed exactly for ease of use. It was an unmitigated DISASTER. Think a user types a character and VS ends up allocating 100gb of RAM in response to it. That sort of a disaster. So we moved to an incremental architecture and exposed it to users. This allowed teams like the runtime team to move all their generators over to being incremental.

This move dropped memory and cpu usage by 3-4 orders of magnitude (not an exaggeration, the actual numbers).

So we don't do this stuff for "jargon". We do this because it's needed by the community as a whole to have an effective system that is actually usable.

[CyrusNajmabadi]

. Oh body I'm such a sublime person" or to impress others. It's called grandiloquence.

You haven't talked to a single person to try to understand the reasoning behind the naming. You haven't talked to a single person to try to understand what's going on here. You've made several posts indicating a misunderstanding about how SG works (for example the initialization step, or viewing the IVP as Filter).

Can you please stop making assumptions about things you haven't fully internalized yet, and then using that to cast aspersions at others? We are here and available to talk, usually at most times of the day world wide. Come to our discord and spend some time learning about these topics with us instead of doing a cursory glance and making assumptions that are not actually true.

This will save a lot of time and headache. And it will also help avoid the unhelpful denigration and disparagement of team members.

[CyrusNajmabadi]

This convo has run it's course. Please come to discord if you would like to learn more about how to use Roslyn best to help out your projects.

[Nefcanto]

Another simplification. Can you provide some common source generation scenarios out of the box?

I compare the source generation to ASP.NET. In ASP.NET we do have a lot of goodies out of the box, like some middleware classes that we can use and if we want something more, then we should customize.

Can you provide some source generators out of the box?

For example, it would be great to have this attribute, out of the box:

[Compose("CodeSnippetName")]
public partial class SomeClassToBeExtendedViaComposition

Can you think of some common needs that people solve using source generation? And if we can get a list, then we might be able to piche the top 10 needs and create source generators for them. That would be very useful.

[iam3yal]

@Nefcanto I genuinely don't understand the point of that unless it's a game changer e.g. GeneratedRegex