proposal: call point generators

[darrenoakey]

I've put in a few proposals, and it just occurred to me that a large number of the things I'd like to add could all be generalised to a single tool - an annotation based generator - that basically allowed you to annotate either a parameter or a method, and whenever the compiler comes to call it - it just calls your generator with information about the current context, and like a code dom of the expression being put in

so basically you would first define an annotation - eg the swift autoclosure:

class AutoClosureAttribute : InsertionPointGeneratorAttribute 
{

     override void Execute( SourceGeneratorContext context )
      {
          // here's where the context contains the tree of the expression being passed in to that parameter, as well as all
          // sorts of stuff like file name, method, enclosing type etc
          // and we just manipulate the codedom to change expression to () => {expression}
       }
}

so then you just make a function that uses it

public static class Log
{
     public void Info( [AutoClosure] Func<string> message ) {...

and finally you call it

     Log.info( ( 1 + 2 + 3).ToString() );

at which the compiler goes "oh this is a generator attribute", and just calls your custom generator. You could do it either at the parameter level, or the whole method level - so you can just essentially make method placeholders, which just build code - they might call an entirely different method, or nothing at all. It's sort of a parallel of the existing source generator - but while that generates and adds to the library assembly, this would generate and add to the consuming assembly. So someone could just implement [CallerMethodName] themselves, or [CallerAssembly] or [CallerFunctionParameters] or [SynchronizedCall] - or [AddSourceCodeSnippetForExceptionMessage] or [ReadDatabaseTableStructureAndReturnATupleWithAllTheCorrectFieldNames] - etc etc.

[Joe4evr]

Have you tried out Source Generators yet?

[benaadams]

You can change the parameter Func<string> message to Expression<Func<string>> message; is same to caller but gives you the expression tree for the passed method rather than just the method to call?

Have a look at Expression Trees https://tyrrrz.me/blog/expression-trees

[canton7]

The problem here is if your Log.Info method is declared in a different assembly to the assembly which uses it.

The user would need your source generator installed. Otherwise, the compiler wouldn't know that it should do anything special: you'd get a silent breakage, but the user wouldn't be told that they need to go and install some other package.

[darrenoakey]

You can change the parameter Func<string> message to Expression<Func<string>> message; is same to caller but gives you the expression tree for the passed method rather than just the method to call?

Have a look at Expression Trees https://tyrrrz.me/blog/expression-trees

yeah I use expression trees a bit - they are cool - but I want to make alterations to the code at the call site - and I don't see any way of doing that with expression trees? (If there IS such a way, please educate me and make me very happy!)

[darrenoakey]

The problem here is if your Log.Info method is declared in a different assembly to the assembly which uses it.

The user would need your source generator installed. Otherwise, the compiler wouldn't know that it should do anything special: you'd get a silent breakage, but the user wouldn't be told that they need to go and install some other package.

I don't really understand this point? the compiler sees attribute X - sees that it implements insertion point generator attribute... so it knows it has to generate code, and it has a reference to the generation code - because it's in the attribute. It has everything it needs why would the user need to install something?

[canton7]

The problem here is if your Log.Info method is declared in a different assembly to the assembly which uses it.
The user would need your source generator installed. Otherwise, the compiler wouldn't know that it should do anything special: you'd get a silent breakage, but the user wouldn't be told that they need to go and install some other package.

I don't really understand this point? the compiler sees attribute X - sees that it implements insertion point generator attribute... so it knows it has to generate code, and it has a reference to the generation code - because it's in the attribute. It has everything it needs why would the user need to install something?

Image two computers.

Computer A is used by the author of the logging library to develop and compile the library. The logging library author defines the attribute, and defines a compiler generator thing which looks for that attribute and does the appropriate rewrites. That compiler generator thing is installed into Visual Studio on Computer A.

Computer B is used by someone who uses the logging library. Computer B is the place where the Log.Info calls happen. The compiler on computer B sees that the logging library defines and uses some attributes, but it sees that as simple attributes. It does not have the source generator thing installed, and it does not know that it needs it. The user here does not get any of the functionality that the compiler generator thing provides, and doesn't know that they're missing out: they don't get any error messages which indicate that they need it.

Remember it's the calls to Log.Info which need to be rewritten, and those happen on Computer B, not on Computer A.

[svick]

@canton7 I think you will be able to distribute source generators as NuGet packages (just like with analyzers today), so if you're writing a library NuGet that requires a source generator, you can add a dependency on that source generator NuGet. That means that when you reference the library, you'll also install the source generator.

[darrenoakey]

Computer A is used by the author of the logging library to develop and compile the library. The logging library author defines the attribute, and defines a compiler generator thing which looks for that attribute and does the appropriate rewrites. That compiler generator thing is installed into Visual Studio on Computer A.

nooooo - not what I'm suggesting at all. Someone - eg - the AutoCloseable library writer.... makes an attribute called autocloseable. that attribute contains all the code to do the generation.

The writer of the log library uses it. So his library depends on the autocloseable library - so therefore brings all the generation code along for the ride.

Someone on computer B uses the log library. Their compiler goes "oh hey, this is a generation attribute" - so it then does a proper load of the autocloseable library, which mind you has already been loaded but probably in reflection only mode... and runs the execute function that's hanging off the attribute. There's no separate source generator in existence at any time.

OMG - does this mean the generation code itself actually will end up being part of the eventual production package? Yes.. yes it does.

[canton7]

If the compiler is compiling an assembly which references your logging library, you can't assume it has access to anything more than a reference assembly. The compiler is unable to instantiate your attribute or execute any of its methods.

[darrenoakey]

I don't understand this at all?
surely if it has a reference to library X - it can instantiate library X?

[canton7]

The library which references your logging library will (we assume) have access to the full logging library at runtime. The compiler cannot assume it has access to anything more than the metadata at compile time.

If you use Roslyn, you will quickly see that you can access information on the attribute constructor which is called, constructor parameters, and properties which are assigned. You cannot instantiate it.

[canton7]

Also, since your attribute references SourceGeneratorContext, you either need to take a dependency on the Roslyn NuGet package which defines it, or Roslyn needs to be provided by the runtime. Both are bad.

I'm afraid having types available at runtime which are called into by the compiler at compile-time, and which use types defined by the compiler, is almost certainly a no-go.

[darrenoakey]

so don't assume it... if someone uses this feature - and they don't have access to the dll - then give a compile error saying "can't instantiate blah.dll" - I don't know how to reference something without having the dll, unless it's like a com interop thing - but I'm sure you're right, it's probably possible... but really ... big deal - in that rare case you just tell people what they need to to fix it.

Yes - you would be bringing in a dependency on roslyn, and have that availalbe at runtime - again... big deal. We do that anyway, even without this, as we use it for various static analysis, and we just release the whole package. And if you don't like it - you can always just not include the dlls in the package - just because there's a dependency - it won't try to load the dll unless you try to use it.

[CyrusNajmabadi]

I don't know how to reference something without having the dll

This is extremely common nowadays. Consider it like how you can reference headers in C++ independently from impls. This is teh .net equivalent. All you get are signatures.

[canton7]

I suspect the effort required to change large chunks of the compiler to support this is beyond what this proposed feature is worth, unfortunately, even if the compiler team changes their original decision on this. I recommend trying to find an approach which is a bit less disruptive.

[333fred]

@darrenoakey you might find reading through this article helpful: https://docs.microsoft.com/en-us/dotnet/standard/assembly/reference-assemblies. Ref assemblies are very commonly used nowadays at compile time, as they can improve compile performance.

[darrenoakey]

so.. I guess ref assemblies are apparently being useful and common, learn something new every day - however given that - I don't really see a problem with this error message:

"sorry mate - the compilation failed... turns out Blah.dll is a reference assembly and because it contains a generator, you'll want to be finding yourself a real version of blah.dll in order to compile"

being a thing.

Also @canton7 - I don't see how you'd need to change large chunks of the compiler to support this? I would see this as having the most minimal impact on the compiler. In fact, wouldn't supporting it be like adding a line or two? ie:

        var attributes = theParameterToGenerate.CustomAttributesOfType<GenerationAttribute>()
        foreach (var attribute in attributes)
               attribute.Execute( currentGenerationContext)

[canton7]

If you think it's that easy, I encourage you to try adding it to the compiler yourself as an experiment. Since you seem to be fairly unfamiar with Roslyn, this would be a great way of understanding what you're working with.

My concern is that currently I don't think the compiler loads the bodies of any methods at all from referenced assemblies, so that's your starting point. The compiler isn't set up to call onto the code it's compiling at all, so you're adding all of that infrastructure. There isn't a CustomAttributesOfType method which instantiates an attribute.

Also, I think this will constrain the assembly containing your attribute to netstandard2.0, since it will need to run inside the compiler.

[YairHalberstadt]

@darrenoakey

There exist various proposals for code generation that don't have these issues and using existing roslyn extension points (e.g. analyzers, source generators). However they've generally been rejected due to the user experience.

For example I thought very carefully in this proposal: #3633 about how to make sure the user experience was workable, as well as solve various other problems. That still wasn't considered sufficient.

I think that's what needs to be focused on.

As a simple example, how would your code generators act when two of them apply to the same bit of code?