Discussion: Code Generator Catalog

[gafter]

@alrz commented on Fri Dec 30 2016

Here is an incomplete list of potential use cases for code generators to explore ideas and discussion. I believe this will help to shape generator APIs and language features around it.

1. NPC implementation (huh)

partial class Person
{
  public string FirstName { get; set; }
  public string LastName { get; set; }
}

// generated
partial class Person : INotifyPropertyChanged
{
  private string _firstName;
  public replace string FirstName
  {
    get => _firstName;
    set 
    {
      if (_firstName != value)
      {
         _firstName = value;
         OnPropertyChanged(nameof(FirstName));
      }
    }    
  }
}

Also, if a property depends on others, we could also raise additional events by inspecting its accessor.

[NPC]
public string FullName => this.FirstName + ", " + this.LastName;

// generated
public replace string FirstName
{
  get => _firstName;
  set 
  {
    if (_firstName != value)
    {
       _firstName = value;
       OnPropertyChanged(nameof(FirstName));
       OnPropertyChanged(nameof(FullName));
    }
  }
}

Note, #850 can lead to simpler code generation i.e does not need to figure out a name for backing field:

public replace string FirstName
{
  string field;
  get => field;
  set 
  {
    if (field != value)
    {
      field = value;
      OnPropertyChanged(nameof(FirstName));
      OnPropertyChanged(nameof(FullName));
    }
  }
}

Or the type (#8364):

public replace string FirstName
{
  get;
  set 
  {
    if (field != value)
    {
      field = value;
      OnPropertyChanged(nameof(FirstName));
      OnPropertyChanged(nameof(FullName));
    }
  }
}

Note: If none of replaced members call original the original declaration should be removed as the backing fields are no longer being used.

2. Dependency Injection

Dependency containers commonly depend on reflection to create objects. This could be moved to compile-time via code generators. In this scenario, there should be a way to parametrize the code generator to switch between implementations. e.g. mocks. (attributes does not belong to MEF).

[Export]
class Service1 : IService1
{
  [Import]
  private IService2 Service2 { get; }
}

Note that this will be only useful to manage object lifecycle. If implementations come from outside of assembly boundaries we should probably fallback to reflection under the hood.

3. Caching / Lazy Initialization

[Cache]
public string Property => ComputeProperty();

// generated
private string _property;
public replace string Property => _property ?? (_property = original);

4. Memoization

[Memoize]
public string Function(string arg) { ... }

// mind you, a simple demonstration without thread-safety
private readonly Dictionary<string, string> _fCache = new();
public replace string F(string arg)
  => _fCache.TryGetValue(arg, out var result) ? result : _fCache[arg] = original(arg);

5. Dependency Properties

partial class Foo : DependencyObject
{
  [PropertyMetadata(defaultValue: string.Empty)]
  public string Name { get; set; }
  public int Size { get; }
}

// generated
partial class Foo 
{
  public static readonly DependencyProperty NameProperty =
    DependencyProperty.Register(nameof(Name), typeof(string), typeof(Foo), new(string.Empty));
    
  public replace string Name
  {
    get => (string)GetValue(NameProperty);
    set => SetValue(NameProperty, value);
  }
  
  internal static readonly DependencyPropertyKey SizePropertyKey =
        DependencyProperty.RegisterReadOnly(nameof(ReadonlyNam), typeof(string), typeof(Foo));

  public static readonly DependencyProperty SizeProperty = SizePropertyKey.DependencyProperty;
    
  public replace int Size
  {
    get => (int)GetValue(SizeProperty);
    internal set => SetValue(SizePropertyKey, value);
  }
}

Note, it would be nice to be able to inspect property initializer and use it in the generated code, e.g.

public string Name { get; set; } = string.Empty;

6. ORMs

Currently ORMs) use runtime code generation (NH) or proxies (EF) to enable change tracking and lazy loading in POCOs. They could ship with a code generator to move this procedure to the compile-time.

class BlogPost
{
   public string Title { get; set; }
   public string Body { get; set; }
   public List<Comment> Comments { get; }
}

7. Mixins

It's possible to implement member delegation as an analyzer but if we want to delegate all members, perhaps a code generator is more likely preferable.

partial class Class
{
  [Mixin] private readonly ISomething _something = new Something();
}

// generated
partial class Class : ISomething
{
  public void DoSomething() => _something.DoSomething();
}

8. Double Dispatch

This could be used to implement visitor pattern or a simple double dispatch:

public object F(T x) { .. }
public object F(U x) { .. }
[DoubleDispatch]
public extern object F(Base p);

// generated
public object F(Base p)
{
  switch(p)
  {
    case T x: return F(x);
    case U x: return F(x);
    default: throw ExceptionHelper.UnexpectedValue(p);
  }
}

You could use a similar technique to generate semi-virtual extension methods. Since this is generated by a generator, you are free to handle the failure case differently.

Generators should be able to produce diagnostics if target members are malformed, e.g. a method instead of a property.

9. Duck Typing

interface IDuck
{
  void Quack();
}

[Duck(typeof(IDuck))]
class A { public void Quack() {} }

void F(IDuck d) { }

F(new A().Wrap());

// generated
static class Extensions
{
  class WrapperA : IDuck
  {
    private readonly A _obj;
    public WrapperA(A obj) => _obj = obj;
    public Quack() => obj.Quack();
  }
  public static IDuck Wrap(this A obj) => new WrapperA(_obj);
}

Though, #11159 + #258 = #8127 can greatly improve this scenario in terms of perf and easier code gen.

Note: An assembly attribute could be used to annotate exterior types,

[assembly: Duck(typeof(IDuck), typeof(AnotherAssembly.B))]

10. Type providers for xml, json, csv, sql, etc

F# type providers can take a string as parameter to bootstrap code generation. This requires code generators to accept a parameter which is not possible in #5561.

11. Variadic generics

There are types with variable arities including Func, Action and ValueTuple. It is a common scenario where we want to have a bunch of similar methods that are only different in number of generic types.

12. Basic implementation

Some of interfaces like IEquitable, IComparable etc, given "key properties" could be implemented via generators . This can also be implemented as an analyzer, but with generators it would be totally transparent. ToString overrides also belong to this category.

13. Serialization

Serialization to various formats like json, can be provided at compile-time without using reflection.

---

@HaloFour commented on Fri Dec 30 2016

I get where you're going with extern but that seems like it would add a big layer of complexity given the member then must be replaced but the replacement must not call original. With that as a possibility the conerns of ordering becomes much more important.

---

@alrz commented on Sat Dec 31 2016

@HaloFour Only if you don't mind the additional backing field per property (or having to return default for methods.) The latter wouldn't be that much trouble (if you don't have out parameters) but the former would definitely affect memory usage. Note that extern is just a non-ambigious alternative to #9178 which was originaly mentioned in dotnet/roslyn#5292 (comment).

---

@HaloFour commented on Fri Dec 30 2016

@alrz

The partial keyword might also work there given that's sort of how it is used today, although with numerous behavioral differences and the compiler eliding calls if never implemented.

Either way, I don't disagree with the concept, just mentioning the additional complexity it creates. Order of member replacement is already a known concern.

---

@alrz commented on Sat Dec 31 2016

The compiler should remove the original declaration if none of replaced members called original. This could address the problem with additional backing field but I think extern is still useful on methods.

/cc @HaloFour I've updated OP to reflect this.

---

@bbarry commented on Sat Dec 31 2016

One could implement IDisposable via a generator:

[GenerateDisposable]
public partial class Resource : IDisposable   
{
    [Dispose]
    private IntPtr nativeResource; 
    [Dispose]
    private AnotherResource managedResource;

    ...
}

---

Generating async (code lifted from https://github.com/npgsql/npgsql, idea courtesy @roji)

[RewriteAsync]
void SendClosureAlert()
{
    _buf[0] = (byte)ContentType.Alert;
    Utils.WriteUInt16(_buf, 1, (ushort)_connState.TlsVersion);
    _buf[5 + _connState.IvLen] = (byte)AlertLevel.Warning;
    _buf[5 + _connState.IvLen + 1] = (byte)AlertDescription.CloseNotify;
    int endPos = Encrypt(0, 2);
    _baseStream.Write(_buf, 0, endPos);
    _baseStream.Flush();
}

//generated:
async Task SendClosureAlertAsync(CancellationToken cancellationToken)
{
    _buf[0] = (byte)ContentType.Alert;
    Utils.WriteUInt16(_buf, 1, (ushort)_connState.TlsVersion);
    _buf[5 + _connState.IvLen] = (byte)AlertLevel.Warning;
    _buf[5 + _connState.IvLen + 1] = (byte)AlertDescription.CloseNotify;
    int endPos = Encrypt(0, 2);
    await _baseStream.WriteAsync(_buf, 0, endPos, cancellationToken);
    await _baseStream.FlushAsync(cancellationToken);
}

---

Allocation free yield (ignore the overflow issue)

public partial class MyFib
{
    public partial struct FibIter {}
    [Iterator(typeof(FibIter), "Fib", BindingFlags.Public)]
    private IEnumerable<int> GenFib()
    {
        int j = 0, i = 1;
        while(true)
        {
            yield return i;
            j += i;
            yield return j;
            i += j;
        }
    }
}

// generated
public partial class MyFib
{
    public FibIter Fib()
    {
        return default(FibIter);
    }
    public partial struct FibIter: IEnumerable<int>, IEnumerator<int>
    {
        private int _1, _2, j, i;
        public FibIter GetEnumerator() => this;
        public int Current => _2;
        public bool MoveNext()
        {
            switch (_1)
            {
            case 0:
                this._1 = -1;
                this.j = 0;
                this.i = 1;
                break;
            case 1:
                this._1 = -1;
                this.j += this.i;
                this._2 = this.j;
                this._1 = 2;
                return true;
            case 2:
                this._1 = -1;
                this.i += this.j;
                break;
            default:
                return false;
            }
            this._2 = this.i;
            this._1 = 1;
            return true;
        }
        object IEnumerator.Current => _2;
        void IDisposable.Dispose() {}
        void IEnumerator.Reset()=> throw new NotSupportedException();
        IEnumerator<int> IEnumerable<int>.GetEnumerator() => this;
        IEnumerator IEnumerable.GetEnumerator() => this;
    }
}

A very similar thing could be done to implement async differently and save allocations, for example dotnet/roslyn#15491 (comment)

---

@alrz commented on Sat Dec 31 2016

@bbarry Worth to mention: #10449.

---

@eyalsk commented on Sat Dec 31 2016

@alrz you can add #15282 to the list. :)

---

@ufcpp commented on Sat Dec 31 2016

Records

This is a temporary solution until Records are released, but,
I have an generator (by using Analyzer and Code Fix) for Record-like purpose:

https://github.com/ufcpp/RecordConstructorGenerator

Type aliases/Strong typedef

[Alias]
struct Name { string value; }
[Alias]
struct Distance { double value; }

// generated

partial struct Name
{
    public string Value => value;
    public Name(string value) { this.value = value; }
    // equality, cast operator, etc.
}

partial struct Distance
{
    public double Value => value;
    public Distance(double value) { this.value = value; }
    // equality, cast operator, etc.
}

Mixins

For mixin use cases, I want an ability to pass this pointer to the mixin.

public class Sample
{
    NotifyPropertyChangedMixin _npc;
}

// generated
public class Sample : INotifyPropertyChanged
{
    NotifyPropertyChangedMixin _npc;

    public event PropertyChangedEventHandler PropertyChanged { add { _npc.PropertyChanged += value; } remove { _npc.PropertyChanged -= value; } }

    protected void OnPropertyChanged(PropertyChangedEventArgs args) => _npc.OnPropertyChanged(this, args);

    protected void OnPropertyChanged([CallerMemberName] string propertyName = null) => _npc.OnPropertyChanged(this, propertyName);

    protected void SetProperty<T>(ref T storage, T value, PropertyChangedEventArgs args) => _npc.SetProperty(this, ref storage, value, args);

    protected void SetProperty<T>(ref T storage, T value, [CallerMemberName] string propertyName = null) => _npc.SetProperty(this, ref storage, value, propertyName);
}

---

@asdfgasdfsafgsdfa commented on Mon Jan 02 2017

Personally I would love having a generator write serialization functions for my objects.
That should easily be possible as well, right?

How would one go about debugging such generated code tho?
I think it would be very important to be able to preview, inspect, debug(including edit and continue) all generated code. Would the additional code be generated into "nested files" (like T4 generated results or code generated by the winforms designer) ?

Manipulating code-text directly might not be the best approach though, so we would end up manipulating the Ast?

So this feature would most likely be some kind of more thightly integrated T4 system (or a variant using the AST)?
What alternatives are there that don't prevent inspection and live debugging of the code?

This could also be utilized in some obfuscators maybe...
To that end users might want to debug the obfuscated versions in two ways:
Viewing and stepping through the code as if it were unmodified (even though the code might be heavily obfuscated by some post processing code generator) and on the other hand someone might want to view the code as it was really compiled, maybe to find some bug in the obfuscation...

Thoughts?

---

@alrz commented on Mon Jan 02 2017

@asdfgasdfsafgsdfa You can refer to the original proposal here: dotnet/roslyn#5561. It would not manipulate AST or anything, it is a "code generator" so it can only add code to the compilation unit. Debugging works directly on the generated code.

---

@asdfgasdfsafgsdfa commented on Tue Jan 03 2017

@alrz
I'm not sure I completely understand.
In the original discussion people mentioned that dealing with strings alone would be somewhat of a hassle.
Same with replacing vs. adding code.

I would rather see replacement instead of (or in addition to) addition of code and AST modification instead of manipulating code-strings. Should I focus on #5561 then?
This issue ( #16160 ) doesn't close or supersede the discussion over at #5561 right? It is just an alternative approach to solve a similar problem, right?

Oh and another quick question: Has there been any official opinion on those questions? I saw it on a "strong interest" list somewhere, but as I understand the actual implementation and usage is still pretty unclear and heavily discussed. Have any of the people that will decide in the end mentioned their own opinion on those more detailed questions?

---

@fanoI commented on Tue Jan 24 2017

One other possible use of Generators could be for bitfield structs this implementation used reflection and so was 10'000 times slower than using bitmasks (!) but using Generators the conversion functions to ToUInt64() and ToBinaryString() could be generated at runtime:

https://www.codeproject.com/Articles/1095576/Bit-Field-in-Csharp-using-struct

---

@eyalsk commented on Mon Jan 30 2017

Another thing that I thought about is using it in the following manner:

[Calculator(Method.Fib, 40)]
const int fib = 0;

[Calculator(Method.CubeVolume, 2)]
const int cube = 0;

---

@gulshan commented on Mon Jan 30 2017

F# community is discussing a type provider to generate types from types. Felt kind of similar. This comment has some probable usage- fsharp/fslang-design#125 (comment)

---

@axel-habermaier commented on Wed Feb 01 2017

Generate PInvoke Code
Code generators could also be used to generate PInvoke code from C/C++ header files.

---

@asdfgasdfsafgsdfa commented on Wed Feb 01 2017

• Having a code generator that does two-way databinding from my settings class to my custom UI controls (when you have to replace the existing databinding solutions).

• A generator that provides your own implementation of [SyncVar] (from Unity3D) either in Unity, or in programs that have nothing to do with Unity.

---

@gulshan commented on Tue Feb 14 2017

I want to add lightweight(local-only) Actor models like nAct and this one to the list- https://github.com/David-Desmaisons/EasyActor
They uses async method calls instead of message passing. A normal worker class attributed as [Actor] can be code generated to have basic thread management(attached to a specific thread) where the methods marked as [Actor.Message] would be put inside a async Task automatically.

[john-cd]

I would add the generation of Trampoline classes to avoid stack overflow in the tail recursion case.

[john-cd]

Tuples - Adding a Deconstruct() method automatically

[iam3yal]

Posting it here just to relink a use-case that was lost in the quotation which was discussed on Roslyn: Events without the boilerplate

[john-cd]

Complex cases of the Dispose patterns with both managed and unmanaged resources

• generate a managed wrapper for unmanaged resources
• generate the Dispose pattern, including the Finalizer

[john-cd]

Storing event instances in a Dictionary

private System.Collections.Generic.Dictionary<string, System.Delegate> eventTable;

        public PropertyEventsSample()
        {
            eventTable = new System.Collections.Generic.Dictionary<string, System.Delegate>();
            eventTable.Add("Event1", null);
            eventTable.Add("Event2", null);
        }

        public event EventHandler1 Event1
        {
            add
            {
                lock (eventTable)
                {
                    eventTable["Event1"] = (EventHandler1)eventTable["Event1"] + value;
                }
            }
            remove
            {
                lock (eventTable)
                {
                    eventTable["Event1"] = (EventHandler1)eventTable["Event1"] - value;
                }
            }
        }

[john-cd]

Generating the boilerplate for "weak events": https://www.thomaslevesque.com/2015/08/16/weak-events-in-c-take-two/

[john-cd]

Visitor pattern

[john-cd]

Generating the lock-free pattern:

public T Mutate(Func<T, T> mutator) {
		T baseVal = value;
		while(true) {
			T newVal = mutator(baseVal);
			#pragma warning disable 420	
			T currentVal = Interlocked.CompareExchange(ref value, newVal, baseVal);
			#pragma warning restore 420

			if (currentVal == baseVal)
				return baseVal;			// Success!
			else
				baseVal = currentVal;	// Try again
		}

[john-cd]

Simple Argument Checking / Code Contracts:

[NotNull("s")]
public void M( string s) {. .. }

[john-cd]

Low-overhead lazy Initialization with LazyInitializer.EnsureInitialized()

[john-cd]

• Interface implementation e.g. in abstract classes.

• Implementation of standard but heavy interfaces like ICollection, IEnumerable, IList and mapping to their generic equivalents.

[john-cd]

Custom exception classes

[john-cd]

Generating custom attribute classes maybe?

[johncrim]

For me, the biggest use for compile-time code-gen is a replacement for the runtime checks generated by CodeContracts - since it seems likely that CodeContracts won't be updated for Roslyn (microsoft/CodeContracts#476). I've spent a little time in the CodeContracts code base, and it seems obvious that it should be scrapped in favor of a Roslyn-integrated code generator.

We mainly use CodeContracts for argument validation, and result assertions. Decent argument validation is IMO the minimum viable product for replacing CodeContracts. For arg validation, a key feature is to only specify the argument expression once - for example:

void RegisterResource(CultureInfo culture, string key, string value)
{
    Arg.NotNull(culture);
    Arg.NotNull(key);
    Arg.Requires<ArgumentException>(! string.IsNullOrWhitespace(key));
...

could be translated to:

void RegisterResource(CultureInfo culture, string key, string value)
{
    Arg.NotNull(culture, nameof(culture));
    Arg.NotNull(key, nameof(key));
    Arg.Requires<ArgumentException>(! string.IsNullOrWhitespace(key), "! string.IsNullOrWhitespace(key)");
...

Obviously the Arg class would need to be implemented to throw appropriate exceptions, log, etc. The problem with the current state of the world is that a class like Arg above requires the parameter name or a string version of the arg validation expression in order to throw useful exceptions. This requires the developer to provide a string literal parameter EVERY.SINGLE.TIME, which is suboptimal for productivity and maintainability.

An expression tree could be made to work (I think), but that's too slow/high overhead. Doing this sort of translation is what code contracts ccrewrite does via IL rewriting. There are many flaws with post-compile IL rewriting for this use-case; one compelling example is that rewritten async methods couldn't be debugged (microsoft/CodeContracts#483) due to a bug in the IL rewriter. Another is overall build performance - ccrewrite slows down build times by 2-5x (excluding tests). Yet another is that incremental compilation isn't practical if you use code contracts.

The most valuable part of CodeContracts IMO is the ability to specify and inherit contracts from base classes and interfaces. The runtime checking is what I'm after, I can live without Code Contracts-equivalent static checking. Code-gen should be usable for inserting these checks using the validation logic in the base class or interface contract class.

I don't know whether this is covered by #105 or not. Either way, I consider it a great use-case for compiler-integrated codegen.

[asdfgasdfsafgsdfa]

@johncrim
That's an awesome usecase, somehow I didn't think about checking arguments, but that's a cool idea.
Could even be made so you can attach attributes directly to the parameters instead.

void RegisterResource([NotNull]CultureInfo culture, [NotNull("{0} is null but is required")]string key, string value)
or something like that

[fanoI]

Let's say that having code generators is a great feature and that probably the applications of it are not yet imagined probably they are endless!

A lot of things as - for example - pattern matching that now are done in Roslyn (it effectively re-write the code as a series of "if" / "else" / "else if") could been done by us users in libraries!

[KodrAus]

Personally I'm very interested in the serialisation usecase, and something like a type provider that accepts types as inputs would probably do the trick.

Is there an active proposal for generators?

[BreyerW]

Im slighty suprised that one thing wasnt mentioned explicitly which could be enabled after generators: reflection in AOT scenarios. Normally, support for reflection in AOT enviroments is very limited or not supported at all. With generators you could build reflection system similar to runtime-like. Then on top of that, everything that use reflection today, most notably serialization etc. (although if you are after every bit of performance then skipping reflection and using generators directly is better idea).

In long term it could even completely replace current runtime reflection that impose great overhead (not to say that AOT wont have overhead, especially memory one but performance should be better). As far as i know Midori did that and those that worked on that experimental language, were satisfied with results.

So for that alone its worth pursuing, in order to push corert into more mature and useable state.

[krk]

Another use case would be eliminating heap allocation for LINQ statements when necessary info can be deduced in the compile time, as in https://github.com/antiufo/roslyn-linq-rewrite

[alrz]

There's a more comprehensive list here: https://github.com/Fody/Fody#addins-list

I think generators (and companion features) should be able to address most of those one way or another.