Proposal: extend the set of things allowed in expression trees

[gafter]

Expression trees support only a subset of the things that could be semantically meaningful. Consider allowing them to support the following constructs:

• The null-coalescing operators such as a?.b, a?[b]
• Expressions involving dynamic
• Invocations that use default or named parameters
• Assignment and other mutation expressions
• A statement-bodied lambda

[Mr-Byte]

I think allowing the null operators in expression trees would be one of the most useful, especially in regards to frameworks that provide LINQ support for databases.

[jdh28]

To add to your list, even subscription and unsubscription.

[VictorBlomberg]

I believe that supporting statement-bodied lambdas would make expression trees a lot more interesting in many real world scenarios. I posted to the Visual Studio UserVoice site a while back, and it seemed like there is some interest in this.

[olmobrutall]

+1 for enabling ?. in expression trees. I did a pull request some months ago (https://roslyn.codeplex.com/SourceControl/network/forks/Olmo/NullPropagationInExpression/contribution/7700) and discuses the idea (https://roslyn.codeplex.com/discussions/571077). It's not complicated but tell me if you're interested in and updated version.

[olmobrutall]

Also Invocations that use default or named parameters in expression trees will be nice :)

[kbirger]

I don't think that dynamic would be possible, because it is handled runtime, which would prevent the ability for the expressions to be handled during compilation.

[D3-LucaPiombino]

I am also very much interested in allowing statement-body like @VictorBlomberg for dsl purposes.
It would open some interesting scenarios in manipulating expression trees that right now are a pain.

[iSynaptic]

We now have many different ways to represent code as an object model - CodeDom, LINQ Expression Trees, and now Roslyn. Since Roslyn is the most complete of all the models (and will continue to be so) is there anyway it could become the basis for lambda expressions going forward? Perhaps by introducing something like a RoslynExpression (name not important) in order to support backward comparability by not changing Expression (and ecosystem). This way there isn't a continuous need to create parallel representations of lambda expressions to support new language features.

That said, a couple things jump out as making this a less ideal approach:

• New model based on Roslyn syntax trees would not be language agnostic
• Differences between C# and VB syntax trees would make consumption a little more involved
• Existing code that uses Expression and ecosystem would require much more rework to take advantage of the new model

Thoughts?

[svick]

@iSynaptic I wouldn't consider the CodeDOM object model for any new code. It doesn't support even some of the most basic things (like static classes).

And regarding Roslyn and Expression trees, I would actually like to see expression trees expand to be able to do more things (e.g. that you could generate whole types using them), not the other way around, like you're proposing. I think that Expression trees and Roslyn syntax trees have very different primary use cases and because of that there are significant differences between them. And those differences make the kind of metaprogramming you want to do with Expression trees much harder to do with Roslyn.

Some of the differences include:

• Roslyn can precisely encode any C#/VB source, including whitespace, comments and invalid code, which makes it much more complex
• Roslyn has separate syntax and semantic layers and neither works with types that you can manipulate at runtime (like Type or MethodInfo)
• Roslyn is language-specific, Expression trees work the same in C#, VB and F#

[hivanov]

I probably think the Expression Trees have the following huge weaknesses:

1. Not reclaimable by GC. Expression.Compile() and friends tend to eat up your memory, thus a special caching logic is needed all the time;
2. Not suitable for writing types -- only static methods are supported. This is probably pain number one. Even if you could compile them into Type methods, the generated methods are still static. This prevents proper proxying of some classes (think Castle.DynamicProxy and friends on steroids). The result is having custom boilerplate to handle class state. There is a whole host of problems that could be solved by expression trees type generation. Think for example:

• Code that optimizes itself on-the-fly based on configuration parameters;
• Custom language parsers (in combination with ANTLR and friends);
• Dynamic code generation (generic interface implementation, handy for, say, network communication).

[mkosieradzki]

@hivanov On the other hand great thing about Expressions: it's currently the only way to run "dynamic code" (without writing own interpreter) inside Windows Store Applications as as far as I understand correctly they are even .NET Native compatible. If tendency is to go towards AOT instead of JIT: dynamic types are probably not coming back.

Regarding GC issues: I am using a lot of Expression.Compile and never had memory issues with them (maybe due to the caching logic) - I must check this out.

And of course: +1 for "null-coalescing operators".

[axel-habermaier]

@mkosieradzki: Regarding System.Linq.Expressions.Expression in .NET Native: Yes, as far as I'm aware, they are supported, albeit not in a compiled way (no runtime code generation is possible), but in a slow, interpreted way.

[tmat]

@axel-habermaier Turns out that the interpreted way is actually faster than the compiled way if the expression is executed less than ~50 times.

[mkosieradzki]

It's not slow. It's good-enough for scenarios I am using it for (scripting inside application).

@hivanov
I am unable to reproduce memory leak behavior when using Compile. I have tried to create millions of different expressions, evaluate them and have literally no memory leaking. I would expect at least one "byte" leaking per expression but after GC memory returns to the initial usage. Are you sure there is a memory leak issue related to Expressions?

[hivanov]

@mkosieradzki
It actually depends on the expressions you build. I have found out, on numerous occasions, that expressions that reference external stuff (mostly, mapped external variables) fail to be released even if the referencing code + the variables are no longer reachable. Especially if the variables are something like channel description.

I still stand behind my general idea for the extension of the Expression Trees to be able to generate whole classes or, at least, methods that support "this", so they could be used in TypeBuilders.

[CyrusNajmabadi]

and break the existing source generators because they don't recognize the new nodes. This is the same with the expression tree.

They're explicitly different situations. One started out in a world where it wsa not changing, and effectively had a contract over time that it would not break (and, indeed, breaks happened and had to be rolled back because it was a significant issue in practice). The other started in a world where we said explicitly it would change and change often.

If we did expression trees a new time, we've said we would go the roslyn model. But that's a new system, not an extension to the existing one.

[Eli-Black-Work]

To everyone commenting here: You might also want to check out #4727, which contains more discussion about how a feature like this might be implemented 🙂

[IS4Code]

and break the existing source generators because they don't recognize the new nodes. This is the same with the expression tree.

They're explicitly different situations. One started out in a world where it wsa not changing, and effectively had a contract over time that it would not break (and, indeed, breaks happened and had to be rolled back because it was a significant issue in practice). The other started in a world where we said explicitly it would change and change often.

If we did expression trees a new time, we've said we would go the roslyn model. But that's a new system, not an extension to the existing one.

There are many solutions that don't cause such issues. Taking use of existing expression types that have been there for 10 years is a good start. Then, if you want to start anew, why not allowing to bind a lambda to the Roslyn model, in addition to Expression<T>?

By the way, what kind of issues did the addition of new expression types in .NET 4.0 cause, and how were they solved?

[bernd5]

The problem is not the existence of new expression types (some where added which are not touched by the compiler).
The problem is that it is expected from the compiler to apply the language semantics to the generated tree - not just the syntax. The compiler handles for example implicit conversions, capturing, overload resolution...

For consumers this is quite great because it avoids the need to handle all that...
But for the compiler it becomes quite hard.

[aradalvand]

Now almost 8 years since this issue was created, numerous new features have been added to C#, yet expressions are still stuck in C# 3.0 or something. This is simply unacceptable at this point. I fail to see why this is not being prioritized or getting any traction whatsoever, given that it's becoming an increasingly evident pain point. Proposals like #4727 also seem to have been effectively abandoned.

Any updates? cc @CyrusNajmabadi, @tmat, @alrz, @MadsTorgersen

[CyrusNajmabadi]

I fail to see why this is not being prioritized or getting any traction whatsoever

Other things have been more important, and have also not have the significant concerns and constraints present here.

Any updates? cc @CyrusNajmabadi, ..

No. THere have been no updates. Any updates would be posted here as we do all this development in the open. :)

Note: you're welcome to come discuss things more with the team on Discord if you'd like! :)

[Tragetaschen]

More than once I missed the ability to compose expressions or have expression with "holes" to plug in another expression. Especially in the context of Entity Framework I have a lot of cases where queries have common sub expressions, but I cannot extract those into a reusable expression. As a very simplistic example, think

public static IQueryable<T> MySpecialFilter<T>(this IQueryable<T> queryable, Expression<Func<T, bool>> predicate)
{
    return queryable.Where(x => x != null && predicate(x)); // does not compile, because expressions cannot be "called"
}

I always naively assumed that this kind thing should be easily expressible in the tree object model by stuffing predicate at the correct place instead of creating that particular subtree.

[FaustVX]

@Tragetaschen
You have to call predicate.Compile()(x)
But expression are not meant to be executed.

[olmobrutall]

More than once I missed the ability to compose expressions or have expression with "holes" to plug in another expression. Especially in the context of Entity Framework I have a lot of cases where queries have common sub expressions, but I cannot extract those into a reusable expression. As a very simplistic example, think

public static IQueryable<T> MySpecialFilter<T>(this IQueryable<T> queryable, Expression<Func<T, bool>> predicate)
{
    return queryable.Where(x => x != null && predicate(x)); // does not compile, because expressions cannot be "called"
}

I always naively assumed that this kind thing should be easily expressible in the tree object model by stuffing predicate at the correct place instead of creating that particular subtree.

Hallo @Tragetaschen
Maybe this links help:
https://github.com/signumsoftware/framework/blob/master/Signum.Utilities/ExpressionTrees/LinqExtensibility.md
https://github.com/signumsoftware/framework/blob/2540317460bd5460fcd155876cc94f8ffa1cab81/Signum.Utilities/Extensions/ExpressionExtensions.cs#L48
https://github.com/signumsoftware/framework/blob/2540317460bd5460fcd155876cc94f8ffa1cab81/Signum.Utilities/ExpressionTrees/ExpressionCleaner.cs#L60
https://tomasp.net/blog/linq-expand.aspx/

[svick]

@olmobrutall Or just use LINQKit (NuGet).

[aradalvand]

Having just re-read this thread and also this one, all the while carefully considering the concerns and explanations provided by the likes of @CyrusNajmabadi, @333fred, et al. I must say they are making about as much sense as how many "new" C# syntactic features have had support added for them in expression trees over the last decade or so, which is, of course, a glorious zero.

Let's consider this comment by @333fred — as I think it succinctly and clearly presents the supposed justification:

The significant cost of implementing new nodes does not come from the compiler. It's fairly easy to do in the compiler. Its significant cost comes from every downstream tool now being broken because there is a new node in the tree they were not expecting, and us needing to work with EF and other consumers of expressions to create a way to signal language version support.

(emphasis mine)

Sounds reasonable, right? Except that it should take no more than 5 seconds for anyone remotely familiar with C#/.NET's and the surrounding ecosystem's progression over the last couple of years, to realize two things, which immediately obliterate this reasoning:

Swap the terms "new node" with "new built-in anything" and you'll get an equally sound argument for never adding any feature at any level in any area to the framework/standard library ever; because "what about existing tools that won't immediately support it?", I guess. It is unbelievably unconvincing.

There is literally a fundamental analogy here between any hypothetical new expression nodes, and any new standard constructs in general — with the most notable recent example being what happened with TimeOnly and DateOnly: They got added to the standard library, tools like EF, various serializers, etc. didn't immediately recognize/support them, so any usage in such unsupported contexts resulted in either a runtime exception or unexpected behavior at the time, until support gradually got added. How would new expression nodes be fundamentally ANY different than that?

Notice that according to the same logic as the one that we're being provided here as to why expressions are stuck at the C# of 10 years ago, apparently the addition of TimeOnly and DateOnly would've been a heinous, unthinkably nightmarish idea, because:

The significant cost of implementing [TimeOnly and DateOnly] does not come from the [standard library]. It's fairly easy to do in the [standard library]. Its significant cost comes from every downstream tool now being broken because there is a new [built-in type] in the [framework's standard library] they were not expecting, and us needing to work with EF and other consumers of [user code] to create a way to signal [framework] version support.

It gets even worse, this argument is, in fact, even LESS applicable in the case of expression trees, because all consumers of expression trees (most prominently EF), ALREADY ONLY support a subset of "all possible expression trees". An expression tree consumer not supporting a certain thing at a given time is already the status quo, and as such, fully expected.

Funny. Some people in the other thread (e.g. @olmobrutall here) tried to point out the same thing, heck, even some comments on related Stack Overflow questions made this point, and made more basic logical sense than anyone from the other side here. It's truly astonishing.

[IS4Code]

I don't think the argument here is simply "they cannot be added because that will break existing libraries", it is "how to add them in a way that both preserves the original meaning and interoperates them with existing code".

You could rewrite x.a == 1 && x.b != null into x is { a: 1, b: not null } expecting it to work, but the latter would result in a different expression, even though the change should not affect the outcome.

That being said, there is a myriad of currently unused expression types that are not simply a lowering of other expressions, like DynamicExpression, so those are "safe" because there are no other choices for the representation.

[xamir82]

You could rewrite x.a == 1 && x.b != null into x is { a: 1, b: not null } expecting it to work

But those are still two different expressions (even if they might be functionally the same); so it's only natural that they're represented by two distinct expression node types. In normal code, the compiler might lower one to the other, yes, but that's just an implementation detail of the compiler and fundamentally not relevant to this conversation.

Expressions reflect what's on the "surface", so to speak, just as you can't just replace (x, y) => x + y with (x, y) => Sum(x, y) in an expression tree (where Sum is just int Sum(int x, int y) => x + y) and "expect it to work". Even though if they were evaluated they'd be functionally equivalent, functional equivalence is not what determines expression node types.

[IS4Code]

@xamir82 It is relevant ‒ it is not always apparent whether a lambda is treated as an expression, and someone might easily rewrite it as a part of modernizing the code. Even x is null is different from x == null, but not as much as changing it to a method (unless you are in Unity or somewhere else where == is overloaded for null testing). I don't personally think that it is such a big deal, since you could easily have an analyzer to let you know if this kind of expression is given to a target that might not support it (attributes to express that would be nice altogether), and the focus now should be on .NET 4.0 types anyway where this is not an issue. Moreover, there is Expression.Reduce which should do exactly what is needed to deal with those issues.

[magicxor]

It would be very beneficial to allow C# switch expressions to be represented within System.Linq.Expressions.

Currently, building expression trees that require multi-way conditional logic often necessitates using deeply nested ternary conditional operators (?:). This approach becomes complex and difficult to read and maintain as the number of conditions increases.

    public static Expression<Func<ActionLog, string?>> GetRelatedEntityValue(ShardContext shard)
    {
        return a => a.RelatedEntityName == ObjectType.Appointment
            ? shard.Appointments.Where(x => x.AppointmentId == a.RelatedEntityId).Select(x => x.AppointmentId).FirstOrDefault().ToString()
            : a.RelatedEntityName == ObjectType.Campaign
                ? shard.Campaigns.Where(x => x.CampaignId == a.RelatedEntityId).Select(x => x.UniqueId).FirstOrDefault()
                : a.RelatedEntityName == ObjectType.ClientFile
                    ? shard.ClientFiles.Where(x => x.ClientFileId == a.RelatedEntityId).Select(x => x.JobId).FirstOrDefault()
                    // and many more levels...
                    : null;
    }