Return a IQueryable<dynamic> with real dynamic and not as object.

[KeyserDSoze]

Hi everyone,

I'm creating a core library https://github.com/KeyserDSoze/RystemV3, in my library I add the chance to serialize and deserialize Linq lambda expression for my repository framework library https://github.com/KeyserDSoze/RepositoryFramework to allow the communication over http for query pattern.
Because my repository framework library works with non generic Lambda expression I tried to rewrite all basic methods in Linq, here you may find the solution https://github.com/KeyserDSoze/RystemV3/blob/master/src/Rystem/System.Linq/QueryableLinqExtensions.cs

All worked well until I wanted to add select and groupby to my core library. With groupby I created a work around but with select I found some issues.

  public static IQueryable<dynamic> Select<TSource>(this IQueryable<TSource> source, LambdaExpression selector)
  {
      var value = Generics
          .WithStatic(typeof(QueryableLinqExtensions), nameof(Select), typeof(TSource), selector.ReturnType)
          .Invoke(source, selector) as IQueryable;
      return value!.OfType<dynamic>();
  }

  public static IQueryable<TResult> Select<TSource, TResult>(this IQueryable<TSource> source, LambdaExpression selector)
      => source.CallMethod<TSource, IQueryable<TResult>>(nameof(Select), selector);

The first method Select calls the second method with reflection to inject the return type of lambda expression as generic in Select<TSource, TResult>. The second method calls a third method that calls the linq method Select with a generic lambda expression (always with reflection).
Generics.WithStatic is a method in my library that caches the reflection static method.
In the end I obtain the correct value, in fact, for instance, value for a lambda expression with return type int creates an IQueryable of int.
Now the problem is, how to return a dynamic value? I tried several different ways.
First, with OfType but it transforms the expression in object, for example with a class with Id as int, x => x.Id becomes x => Convert(x.Id, Object) and when the expression is evaluated from entity framework, for example with MaxAsync method it explodes because EF integration can retrieve a Task of int instead of a Task of object, here the exception.

System.ArgumentException : Expression of type 'System.Threading.Tasks.Task`1[System.Int32]' cannot be used for return type 'System.Threading.Tasks.Task`1[System.Object]'

  Stack Trace: 
Expression.ValidateLambdaArgs(Type delegateType, Expression& body, ReadOnlyCollection`1 parameters, String paramName)
Expression.Lambda[TDelegate](Expression body, String name, Boolean tailCall, IEnumerable`1 parameters)
Expression.Lambda[TDelegate](Expression body, Boolean tailCall, IEnumerable`1 parameters)
QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
Database.CompileQuery[TResult](Expression query, Boolean async)
QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
<>c__DisplayClass12_0`1.<ExecuteAsync>b__0()
CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken cancellationToken)
EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, CancellationToken cancellationToken)
EntityFrameworkQueryableExtensions.MaxAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
SystemLinq.Test3() line 210

and here the code (_context is entity of EF and sql is a mssql database)

var selected = _context.Users.Select(newLambda);
var max2 = await selected.MaxAsync().NoContext();

Here the unit-test
https://github.com/KeyserDSoze/RystemV3/blob/master/src/Rystem.Test/Rystem.Test.UnitTest/System.Linq/SystemLinq.cs

Second attempt, with dynamic

  public static dynamic Select<TSource>(this IQueryable<TSource> source, LambdaExpression selector)
  {
      var value = Generics
          .WithStatic(typeof(QueryableLinqExtensions), nameof(Select), typeof(TSource), selector.ReturnType)
          .Invoke(source, selector) as IQueryable;
      return value;
  }

It returns the correct value but when I try to use MaxAsync the runtime doesn't find the method. I don't like this solution because returning a dynamic is a bad practice, but if it worked well I could have pass through, by the way it didn't work properly because the dynamic returns the real type and that type doesn't find MaxAsync method.
Here the error.

Microsoft.CSharp.RuntimeBinder.RuntimeBinderException : 'Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable' does not contain a definition for 'MaxAsync'

So I tried differently, with new approach.
A class that maps an IQueryable without changing the Expression, it doesn't transform x => x.Id in x => Convert(x.Id, Object). A facade for any provider.

    public static IQueryable<object> Select<TSource>(this IQueryable<TSource> source, LambdaExpression selector)
    {
        var value = Generics
            .WithStatic(typeof(QueryableLinqExtensions), nameof(Select), typeof(TSource), selector.ReturnType)
            .Invoke(source, selector) as IQueryable;
        return new SelectQueryable { Queryable = value! };
    }
    public sealed class SelectQueryable : IQueryable<object>
    {
        public IQueryable Queryable { get; init; }
        public Type ElementType => Queryable.ElementType;

        public Expression Expression => Queryable.Expression;

        public IQueryProvider Provider => Queryable.Provider;

        public IEnumerator<object> GetEnumerator()
        {
            return Queryable.OfType<object>().GetEnumerator();
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            return Queryable.GetEnumerator();
        }
    }

But, obviously, the same error happens (I pursued anyway).

 System.ArgumentException : Expression of type 'System.Linq.IQueryable`1[System.Int32]' cannot be used for parameter of type 'System.Linq.IQueryable`1[System.Object]' of method 'System.Object Max[Object](System.Linq.IQueryable`1[System.Object])' (Parameter 'arg0')

  Stack Trace: 
ExpressionUtils.ValidateOneArgument(MethodBase method, ExpressionType nodeKind, Expression arguments, ParameterInfo pi, String methodParamName, String argumentParamName, Int32 index)
Expression.Call(Expression instance, MethodInfo method, Expression arg0)
Expression.Call(Expression instance, MethodInfo method, IEnumerable`1 arguments)
Expression.Call(Expression instance, MethodInfo method, Expression[] arguments)
EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken cancellationToken)
EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, CancellationToken cancellationToken)
EntityFrameworkQueryableExtensions.MaxAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
SystemLinq.Test3() line 210

does a chance exist to pursue what I want to do in C#?

[svick]

with OfType but it transforms the expression in object, for example with a class with Id as int, x => x.Id becomes x => Convert(x.Id, Object) and when the expression is evaluated from entity framework, for example with MaxAsync method it explodes because EF integration can retrieve a Task of int instead of a Task of object,

I'd say the problem is that you are missing the Convert. If I add the following line to your Select<TSource>, then the test will pass for me (using an InMemory database):

selector = Expression.Lambda(
    Expression.Convert(selector.Body, typeof(object)),
    selector.Parameters);

(Also note that I had issues compiling your code, because of the use of Windows 1252 in SystemLinqExpressions.cs. If you want to use non-ASCII characters in your code, I'd suggest using UTF-8 instead.)

[KeyserDSoze]

Thanks a lot, I would have preferred a solution without a conversion in object, but I really appreciated your help. Do you think there is any performance problem with boxing and unboxing when I use int/double or similar?

I will use UTF8 😁, thanks again. I didn't know visual studio saved by default in ANSI.

[CyrusNajmabadi]

You're using expression trees. Everything is going through objects here :)

[svick]

Do you think there is any performance problem with boxing and unboxing when I use int/double or similar?

It depends. If you're talking about something like MaxAsync, then a single boxed int is nothing compared with a database query. If you were boxing thousands of ints per query, that might make a difference.

Also, I would be more worried about the overhead of dynamic than of boxing.