ArgumentNullException in CreateJsonShapers when querying entities with nested OwnsMany inside ToJson() in EF Core 10

[hopkienne]

After upgrading from EF Core 9 to EF Core 10.0.0-rc.2, queries involving entities with nested OwnsMany collections mapped with ToJson() throw an ArgumentNullException during query compilation.

System.ArgumentNullException: Value cannot be null. (Parameter 'key')
at System.Collections.Generic.Dictionary2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary2.set_Item(TKey key, TValue value)
at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.CreateJsonShapers(...)

[achiez]

I am experiencing the same issue with OwnsOne().ToJson() and ComplexProperty(..., x => x.ToJson()) after migrating to EF Core 10.0.1.
Out of curiosity, are you using the EFCore.NamingConventions plugin? After some debugging, it appears that the properties are missing the required code annotations after the model is built. This might be related to an incomplete implementation in that library, but this is only a hypothesis and I don’t yet have direct confirmation.

[roji]

Can you please submit a minimal code sample showing the exception? It's also indeed important to understand whether EFCore.NamingConventions has something to do with this or not.

[mariosimao]

I am also facing the same issue after migrating from EF Core 8.0.2 to 10.0.1.

The problem seems to be on the PriceDetails property.

public class RoomBookingEntityConfiguration : IEntityTypeConfiguration<RoomBooking>
{
    public void Configure(EntityTypeBuilder<RoomBooking> builder)
    {
        builder.HasKey(b => b.Id);
        builder.Property(b => b.Id)
            .HasConversion(x => x.Value, y => new(y))
            .IsRequired();
        builder.Property(b => b.RoomExternalId).IsRequired();
        builder.Property(b => b.TotalGuests).IsRequired();

        builder.ComplexProperty(b => b.Period, p =>
        {
            p.Property(x => x.Start).IsRequired();
            p.Property(x => x.End).IsRequired();
        });

        builder.OwnsMany(b => b.PriceDetails, bu =>
        {
            bu.Property(l => l.PricePerNight).IsRequired();
            bu.OwnsOne(l => l.Period, b =>
            {
                b.Property(p => p.Start).IsRequired();
                b.Property(p => p.End).IsRequired();
            });
            bu.ToJson();
        });
    }
}

My DbContext config:

services.AddDbContext<CoreDbContext>((sp, options) =>
{
    options.UseNpgsql(connectionString, x
        => x.MigrationsHistoryTable("__EFMigrationsHistory", "core"));
    options.UseSnakeCaseNamingConvention();
});

[mariosimao]

@roji I forgot to share the stack trace (merry xmas 🎄)

System.ArgumentNullException: Value cannot be null. (Parameter 'key')
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.CreateJsonShapers(ITypeBase structuralType, Type clrType, Boolean nullable, ParameterExpression jsonReaderDataParameter, Expression keyValuesParameter, Expression containerEntityExpression, IPropertyBase structuralProperty)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.ProcessShaper(Expression shaperExpression, Expression& relationalCommandResolver, IReadOnlyList`1& readerColumns, LambdaExpression& relatedDataLoaders, Int32& collectionId)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.ProcessShaper(Expression shaperExpression, Expression& relationalCommandResolver, IReadOnlyList`1& readerColumns, LambdaExpression& relatedDataLoaders, Int32& collectionId)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitShapedQuery(ShapedQueryExpression shapedQueryExpression)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutorExpression[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass11_0`1.<ExecuteCore>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteCore[TResult](Expression query, Boolean async, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetAsyncEnumerator(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1.GetAsyncEnumerator()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Core.Infrastructure.Persistence.BookingRepository.ListByAsync(HotelId hotelId, Nullable`1 checkinFrom, Nullable`1 checkinTo, Nullable`1 paymentMethod, Nullable`1 status, CancellationToken ct) in C:\Users\mario\Documents\Dev\BookingEngine\Core\Core.Infrastructure\Persistence\BookingRepository.cs:line 63
   at Core.Application.Queries.ListBookings.ListBookingsHandler.Handle(ListBookingsQuery query, CancellationToken ct) in C:\Users\mario\Documents\Dev\BookingEngine\Core\Core.Application\Queries\ListBookings\ListBookingsHandler.cs:line 9
   at Core.Infrastructure.RestApi.ListBookingsAction.Execute(Guid hotelId, Nullable`1 checkinFrom, Nullable`1 checkinTo, Nullable`1 paymentMethod, Nullable`1 status, ClaimsPrincipal user, AuthorizationService auth, ListBookingsHandler handler, CancellationToken ct) in C:\Users\mario\Documents\Dev\BookingEngine\Core\Core.Infrastructure\RestApi\ListBookings.cs:line 36
   at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

[achiez]

In my case, I managed to partially resolve the issue by replacing OwnsOne with ComplexProperty. I'm still not 100% certain what the root cause is, and I haven't been able to produce a perfect minimal repro yet, but I'm actively working on that and continuing to debug.

Below is the evolution of my approach. The first two variants throw 'System.ArgumentNullException', while the third one works.

private void Variant1Owned(ModelBuilder modelBuilder) // System.ArgumentNullException
{
    modelBuilder.Entity<Root>(b => 
    {
        b.HasKey(x => x.Id);

        b.OwnsOne(x => x.Level1, l1 =>
        {
            l1.OwnsOne(x => x.Level2, l2 =>
            {
                l2.Property(x => x.Number);
            });

            l1.ToJson();
        });
    });
}

private void Variant2ComplexTypesWithToJson(ModelBuilder modelBuilder) // System.ArgumentNullException
{
    modelBuilder.Entity<Root>(b =>
    {
        b.HasKey(x => x.Id);
        b.ComplexProperty(x => x.Level1, l1 => // Replace OwnsOne with ComplexProperty
        {
            l1.ComplexProperty(x => x.Level2, l2 =>
            {
                l2.Property(x => x.Number);
                l2.ToJson();
            });
            l1.ToJson();
        });
    });
}

private void Variant3ComplexTypesWithoutToJson(ModelBuilder modelBuilder) // Passes
{
    modelBuilder.Entity<Root>(b =>
    {
        b.HasKey(x => x.Id);
        b.ComplexProperty(x => x.Level1, l1 =>
        {
            l1.ComplexProperty(x => x.Level2, l2 =>
            {
                l2.Property(x => x.Number);
                // l2.ToJson(); // Removing nested ToJson here avoids the exception
            });
            l1.ToJson();
        });
    });
}

Hopefully this helps someone until the issue is resolved at the EF Core level.

[achiez]

@mariosimao

Could you try adding HasJsonPropertyName to the nested owned type?

In my case, this makes the model work correctly:

modelBuilder.Entity<Root>(b =>
{
    b.HasKey(x => x.Id);

    b.OwnsOne(x => x.Level1, l1 =>
    {
        l1.OwnsOne(x => x.Level2, l2 =>
        {
            l2.Property(x => x.Number);
            l2.HasJsonPropertyName("Level2"); // essential
        });

        l1.ToJson();
    });
});

After further investigation, the root cause appears to be related to EFCore.NamingConventions.

When UseSnakeCaseNamingConvention() is enabled, the following code path adds a ContainerColumnName annotation during ownership processing:

https://github.com/efcore/EFCore.NamingConventions/blob/2e2d02e1299feb63db20f0a9b1755809aeef0e31/EFCore.NamingConventions/Internal/NameRewritingConvention.cs#L181

Later, EF Core checks for this annotation when resolving JSON container metadata. If the annotation is present, the condition below returns null:

efcore/src/EFCore.Relational/Extensions/RelationalTypeBaseExtensions.cs

Lines 474 to 478 in 0b1afb4

	return typeBase.FindAnnotation(RelationalAnnotationNames.ContainerColumnName) != null || !typeBase.IsMappedToJson()
	? null
	: typeBase is IReadOnlyEntityType entityType
	? entityType.FindOwnership()!.GetNavigation(pointsToPrincipal: false)!.Name
	: ((IReadOnlyComplexType)typeBase).ComplexProperty.Name;

This null value is then used as a dictionary key further down the call stack, which results in the ArgumentNullException observed in this issue.

efcore/src/EFCore.Relational/Query/RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.cs

Lines 1648 to 1649 in 0b1afb4

	var navigationJsonPropertyName = relatedStructuralType.GetJsonPropertyName()!;
	innerShapersMap[navigationJsonPropertyName] = innerShaper;

As a workaround, explicitly specifying HasJsonPropertyName for nested owned types seems to avoid the problem. Alternatively, removing or adjusting the ContainerColumnName annotation logic in EFCore.NamingConventions also resolves the issue.

@roji
Based on this analysis, the problem does not appear to originate in EF Core itself, but rather in how EFCore.NamingConventions currently interacts with JSON-mapped owned types.

[mariosimao]

Could you try adding HasJsonPropertyName to the nested owned type?

@achiez adding HasJsonPropertyName did the trick and worked as workaround. I spent 3 days trying to fix this bug. This was an awesome christmas gift to be able to solve it. Thanks :)

[mariosimao]

I'm still not 100% certain what the root cause is

I found older issues regarding the Variant2ComplexTypesWithToJson.

#33831 #35616

[hopkienne]

Can you please submit a minimal code sample showing the exception? It's also indeed important to understand whether EFCore.NamingConventions has something to do with this or not.

This is configuration:
modelBuilder.Entity().OwnsOne(x => x.Data, dataBuilder =>
{
dataBuilder.ToJson("data");
dataBuilder.OwnsOne(d => d.Metadata);
dataBuilder.OwnsMany(d => d.Nodes, nodeBuilder =>
{
nodeBuilder.OwnsOne(n => n.Position);
nodeBuilder.OwnsOne(n => n.Dimensions);
});
dataBuilder.OwnsMany(d => d.Edges);
});

and example query that fails:
await dbContext.Set()
.FirstOrDefaultAsync(w => w.Name == "test");

Expected behavior
The query should compile and execute successfully, as it did in EF Core 9.

[hopkienne]

Can you please submit a minimal code sample showing the exception? It's also indeed important to understand whether EFCore.NamingConventions has something to do with this or not.

Entity classes:
public class Workflow : Entity
{
public Guid Id { get; set; }
public string Name { get; set; }
public WorkflowData Data { get; set; } = new();
}

public class WorkflowData
{
public string Name { get; set; } = "";
public WorkflowMetadata Metadata { get; set; } = new();
public List Nodes { get; set; } = [];
public List Edges { get; set; } = [];
}

public class WorkflowNode
{
public string Id { get; set; } = ""; // Note: property named "Id"
public string Name { get; set; } = "";
public Position Position { get; set; }
public Dimensions Dimensions { get; set; }
}

public class WorkflowEdge
{
public string Id { get; set; } = "";
public string Source { get; set; } = "";
public string Target { get; set; } = "";
}

public class Position { public double X { get; set; } public double Y { get; set; } }
public class Dimensions { public double Width { get; set; } public double Height { get; set; } }
public class WorkflowMetadata { public string Version { get; set; } = ""; }

---

Additional context:
The issue occurs during query compilation, not data retrieval
Database contains valid data with no null Id values
The owned types have properties named Id which may be incorrectly interpreted as keys

[roji]

Sorry, but all I'm seeing above are various disconnected and confusing code snippets... What we need is a runnable, minimal console program that can be run as-is, and shows the error happening. Otherwise you're asking us to try and piece together the repro from the partial info above.