Cosmos 11.0 CUD batching RequestEntityTooLarge errors #36903
Description
Bug description
On the main branch, (unreleased 11.0), EFCore will throw an DbUpdateException when adding 2 or more entities of total size larger than 2MB in the same partition key. This would succeed on 10.0. It will also succeed when using AutoTransactionBehavior.Never on 11.0 as it will skip batching.
This is because the total request body size limit for both single writes and transactional batches is 2MB. Meaning all writes in a single transactional batch combined can not exceed 2MB.
We could get the current batch entry size as we are serializing to a stream already, and add a safe and/or calculated margin for request metadata. We would most likeley need to calculate the size of some request metadata like id and session token, and add a safe margin for the property serialization. Then EF Core could automatically split the writes into 2 batches, and only throw with AutoTransactionBehavior.Always. However, I am currently not 100% sure what this safe margin should be, but I found this 200 byte margin in the cosmos db source code. Also see: Also see: Azure/azure-cosmos-dotnet-v3#5406. I think if a single entity is larger than 2MB, we should still throw a DbUpdateException. (by sending the request (as a single write))
We could also choose to leave the management of the batch size up to the user. The user would be responsible for adding a SaveChanges call when the request limit might have been reached. However, since ef core internally already knows much more about the request size, it would be nice if it could handle this for the user.
Your code
[ConditionalFact]
public virtual async Task SaveChanges_three_1mb_entries_succeeds()
{
var contextFactory = await InitializeAsync<TransactionalBatchContext>();
using var context = contextFactory.CreateContext();
context.Customers.Add(new Customer { Id = "1", Name = new string('x', 1_000_000), PartitionKey = "1" });
context.Customers.Add(new Customer { Id = "2", Name = new string('x', 1_000_000), PartitionKey = "1" });
context.Customers.Add(new Customer { Id = "3", Name = new string('x', 1_000_000), PartitionKey = "1" });
await context.SaveChangesAsync(); // Fails: DbUpdateException: CosmosException: Response status code does not indicate success: RequestEntityTooLarge
using var assertContext = contextFactory.CreateContext();
var customersCount = await assertContext.Customers.CountAsync();
Assert.Equal(3, customersCount);
}Stack traces
Microsoft.EntityFrameworkCore.DbUpdateException : An error occurred while saving the item with id '1'. See the inner exception for details.
---- Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestEntityTooLarge (413); Substatus: 0; ActivityId: 86a70068-2c8f-4076-a1f9-fafd894c963b; Reason: ({"code":"RequestEntityTooLarge","message":"Message: {\"Errors\":[\"Request size is too large\"]}\r\nActivityId: 86a70068-2c8f-4076-a1f9-fafd894c963b, Request URI: /apps/DocDbApp/services/DocDbServer20/partitions/a4cb4960-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \r\nRequestStartTime: 2025-10-02T08:02:20.0451545Z, RequestEndTime: 2025-10-02T08:02:20.0520695Z, Number of regions attempted:1\r\n{\"systemHistory\":[{\"dateUtc\":\"2025-10-02T08:01:23.5087928Z\",\"cpu\":7.172,\"memory\":26977176.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0718,\"availableThreads\":32765,\"minThreads\":16,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":1},{\"dateUtc\":\"2025-10-02T08:01:33.5241220Z\",\"cpu\":18.503,\"memory\":26808748.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.032,\"availableThreads\":32765,\"minThreads\":16,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":1},{\"dateUtc\":\"2025-10-02T08:01:43.5298142Z\",\"cpu\":13.473,\"memory\":26748008.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.1671,\"availableThreads\":32765,\"minThreads\":16,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":1},{\"dateUtc\":\"2025-10-02T08:01:53.5354127Z\",\"cpu\":10.751,\"memory\":26639312.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.3375,\"availableThreads\":32765,\"minThreads\":16,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":1},{\"dateUtc\":\"2025-10-02T08:02:03.5497815Z\",\"cpu\":51.983,\"memory\":26308516.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.4465,\"availableThreads\":32765,\"minThreads\":16,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":1},{\"dateUtc\":\"2025-10-02T08:02:13.5647290Z\",\"cpu\":19.250,\"memory\":26024224.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0572,\"availableThreads\":32765,\"minThreads\":16,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":1}]}\r\nRequestStart: 2025-10-02T08:02:20.0454558Z; ResponseTime: 2025-10-02T08:02:20.0520695Z; StoreResult: StorePhysicalAddress: rntbd://127.0.0.1:10253/apps/DocDbApp/services/DocDbServer20/partitions/a4cb4960-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 413, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: , ActivityId: 86a70068-2c8f-4076-a1f9-fafd894c963b, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 30/09/2025 13:28:29)], TransportRequestTimeline: {\"requestTimeline\":[{\"event\": \"Created\", \"startTimeUtc\": \"2025-10-02T08:02:20.0454579Z\", \"durationInMs\": 0.0317},{\"event\": \"ChannelAcquisitionStarted\", \"startTimeUtc\": \"2025-10-02T08:02:20.0454896Z\", \"durationInMs\": 0.004},{\"event\": \"Pipelined\", \"startTimeUtc\": \"2025-10-02T08:02:20.0454936Z\", \"durationInMs\": 6.0733},{\"event\": \"Transit Time\", \"startTimeUtc\": \"2025-10-02T08:02:20.0515669Z\", \"durationInMs\": 0.1215},{\"event\": \"Received\", \"startTimeUtc\": \"2025-10-02T08:02:20.0516884Z\", \"durationInMs\": 0.0722},{\"event\": \"Completed\", \"startTimeUtc\": \"2025-10-02T08:02:20.0517606Z\", \"durationInMs\": 0}],\"serviceEndpointStats\":{\"inflightRequests\":1,\"openConnections\":1},\"connectionStats\":{\"waitforConnectionInit\":\"False\",\"callsPendingReceive\":0,\"lastSendAttempt\":\"2025-10-02T08:02:19.9980359Z\",\"lastSend\":\"2025-10-02T08:02:19.9980455Z\",\"lastReceive\":\"2025-10-02T08:02:19.9991617Z\"},\"requestSizeInBytes\":3000849,\"requestBodySizeInBytes\":3000359,\"responseMetadataSizeInBytes\":23,\"responseBodySizeInBytes\":40};\r\n ResourceType: Document, OperationType: Batch\r\n, SDK: Microsoft.Azure.Documents.Common/2.14.0"}
RequestUri: https://127.0.0.1:8081/dbs/CosmosTransactionalBatchTest/colls/TransactionalBatchContext/docs;
RequestMethod: POST;
Header: Authorization Length: 80;
Header: x-ms-date Length: 29;
Header: x-ms-documentdb-partitionkey Length: 5;
Header: x-ms-cosmos-batch-atomic Length: 4;
Header: x-ms-cosmos-batch-ordered Length: 4;
Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1;
Header: x-ms-cosmos-is-batch-request Length: 4;
Header: x-ms-activity-id Length: 36;
Header: Cache-Control Length: 8;
Header: User-Agent Length: 153;
Header: x-ms-version Length: 10;
Header: Accept Length: 16;
, Request URI: /dbs/CosmosTransactionalBatchTest/colls/TransactionalBatchContext/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.26100 cosmos-netstandard-sdk/3.38.0);
Stack Trace:
CosmosDatabaseWrapper.SaveChangesAsync(IList`1 entries, CancellationToken cancellationToken) line 164
StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) line 1335
StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) line 1429
<<ExecuteAsync>b__0>d.MoveNext() line 307
--- End of stack trace from previous location ---
ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken) line 330
ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken) line 354
ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) line 306
DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) line 787
DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) line 822
CosmosTransactionalBatchTest.SaveChanges_three_1mb_entries_succeeds() line 267
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
Verbose output
EF Core version
11.0.0
Database provider
Microsoft.EntityFrameworkCore.Cosmos
Target framework
10.0-rc1
Operating system
Windows 11
Related: #17308