Clarification around EF transactions for outbox in v10

[jakoss]

I'm working on this issue: #3879 . I have implementation ready, but when i test it - throwing inside transaction like this:

await testRepository.RunInTransaction(async token =>
{
    await testRepository.Create(entity, cancellationToken: token);

    await messageBrokerProcessor.DepositPostAsync(new MessageBrokerTestEvent
    {
        Name = request.Name
    }, cancellationToken: token);
    
    throw new InvalidOperationException("Test exception");
}, cancellationToken: context.CancellationToken);

does not cancel the outbox entry - it is send to message broker. I'm trying to poke around the code to find out how automatic the transaction handling should be, but i cannot find anything specific.

Documentation here: https://brightercommand.gitbook.io/paramore-brighter-documentation/guaranteed-at-least-once/efcoreoutbox suggests that all i need to do is to create transaction from dbContext and Post message inside. But it's clearly not working for me, i've added breakpoints inside my EFTransactionProvider and it's not being called anywhere.

Is that a bug or maybe i need to use transaction explicitly, because this sample suggests that i should inject and use IAmATransactionConnectionProvider directly: https://github.com/BrighterCommand/Brighter/blob/master/samples/WebAPI/WebAPI_EFCore/GreetingsApp/Handlers/AddGreetingHandlerAsync.cs . But i think that would completely defeat the purpose of EF transaction provider, does it not?

[lillo42]

Maybe @iancooper can help you on this

[jakoss]

@iancooper Could you find some time to comment on this? I think this is quite important, because there is no official guideline to EF integration

[iancooper]

Let me take a look

[iancooper]

I am working on some updates for V10 documentation, hopefully it will help.

Some assumptions that we make:

• We are a framework, so we don't know what database you are using for your Outbox. We have to be told.
• We assume that we write to the Outbox, not you. (In the case of EF Core, this means the Outbox is not part of your DbContext)
• To write to the Outbox we may need to share the connection that you have created a transaction on, to write your entities

To get hold of the context for your transaction we abstract your transaction behind the transaction provider. There are tailored to the persistence method you are using (EF Core, Dapper, DynamoDb)

We recommend you to make that a dependency of your handler, so that it is injected into your handler (we do this work for you, once you tell us what it will be at configuration).

But, what you must do is pass the transaction provider to us via the DepositPost call. (In principle, you could create an instance of the transaction provider, there in the handler, and pass that in, over using DI, YMMV on which works better).

(The version of DepositPost that does not take a transaction provider supports you just defaulting to an in-memory outbox)

Once you pass it in, we can use its connection to write to our Outbox, safely within your transaction. (In principle, with something like DynamoDb, we could just use a global connection as there is no transaction-connection affinity, but we hit problems with this in V10, and most likely won't re-address until V11)

So, the key here is that you must pass the transaction provider to DepositPost so that we can use your connection, and an easy way to do that is via DI.

[jakoss]

Ok then, but if i can just inject the transaction provider myself and pass it to DepositPost - could we inject it in the default command processor and do that for the user? Is there anything that stops us here?

[iancooper]

Generally, the pain point here became "scope". EF requires that we use a scoped lifetime. That means everything that depends on that transaction provider has to use a scoped lifetime.

A scoped command processor creates pain points around caching pipelines and producers. Now we can offload those to something that holds the singleton, etc. But generally, because we are often downstream of ASP.NET, with no control over the scope, we don't benefit from that scope in the same way that ASP.NET does; it is just propagated to us. It is much easier for us to declare the handlers as having a scoped lifetime and take that dependency.

We tried the "just inject into the CP strategy," but our experience showed that the trade-offs were worse, so we reverted to a singleton CP due to the complexity associated with it.

We also don't know that you want to use a transaction for a given flow, so auto-injecting it in every deposit call would mean we are creating transactions to write to the Outbox when you didn't necessarily intend to do so.

Obviously, the ideal is that this is mostly invisible to you; we magically participate.

One option might be to create a new base class. So make an OutboxRequestHandler that exposes a DepositPost method that takes the injected transaction provider and adds it to the call to the command processor. It is a convenience approach over a fundamental shift, if folks might find that easier to work with.

[jakoss]

Ok, that makes sense. Creating my "wrapper" for command processor that's scoped should fix the issue

[jakoss]

@iancooper I'm trying to implement the wrapper, but i stumbled across the issue somewhat related to this: #3878 . Since we have shared messaging library using Brighter that is used on multiple microservices - some of those will run on postgres and some on mongo.

I created the mongo transaction provider, but i had to set DbTransaction generic parameter to IClientSessionHandle since it has to be compatible with the mongodb outbox provider. But the mongodb ef provider could provide the DbTransaction itself no issues, the provider itself is wrapping the IClientSessionsHandle. Even more, i can get the handle in GetTransaction() by using internal property of the mongodb ef provider, which makes the implementation brittle. Is there a way to have different DbTransaction generic type for the base mongodb outbox provider and the EF provider?

[iancooper]

Top of my head, we probably need to provide an EF wrapper for MongoDb, the same way that we separate between using an RDBMS connection through ADO.NET for tools like Dapr and using it through EF. Pulling in @lillo42 who did our MongoDb implementation to check my thinking

[jakoss]

I'll create a PR draft with what i've got so far. It works, but it does rely on the internal api from the MongoDB EF provider. It should be easier to talk over the problem