Immidate publishing

[Angel-Velkov]

Currently, the outbox pattern provides durability through database persistence plus asynchronous polling. However, there's no built-in way to emit events immediately after scheduling while maintaining durability guarantees without risking duplicate processing.

My question is whether we could have a way to emit the event in the same transaction in which it’s persisted. If that synchronous emission fails, the record would remain in the outbox so that the scheduler can pick it up and retry it afterwards.

[rolandbeisel]

Hey @Angel-Velkov,

thanks for your idea. Let me explain, why this is currently not implemented and will never be.

The Core Problem: Transaction Boundaries

The idea is to emit immediately during the transaction and fall back to polling on failure. This violates the fundamental principle of the Outbox Pattern:

If you emit inside the transaction:

• The event reaches consumers before the transaction commits
• Consumers query the database → data doesn't exist yet (uncommitted)
• If the transaction rolls back after a successful emit → inconsistent state

If you emit after commit:

• The transaction is already closed
• If emit fails, you can't update the outbox record (no active transaction)
• You'd need a second transaction → back to the original problem

Violation of Polling Publisher Principle

The Transactional Outbox Pattern is based on:
"The database is the single source of truth. Only the polling publisher emits events."

This guarantees:

• Atomicity: Business data + outbox record saved together
• No distributed transactions: DB and message broker are decoupled
• Safe retries: Polling can repeat without side effects

The proposal would introduce:

• Two emit paths (sync + polling) → complexity, potential duplicates
• Coupling between transaction success and emit → distributed transaction "lite"
• Race conditions between event delivery and data visibility

Record Ordering
Direct (synchronous) emit cannot guarantee event ordering. Concurrent transactions emit events simultaneously, and the broker receives them in arbitrary order. Additionally, if an event fails and is retried later, it arrives after events that were created after it breaking the sequence.
Polling solves this by using the database as an ordered queue. A single poller reads records sorted by creation time and processes them sequentially per key. Retries happen in place - subsequent events wait until earlier ones succeed. This ensures strict ordering is always maintained.

What You Actually Want: Lower Latency

Instead of synchronous emit, consider:

• Shorter poll interval (100ms instead of 2000ms)
• Adaptive polling (faster when load is high)

The Outbox Pattern deliberately chooses consistency over latency. This is a feature, not a limitation.

Side note on Adaptive Polling in Version 1.1.0:

This feature will be introduced in v1.1.0, scheduled for release in March.

Adaptive Polling dynamically adjusts the polling interval based on workload. It reduces unnecessary database queries during idle periods while maintaining responsiveness under high load.

Benefits:

• Reduced database load – Fewer unnecessary queries during idle periods
• Lower latency – Faster polling when activity is high
• Self-tuning – No manual interval adjustment required
• Resource efficient – Automatically backs off when the system is quiet

[Alek96]

One more note, we did consider supporting immediate delivery after the transaction commits. However, for now, we intentionally decided to keep the current polling-based approach because it provides better simplicity, scalability, and database efficiency.

Why we don’t emit immediately (for now)

Our current design works as follows:

• The outbox event is stored within the same transaction as the business change.
• Each record is assigned to a partition (1–255).
• Each application instance owns a subset of partitions.
• Only one instance and one thread are responsible for processing a given partition.

This guarantees:

• No race conditions
• No need for explicit locking
• Minimal database interactions (select + update)
• Predictable and scalable performance

Because ownership is deterministic (partition-based), we avoid competing consumers entirely.

---

What would change with immediate delivery?

If we introduced immediate post-transaction emission, we would effectively introduce a second concurrent delivery mechanism:

1. The polling job
2. The immediate-delivery thread triggered after transaction commit

These two could compete for the same record. To prevent double processing, we would need an ownership mechanism, for example:

• After transaction commit, spawn an async thread

• That thread attempts to “lock” the record by updating:

  • next_retry_at = now() + timeout
  • version = version + 1
  • where version = event.version

• If the update succeeds → immediate thread owns the record

• If it fails → polling job already owns it

While this is feasible, it introduces:

• Additional update (locking step)
• Extra database round trip
• Optimistic locking logic
• Timeout management
• More complex failure scenarios
• More edge cases to test and maintain

In other words, we move from a clean partition-based ownership model to a competitive ownership model.

---

Trade-offs

Immediate delivery would provide lower latency in the happy path, but:

• It increases database utilization
• It introduces lock-style coordination
• It complicates the concurrency model
• It weakens the elegance of the partition ownership approach

Our current design optimizes for:

• Deterministic ownership
• Horizontal scalability
• Operational simplicity
• Lower database overhead

Given that the polling interval can already be tuned to low latency, we believe the added complexity is not justified at this time.