Question: Why it is not allowed to mix :any and integer version?

[volodymyr-mykhailyk]

While writing my event sourced app encountered this limitation (inability to mix :any and integer version). Setting aside implementation of gem - on conceptual level - why you designed such limitation?

In my mind from business prespective - it is ok to have a stream that stores a collection of random events related to one entity while there is certain point in a lifecycle of this entity I want to synchronize (lock) across threads.

So my assumption was that I can publish events with any, and then at some point get last event from a stream and use it position as optimistic locking mechanism to prevent concurrent updates of the entity. I know I can have a separate stream for such synchronization but I hesitate to pollute my app with dozen/hundreds of streams

[fidel]

Hi @volodymyr-mykhailyk, thanks for the question!

The prohibition against mixing :any and integer versions exists because these modes have fundamentally incompatible concurrency guarantees that would break in unpredictable ways if combined.

The problem

When you use :any:

• All concurrent writes succeed
• Events may arrive in non-deterministic order
• Stream position becomes unreliable for optimistic locking

When you use integer version:

• Only one concurrent write succeeds (optimistic locking)
• Events are in strict, deterministic order
• Stream position is a reliable consistency boundary

If you mix them in one stream:

1. Your optimistic locking breaks — :any writes change the stream position without participating in the lock, so your integer-based writes will fail randomly when :any events interleave
2. Ordering becomes ambiguous — You lose the guarantee that events are in deterministic order, which is critical if you're rebuilding state from the stream
3. Concurrent conflicts become invisible — You can't tell if a WrongExpectedEventVersion is a legitimate business conflict or just interference from :any writes

Your use case

I can publish events with any, and then at some point get last event from a stream and use its position as optimistic locking

This approach has a race condition: between reading the last position and writing with that position, other :any writes can succeed, making your "last position" stale. Your integer write will then fail, but not because of a real conflict — just because of :any interference.

Better alternatives

1. Use integer version for the entire stream — If you need synchronization at certain points, you likely need it everywhere. Consider using integer version throughout and handling WrongExpectedEventVersion appropriately.

2. Separate concerns into different streams — This isn't "pollution" — it's proper bounded context separation. Streams are cheap — they're just logical groupings with essentially no overhead. If you have "random events" and "synchronized lifecycle events" for an entity, these might actually be different concerns that belong in different streams. Having dozens or hundreds of streams is perfectly normal and expected in event-sourced applications.

3. Application-level locking with :auto — Use :auto with explicit application locks (mutex, database lock) around the entire operation when you need synchronization.

Could you share more about your specific use case? What kinds of events are you publishing with :any, and what synchronization points do you need? There might be a better domain model that addresses this.

[volodymyr-mykhailyk]

@fidel Thanks for explanation. So far I selected usage of separate streams for synchronized usecases.

Little bit more about usecase: Goal of the project is to reconstruct state of the insurance policy based on transactions and events we receive from external sources. Essentially there is one aggregate - "policy".

In order to reconstruct correct state we use valid_at as events from external systems are not arriving in correct order and always have delivery delays.

Majority of events are recorded "as is" and are linked to the policy stream with :any expected version. However in certain cases when external events are transformed/handled - they will result in internal "system" event. For example - we associate policies between two external systems (system 1 id = system 2 id). For this usecase - I want to make sure there is only one association event, even if there are multiple events at the same time telling about same association. Thats when I encountered this limitation when I tried to use integer version on main policy stream.

For my use case optimistic locking is good enough and regardless of reason for WrongExpectedEventVersion exception. I simply retry my handler on this exception:

  Get All Aggregate Events
  Exit if Aggregate already Associated (another association event published)
  Publish Association event with expected version = (All Events).size
  Rescue Wrong Version and Retry from beginning

[volodymyr-mykhailyk]

Anyway I see your point and will proceed with separate "synchronization queues".

The real reason I went this rabit hole is not informative exception and this limitation written in very small font in the docs here https://railseventstore.org/docs/core-concepts/expected-version.

Here is example that sent me chasing wrong problem:

  it 'position in stream equals to number of events' do
    5.times do
      event_store.publish(RailsEventStore::Event.new, stream_name: 'test', expected_version: :any)
    end
    events = event_store.read.stream('test')
    expect(event_store.position_in_stream(events.last, 'test')).to eq(5)
  end

The problem is that in memory store returns nil for position in stream. I suspect activerecord store will do the same. So at first I thought there was a mistake in the code.

Closing the issue anyway, you helped me to better understand your reasoning

[fidel]

Thanks for sharing more context! Your use case makes a lot of sense now.

Your retry pattern is solid

Your approach with the retry loop is exactly right for this kind of idempotency check:

Get All Aggregate Events
Exit if Aggregate already Associated
Publish Association event with expected version = (All Events).size
Rescue Wrong Version and Retry

This pattern works well when you need exactly-once semantics for specific events within a larger event stream.

Consider separating technical and domain events

Given that you're receiving events from external systems with delivery delays and processing them into internal events, you might benefit from the pattern we describe in Handling third-party webhooks with Rails Event Store, it's also described more briefly in the docs:

1. Technical events (:any version) — Store raw payloads as received: ExternalSystem1EventReceived, ExternalSystem2EventReceived
2. Domain events (integer version when needed) — Your processed events like PolicyAssociated

Benefits for your use case:

• Audit log — Keep all external events exactly as received for debugging and compliance
• Reprocessability — If your association logic changes, reprocess from technical events
• Fast responses — Acknowledge external events immediately, process async
• Clear separation — External events (:any) vs. internal synchronized events (integer version)

Your association logic would become:

# Webhook receives external event
publish_technical_event(ExternalSystemEventReceived, payload, expected_version: :any)

# Async handler processes it
handle(ExternalSystemEventReceived) do |event|
  # Your retry logic here for PolicyAssociated with integer version
end

About position_in_stream returning nil

This is actually expected behavior when using :any — since concurrent events can be written in non-deterministic order, there's no meaningful "position" concept. The events exist in the stream, but their relative ordering isn't guaranteed, so returning a position number would be misleading.

With integer version, position is meaningful because you have strict ordering guarantees. This is another reason why mixing the two modes would be problematic — what would position mean for :any events interspersed with integer-versioned ones?

Bi-temporal event sourcing consideration

Your use case with valid_at timestamps and out-of-order external events is actually a textbook example of bi-temporal event sourcing — you're tracking both when events happened in the real world (valid_at) and when you learned about them. RES has built-in support for this pattern: https://railseventstore.org/docs/advanced-topics/bi-temporal/

You might find that approach cleaner than relying solely on linking.

Separate streams is the right call

For your association events, separate streams are definitely the way to go. You could have:

• Policy-{id} stream with :any for all incoming external events
• PolicyAssociation-{id} stream with integer version for your system events

This gives you clean separation of concerns and makes it explicit which events need synchronization guarantees.

Thanks for the detailed explanation of your use case — it's helpful to understand how people are using RES in the wild. Good luck with your insurance policy reconstruction project!

PS. PRs improving our docs are also welcome 😄