Standardize Stellar Event IDs

[chowbao]

Proposal

I'd like to have standardized event IDs for Stellar events similarly to how how there is a standardized operation ID in SEP-35

Motivation

It is useful for DB system checks to have a unique ID associated with Stellar events such as for deduplication. This event ID should not be exposed to downstream users unless this discussion provides strong evidence that the event IDs should be exposed to users.

Real user cases from the ecosystem:

An ecosystem partner (Dune) has already requested some sort of event ID for their ingestion of Stellar events into the history_contract_events table.

@tomerweller for his mini-indexer

I used the same event id [as Stellar RPC] is that it allowed me to have claude compare the databases to verify that I was indexing what I thought I was indexing

Requirements

The main requirement is that the event IDs are unique, ordered, and deterministic

Caveats and Concerns

Event Index

AFAIK there isn't a way to give events ordered deterministic unique IDs without the use of an event index.

For example, a smart contract may emit events that are identical but are each legitimate events. This is the case when say within the same operation userA sends 10 XLM to userB twice which would then create two {type:transfer, from:userA, to:userB, asset:XLM, amount:10} that would have the same operation TOID

This could cause problems in the case the event index changes either through new events being emitted whether through a protocol upgrade that adds new historical events or in the case of the state archival incident where new events were added to cover the discrepancy between state and events.

It's not likely but not impossible for event index changes to happen in the future. But because of the requirements and user cases listed above I don't think needing to regenerate or reingest different event IDs is a problem because the purpose of the event IDs are for backend DB integrity checks and NOT for downstream users to actually use the event IDs.

Diagnostic Events and Contract Events

When diagnostic events are enabled there is a whole other stream of events. Diagnostic events also contain contract events. So contract events will in theory be located in two spots within the ledger metadata.

One problem this can cause is the exact same contract event could possibly have 2 different event IDs. One from the index within contract events and one from index of diagnostic events

The proposed fix for this is to add a type character to the event ID to signify whether the event comes from contract events or diagnostic events.

Event ID Format

<TOID of ledger>-<type; either 'c' or 'd'>-<event index>

• <TOID of ledger> refers to TOID called with (ledger sequence, 0, 0)
• <type> for either
  • c for contract events
  • d for diagnostic events
    • Note as mentioned above, diagnostic events also have the same contract events in the diagnostic events part of the ledger close meta. This means that the same contract event will have two different event IDs (one with c type and one with d type if diagnostic events are enabled
• <event index> is the correctly ordered events
  • Order of events is defined as
    1. Pre-application events
    2. Operation events
    3. Post-transaction events
    4. Post-application events

Alternative - Stellar RPC Event ID

Alternatively, we can use Stellar RPC's current event ID definition defined here

This would effectively be <TOID>-<event index> with the following TOID definitions

	//  - Pre-application events have a TOID with  { ledger seq, 0, 0 }
	//  - Operation events have a TOID with        { ledger seq, tx index, op index }
	//  - Post-transaction events have a TOID with { ledger seq, tx index, -1 }
	//  - Post-application events have a TOID with { ledger seq, -1, 0 }

[chowbao]

This event ID should not be exposed to downstream users unless this discussion provides strong evidence that the event IDs should be exposed to users.

Does anyone have any concerns about NOT exposing event ID to downstream users?

[chowbao]

Possible changing event index in the future

The proposed fix for this is to add a type character to the event ID to signify whether the event comes from contract events or diagnostic events.

Does anyone have any concerns with these two problems?
Are there any alternative methods to create event IDs that could resolve this potential weirdness?

[Shaptic]

When you iterate through the diagnostic events, you also have system events. Are these lumped into d? Also, there's no notion of an operation in that case, will just be standardized to be zeroes?

Also, for contract events, they aren't always coming from a contract. There's transaction-level events, and classic events aren't associated with a contract. Should we use a different letter than c xor omit it entirely?

[dmkozh]

classic events aren't associated with a contract.

Classic events are associated with the respective SAC.

However, classic events are not hashed into ledger, which means that they can be changed retroactively. I don't love that, but we already had two occasions for that (initial backfill when classic events were introduced, and then retroactive token mints/burns due to the state archival bug). I'd hope this never happens in the future, but we can't be 100% sure it won't. That said, at least for the case of the mints/burns the events were not 'interleaved' with the old events, so order-based index option would work for it. I guess we can try to make sure that if we ever need to backfill events, we only add them after the existing events in the respective stream. This still may bite us in the future though.

Are there any alternative methods to create event IDs that could resolve this potential weirdness?

FWIW for the 'real' contract events the event index will never change. So I wonder if we should separate the problem based on the event kind.

[chowbao]

I guess we can try to make sure that if we ever need to backfill event

I want to propose this as a best effort. We can't predict what will happen in the future (like the really old proposal to add all state emitted as events. If that were to happen it's possible the whole event ID notation would need to be updated)

Should we use a different letter than c xor omit it entirely?

I'm fine with a different letter or omitting it for contract events (as in still have -d- for diagnostic events). Do you (or anyone) have a preference or letter or completely omitting?

[chowbao]

you also have system events

I forgot about these 🤦
Does that mean we want a separate type letter notation for system events?

[chowbao]

event index

What's a reasonable # bit/byte to allocate for event index given the potentially larger number of diagnostic events compared to just contract events?

[tomerweller]

The main requirement is that the event IDs are unique and ordered

nit: I think you're missing deterministic.

[chowbao]

Thanks updated

[dmkozh]

When diagnostic events are enabled there is a whole other stream of events. Diagnostic events also contain contract events. So contract events will in theory be located in two spots within the ledger metadata.

These can (and should) be ignored. Only the actual contract events are hashed into ledger, everything else is subject to change. Requirement for deterministic ids for the diagnostic events seems problematic to me, as it requires diagnostic events to be stable, which has never been a requirement.

[chowbao]

Requirement for deterministic ids for the diagnostic events seems problematic to me, as it requires diagnostic events to be stable, which has never been a requirement

Yeah diagnostic events is a problem. I don't have a great solution to this cause we do need to have some ID for diagnostic events cause (like the core_metrics).

Maybe other people can chime in but what if we adjust the requirements to ONLY contract events being deterministic? and diagnostic events still follow the same pattern but only require being ordered and unique?

From data team and Dune's perspective I think non deterministic diagnostic events would be fine

[dmkozh]

I don't have a great solution to this cause we do need to have some ID for diagnostic events cause (like the core_metrics).

What is the use case for indexing these? In general, diagnostic events are supposed to be human readable (not machine-readable), and indexing singular diagnostic events seems wrong to me conceptually. For example, the 'duplicate' contract events mentioned in this discussion - the contract events inside diagnostic are included just for the debugging purposes (as in 'contract call A emitted event B', notably the event could be rolled back later), and they don't make any sense in a vacuum without the full call stack.

I guess I'm fine with non-deterministic diagnostic event ids if everyone else is fine, but I just wanted to double-check that we're not doing anything weird with diagnostic events that might be a potential footgun (like developers being able to accidentally use a diagnostic event as a contract event, or ingest an event from a failed call as a valid event etc.).

[chowbao]

What is the use case for indexing these?

For data quality checking in terms of one system sending data to another system. For example, if Hubble/BigQuery were to be used as the source of the events (note that our BQ tables also dump out diagnostic events) a downstream user would want an event ID associated with each event to make sure they didn't accidentally ingest the same event twice. Right now without an event ID you just have to assume you didn't accidentally duplicate data even if there is a completely separate event that just looks like a duplicate like with the example {type:transfer, from:userA, to:userB, asset:XLM, amount:10} possibly appearing twice for a single operation.

Hmm so if we don't want to standardize a non-deterministic diagnostic event ID we don't have to include it in the SEP. I would just feel bad cause that means any indexer or user that does want to do these kind of checks would have to create their own event ID without any direction

[dmkozh]

To be clear, what I've been thinking about is potentially treating the diagnostic event vector as a single blob without trying to interpret it. Basically, I'm trying to understand if it's even necessary to treat diagnostic events as separate entities.