Proposal for getEvents v2 endpoint

[tamirms]

Overview

This proposal specifies an improved getEvents Stellar RPC API for querying events on the Stellar blockchain.

Design Goals

The API addresses specific problems reported in GitHub issues stellar/stellar-rpc#426 and stellar/stellar-rpc#575:

Problem (#426, #575)	Solution
Filters match on topic count, not content	Positional semantics—position in array matches topic position
Cursor encoding exposes implementation details	Opaque cursor string hides internal structure
endLedger ignored during pagination	Cursor encodes original bounds, honored throughout
No recovery info when cursor expires	Errors include oldestLedger for client recovery
Empty pages confuse clients with sparse data	hasMore: true + scannedLedger shows scan progress
No way to filter by transaction	txHash query mode
No descending order for "latest events"	order: "desc" iterates newest-first

DoS Prevention

The API includes constraints which can be configured by an RPC provider to prevent any single query from overwhelming the backend

Constraint	Limit	Rationale
contractIds	Max 3	Bounds index lookups
Topic positions	Max 4	Matches Stellar event structure
Values per position	Max 3	Bounds index lookups
limit	Max 1,000	Bounds response payload size
Internal scan limit	10k ledgers	Prevents unbounded scans; returns partial progress via scannedLedger

How it works: When a query matches sparse data, the backend scans up to its configured limit, returns whatever matches were found (possibly zero), sets hasMore: true, and provides scannedLedger so clients can track progress. This ensures predictable response times regardless of data distribution. The scan limit is configurable per RPC implementation.

Client impact: Queries exceeding limits return invalid_params errors. Large filter sets must be split into parallel queries and merged client-side.

Request

The API accepts three mutually exclusive request modes:

Mode	Discriminator	Purpose
RangeQuery	Default (no cursor or txHash)	Query events within a ledger/event range
TransactionQuery	txHash present	Query events within a single transaction (returned in execution order)
CursorQuery	cursor present	Continue pagination from a prior query

Type Definitions

// SCVal can be specified as base64 XDR or JSON representation
type ScVal = 
  | string    // base64-encoded XDR
  | object    // JSON SCVal ({ symbol: "..." }, { address: "..." }, etc.)

// A segment matches either a specific value or any value
type TopicSegment = 
  | ScVal           // Exact match
  | ScVal[]         // OR: match any of these values (max 3, no nulls)
  | null            // Wildcard: match any single value

// Positional topic filter: array of up to 4 segments
// Position 0 checks topic[0], Position 1 checks topic[1], etc.
// Positions beyond the filter length are not checked
// Example: a 2-element filter only checks topic[0] and topic[1]
type TopicFilter = TopicSegment[]

type RangeQuery = {
  min?: number,           // Starting ledger (inclusive)
  max?: number,           // Ending ledger (inclusive)
  order?: "asc" | "desc",
  contractIds?: string[],
  topics?: TopicFilter,
  limit?: number
}

type TransactionQuery = {
  txHash: string,
  contractIds?: string[],
  topics?: TopicFilter,
  limit?: number
}

type CursorQuery = {
  cursor: string,
  limit?: number
}

type GetEventsRequest = RangeQuery | TransactionQuery | CursorQuery

Field Definitions

Field	Type	Description
min	number	Starting ledger (inclusive)
max	number	Ending ledger (inclusive)
order	"asc" | "desc"	Iteration direction. Default: "asc" (oldest first)
contractIds	string[]	Filter to these contracts (OR within list). Max 3
topics	TopicFilter	Positional topic filter. See Topic Filter
limit	number	Max events per response. Default: 100, max: 1,000
txHash	string	Return events from this transaction only
cursor	string	Opaque token to continue from previous response

When both contractIds and topics are specified, events must match both filters (AND).

Empty arrays (contractIds: [] or topics: []) are invalid. Omit the field entirely to match all contracts or all topics.

Ledger bounds are inclusive: min: 1000, max: 5000 includes events from both ledgers 1000 and 5000.

Topic Filter

The TopicFilter uses positional semantics, matching the Ethereum eth_getLogs convention:

• Position 0 filters topic[0], Position 1 filters topic[1], etc.
• null matches any value at that position
• Positions beyond the filter length are not checked
• Arrays express OR within a position

SCVal formats: Topic values can be specified as base64-encoded XDR or JSON:

// Base64 XDR (compact)
"AAAADwAAAAh0cmFuc2Zlcg=="

// JSON representation (readable)
{ symbol: "transfer" }

Matching rules:

• Filters are AND across positions
• Arrays are OR within a position (must be exact values, no null)
• Events must have at least as many topics as the filter length

Examples:

// All transfer events (JSON format)
[{ symbol: "transfer" }]

// Transfers from specific address
[
  { symbol: "transfer" },
  { address: "GABC...XYZ" }
]

// Any topic[0], specific topic[1], any topic[2]
[null, { address: "GABC...XYZ" }]

// Multiple event types (OR)
[[
  { symbol: "transfer" },
  { symbol: "approve" }
]]

// Mix formats (base64 still works)
["AAAADwAAAAh0cmFuc2Zlcg==", null, { address: "GABC..." }]

Typical SEP-41 transfer event:

topic[0]: symbol "transfer" (event type)
topic[1]: from address
topic[2]: to address  
topic[3]: amount or asset info

Query patterns:

Use Case	Filter
All transfers	[{ symbol: "transfer" }]
Transfers I sent	[{ symbol: "transfer" }, { address: "G..." }]
Transfers I received	[{ symbol: "transfer" }, null, { address: "G..." }]
Transfers involving me	Two queries: sent + received, merged client-side

RangeQuery Rules

Order	Required	Optional (defaults)
asc (default)	min	max (latest ledger)
desc	—	max (latest ledger), min (oldest ledger)

Rationale:

• Ascending requires min: Ensures portable queries across RPCs with different retention policies (archive vs pruned)
• Descending requires nothing: "Latest N events" gives consistent results regardless of RPC retention

Examples:

// Ascending, bounded: ledgers 1000-5000
{ min: 1000, max: 5000 }

// Ascending, unbounded: ledger 1000 to chain tip (for live tracking)
{ min: 1000 }

// Descending, latest 50 events
{ order: "desc", limit: 50 }

// Descending, bounded range newest-first
{ min: 1000, max: 5000, order: "desc" }

// Error: ascending requires min
{ order: "asc" }  // ❌ invalid_params: "min is required for ascending order"

hasMore semantics by direction:

Query type	hasMore: false means
Ascending, unbounded	Caught up to chain tip — poll later for new events
Ascending, bounded	Reached max — query complete
Descending, unbounded	Reached genesis — done forever
Descending, bounded	Reached min — query complete

Cursor

The cursor is an opaque string encoding the complete query state: position, bounds, order, filters, and mode. This enables simple continuation:

// Initial query
let response = await getEvents({ min: 1000, topics: ["transfer", "0xMe"] });

// Continue with just the cursor
response = await getEvents({ cursor: response.cursor });

Only limit may be changed on continuation. All other parameters are encoded in the cursor.

Response

type GetEventsResponse = {
  events: Event[],
  cursor: string,
  hasMore: boolean,
  scannedLedger: number,
  availableLedgers: {
    oldest: number,
    latest: number
  }
}

Field	Description
events	Matching events for this page
cursor	Token to continue pagination
hasMore	true if more events available now, false if query complete or caught up
scannedLedger	Scan frontier in iteration direction. For ascending, the highest ledger scanned. For descending, the lowest ledger scanned
availableLedgers.oldest	Earliest ledger in retention (system boundary)
availableLedgers.latest	Most recent ledger (chain tip)

Pagination States

hasMore	Query Type	Meaning
true	Any	More events available, continue with cursor
false	Bounded	Query complete
false	Unbounded ascending	Caught up to chain tip — poll later for new events
false	Unbounded descending	Reached genesis — done forever

Sparse Data Handling

When querying for rare events, the API may return empty pages due to internal scan limits:

{
  events: [],
  hasMore: true,
  cursor: "abc",
  scannedLedger: 400000,
  availableLedgers: { oldest: 1, latest: 2000000 }
}

This indicates: no matches yet, but 20% through the scan (400k of 2M ledgers). Continue with cursor until hasMore: false.

Progress calculation:

• Ascending: (scannedLedger - min) / (max - min)
• Descending: (max - scannedLedger) / (max - min)

Errors

type ErrorResponse = {
  error: string,
  message: string,
  // Additional context fields vary by error type
}

Error	Context	When
invalid_params	varies	Validation failures (see below)
ledger_pruned	ledger, oldestLedger	Ledger before retention boundary
ledger_future	ledger, latestLedger	Ledger beyond latest available
cursor_malformed	—	Cursor cannot be decoded
cursor_expired	oldestLedger	Cursor references pruned data
cursor_future	cursorLedger, latestLedger	Cursor ahead of latest ledger (data loss)
transaction_not_found	txHash	Transaction does not exist

Validation Errors (invalid_params)

Condition	Message
cursor with other query params	"cursor is mutually exclusive with min, max, order, contractIds, topics, txHash"
txHash with range params	"txHash is mutually exclusive with min, max, order, cursor"
min > max	"min must be <= max"
order: "asc" without min	"min is required for ascending order"
limit < 1	"limit must be at least 1"
limit > 1000	"limit exceeds maximum of 1000"
contractIds.length === 0	"contractIds cannot be empty"
contractIds.length > 3	"contractIds exceeds maximum of 3"
topics.length === 0	"topics cannot be empty"
topics.length > 4	"topics exceeds maximum of 4 positions"
Topic position array empty	"topic position array cannot be empty"
Topic position array length > 3	"topic position array exceeds maximum of 3 values"
null inside topic position array	"wildcard not allowed inside topic value array"

Supported Use Cases

Use Case	Example	Applications
Historical range	{ min: 1000, max: 5000 }	Block explorers, analytics, backfills
Latest events	{ order: "desc", limit: 50 }	Dashboards, activity feeds
Live tracking	{ min: 1000 } + poll with cursor	Indexers, notifications
Resume from checkpoint	{ cursor: savedCursor }	Indexers after restart
Transaction inspection	{ txHash: "abc" }	Transaction detail pages
Transfers I sent	{ min: 1000, topics: ["transfer", "0xMe"] }	Wallet outgoing history
Transfers I received	{ min: 1000, topics: ["transfer", null, "0xMe"] }	Wallet incoming history
Multiple event types	{ min: 1000, topics: [["transfer", "approve"]] }	dApps tracking related events
Multi-contract	{ min: 1000, contractIds: ["pool1", "pool2"] }	DEX aggregators

Use Case Gaps

The following patterns require client-side handling:

Use Case	Workaround
Cross-position OR (e.g., "transfers involving me" = sent OR received)	Two queries merged client-side
Large filter sets (>3 contracts or >3 values per position)	Split into parallel queries, merge results

Design Rationale

Why min/max instead of start/end?
start/end implies iteration direction. With order: "desc", "start at 1000" is confusing when iteration begins at 5000. min/max defines bounds independent of direction.

Why opaque event IDs?
Decouples clients from internal position representation, enabling backend flexibility.

Why cursor encodes filters?
Continuation requires only { cursor }. No parameter spreading, no mismatch risk.

Why hasMore instead of nullable cursor?
Explicit signal for "more available now". Combined with always-present cursor, enables simple live tracking loops.

Why does ascending require min but descending doesn't?
Descending from latest gives consistent results regardless of RPC retention policy — you always get recent events. Ascending without min would start from the oldest available ledger, which varies by RPC (genesis for archive nodes, 7 days ago for pruned nodes). Requiring min ensures portable queries across different RPC configurations.

Appendix: Code Examples

Latest N events:

const transfer = { symbol: "transfer" };

const response = await getEvents({
  order: "desc",
  limit: 50,
  topics: [transfer]
});
display(response.events);  // Already in newest-first order

Historical range with pagination:

const transfer = { symbol: "transfer" };
const myAddr = { address: "GABC...XYZ" };

let response = await getEvents({
  min: 1000,
  max: 5000,
  topics: [transfer, myAddr]  // Transfers I sent
});

while (response.hasMore) {
  process(response.events);
  response = await getEvents({ cursor: response.cursor });
}
process(response.events);

Live tracking:

const transfer = { symbol: "transfer" };
const myAddr = { address: "GABC...XYZ" };

let response = await getEvents({
  min: 1000,
  topics: [transfer, null, myAddr]  // Transfers I received
});

while (true) {
  process(response.events);
  if (!response.hasMore) await sleep(5000);
  response = await getEvents({ cursor: response.cursor });
}

Resume from database:

const saved = await db.load();

let response = await getEvents({ cursor: saved.cursor });

while (true) {
  for (const event of response.events) {
    await process(event);
  }
  await db.save({ cursor: response.cursor });
  if (!response.hasMore) await sleep(5000);
  response = await getEvents({ cursor: response.cursor });
}

Tracking multiple event types:

const transfer = { symbol: "transfer" };
const approve = { symbol: "approve" };
const mint = { symbol: "mint" };

let response = await getEvents({
  min: 1000,
  topics: [[transfer, approve, mint]]  // Any of these event types
});

Wallet activity (sent + received):

const transfer = { symbol: "transfer" };
const myAddr = { address: "GABC...XYZ" };

// Requires two queries — cross-position OR not supported
const [sent, received] = await Promise.all([
  getEvents({ min: 1000, topics: [transfer, myAddr] }),
  getEvents({ min: 1000, topics: [transfer, null, myAddr] })
]);

// Merge and dedupe by event ID
const all = mergeByEventId(sent.events, received.events);

[tomerweller]

How do you express a topic filter that is "all TRANSFER events to 0xme"?

[tamirms]

Do we have a way to get usage metrics of the topic filtering functionality?

Unfortunately we do not run a public rpc node like we do for horizon. The only rpc node we run is used by the wallet backend which doesn't make use of topic filtering. However, we cannot conclusively say the wallet backend's use of rpc is representative of the entire ecosystem.

Maybe we can reason about this by analyzing use cases.

@tomerweller mentioned the transfer history use case. If you are interested in only Transfers where my address is the recipient then the current proposal requires us to filter away events where my address is the recipient. On the other hand, if you are showing a feed for a specific address which shows transfers sent and transfers received the current proposal is better:

// 1. Fetch Sent
const sent = await getEvents({ topic1: "Me" });
// 2. Fetch Received
const received = await getEvents({ topic2: "Me" });

// 3. EXTRA CODE: Merge and Sort
const allEvents = [...sent, ...received].sort((a, b) => b.block - a.block);

// 4. Render
allEvents.forEach(e => {
  if (e.topics[1] == "Me") render("Sent", e);
  else render("Received", e);
});

vs

// 1. Fetch Everything (One Request)
const allEvents = await getEvents({ includes: "Me" });

// 2. EXTRA CODE: Categorize
allEvents.forEach(e => {
  // This isn't "discarding false positives" — it's distinguishing UI states
  const isSender = (e.topics[1] === "Me");
  
  if (isSender) {
    render("Sent", e); // Red Arrow
  } else {
    render("Received", e); // Green Arrow
  }
});

If positional topic filtering is seldom used, should we be considering deprecating it all together rather than replacing it with a fuzzy search that is a new feature?

I would be ok with removing topic filtering altogether. That would make things a lot easier to implement on the backend. However, I think it is a riskier change than the current proposal. If you are already doing topic filtering the current proposal should work for your use case without significant changes to your code.

I can give some more context on the performance implications of supporting topic position filters:

The trade off for supporting positional topic filters vs the current proposal is that we would need an index for each position we want to support. So if you allow a topic filter to include up to 4 positions that would mean 4x the amount of indexes. The queries would also require loading more indexes as well depending on how many topic positions you are filtering on. Keep in mind that we want to support getEvents on full history and that is what I had in mind when proposing this new getEvents API.

If you think filtering topic positions are necessary then I think we should remove the possibility of nesting in the topic filtering and we make the topic filter a flat array which does only intersection. By getting rid of unions we reduce the number of indexes we would need to merge during query time for the worst case possible query.

[mootz12]

I do share concerns that people will use the existing topicFilter as if position matters, especially since the input is taken as an array. If we opt for not enforcing position, this needs to be repeated x10 in documentation.

This being said, I think what is being proposed is functional for just about any use case I can think of, we just need to be as clear as possible how it works so the proper validation is included on the client side.

--

Curious if we explored the eth_getLogs API for topic filters?

e.g. assume event is topics = ["transfer", sender, receiver]

topics = ["transfer", *, ["alice", "bob"]]

would match all transfer events where alice or bob are the receiver and anyone was the sender. Not sure if this simplifies the current query at all given it contains "or" logic (given it can't be nested further than once), but it would reduce the topic filters to only 1.

[tamirms]

@mootz12 the ethereum API is very similar to the current stellar getEvents API and it also features positional topic filters. But, it is not possible to do a request which finds all transfers where 0xme is the sender or recipient. You would have to do two requests and combine the results to get a view of all incoming and outgoing transfers for a particular address:

// 1. Fetch Sent
const sent = await getEvents({ topic1: "Me" });
// 2. Fetch Received
const received = await getEvents({ topic2: "Me" });

// 3. EXTRA CODE: Merge and Sort
const allEvents = [...sent, ...received].sort((a, b) => b.block - a.block);

// 4. Render
allEvents.forEach(e => {
  if (e.topics[1] == "Me") render("Sent", e);
  else render("Received", e);
});

[tomerweller]

Thanks for the detailed response @tamirms. I think your reasoning makes sense in terms meeting general indexing requirements and why this is good for performance.

The thing that gives me pause is the non-obviousness of it. I recognize that this is a subjective quality but the fact that @mootz12 insists that we need to document this behavior 10x times is a red flag - an API should make sense.

If you think filtering topic positions are necessary then I think we should remove the possibility of nesting in the topic filtering and we make the topic filter a flat array which does only intersection.

I agree that the nesting in the current getEvents is excessive (and also confusing) but I think it's worth supporting a small numbers of topic filters and union between them. I suggested something like this in my comment here

But if you think this is prohibitive from a performance perspective then I think a single topic filter per request (similar to your eth example above) is still easier to reason about than the current proposal.

[tamirms]

Updated the proposal to use positional topic filters

[kalepail]

I had AI do a deep dive into all of the GitHub issues and even our own Discord to find all of the complaints and issues that we've heard over the last few years and matched it against this proposal. The results are here in this Gist:

https://gist.github.com/kalepail/f0b14c806f01b69817d128504d0ec523

The TLDR is that it seems like what's being proposed is a good solution. And personally, as I reviewed it, I'm pretty excited about it. Great work.

[urvisavla]

Quick clarification: for order=desc, is the ordering applied to the full event position tuple (ledger, tx_index, op_index, event_index) (i.e. fully reversed) or the intent is to only reverse ledger order while keeping tx/event order ascending within each ledger? Asking because this may impact cursor and index implementation.

[tamirms]

I think the most intuitive behavior would be for order=desc to mean reverse order applied on the full event position tuple. But, I'm open to discussing this if you think it would be detrimental to backend performance

[leighmcculloch]

Form	Example	Meaning
Number	1000	Ledger 1000, inclusive
String	"evt_abc"	Event ID, inclusive
Object	{ ledger: 1000, exclusive: true }	Ledger 1000, exclusive
Object	{ eventId: "evt_abc", exclusive: true }	Event ID, exclusive

Field	Type	Description
min	Boundary	Lower bound of range (earlier in time)
max	Boundary	Upper bound of range (later in time)
...	...	...
cursor	string	Opaque token to continue from previous response

Are there two ways to do cursor-like pagination? I see the boundary type supports an event ID, and separately there's a cursor. What are the planned use cases for the former?

[leighmcculloch]

@Shaptic then is your thinking that in the use case @tamirms showed above the integrator would store the opaque cursor in the db so that they'd resume from that point using the opaque cursor?

[tamirms]

@leighmcculloch I'm confused could you elaborate on why the problem of resuming from db is created by the existence of an opaque cursor? If we got rid of the opaque cursor what exactly would be different and better?

[leighmcculloch]

@tamirms I think either one of a transparent event id cursor, or a opaque cursor, are good for resuming. I'm just raising the concern that there are two cursor-like pagination mechanisms in the api.

I wasn't clear in my response in #1872 (reply in thread) but I 👍🏻 @Shaptic's comment to keep supporting a cursor while leaving out the event id range lookup. That seems great.

The resume example shared at #1872 (reply in thread) should work fine if the id used to resume is just the cursor instead of the event id. Is that accurate?

[tamirms]

yeah, that works. I think it make sense to proceed with @Shaptic's suggestion

[tamirms]

Updated the proposal to get rid of event id bounds

[leighmcculloch]

Form	Example	Meaning
Number	1000	Ledger 1000, inclusive
String	"evt_abc"	Event ID, inclusive
Object	{ ledger: 1000, exclusive: true }	Ledger 1000, exclusive
Object	{ eventId: "evt_abc", exclusive: true }	Event ID, exclusive

It looks like the proposal introduces a new lever to control whether queries should be inclusive or exclusive of inputs. I don't think I've ever seen this in an API before, it seems novel. What problem is it solving for? It's not clear to me which problem at the start of the proposal it aligns with.

When the min/max is a ledger I think inclusive makes sense as a default. And if someone wants exclusive it is very easy to plus or minus one.

The event ID, I'm not clear on what the use case is for that and started a separate thread about that: #1872 (comment). If the goal is pagination, those inputs should probably be always exclusive. Unless the goal is to get a single event, in which case it might be clearer to offer a getEvent API to get a single event.

[tamirms]

I'm ok with removing the exclusive field and making the behavior inclusive for ledgers and exclusive for event ids. the rationale for the exclusive field was to make the behavior explicit but I think simplifying the behavior based on ledger vs event ids is a fine approach

[mootz12]

+1 for removing the exclusive field. The following seems adequate:

ledger (inclusive)
eventId (exclusive)

It would also be easier to add this than remove it in the future if we are wrong.

[leighmcculloch]

I think it will be easy for someone to assume that the inclusive / exclusive property applies to the field min, rather than the value ledger or eventId being passed in, and therefore to introduce off-by-one errors into their program because of it. So it might be better to more fully separate the query input variables for the two.

Note that this issue disappears if we do what @Shaptic suggested and remove the eventId as a min/max input. (#1872 (reply in thread))

[tamirms]

Updated the proposal to get rid of event id bounds. Ledger bounds are always inclusive

[leighmcculloch]

I realise just after posting this that doing this ☝🏻 would require removing the unbounded max on the ledger range, which makes the API harder to use and is a pain point today worth solving for I think. And the small complexity of scannedLedger seems like a good tradeoff to be able to offer unbounded, especially given that a developer would not need to do anything with the field and just keep iterating if hasMore is true.

[teddav]

The events system needs to be more robust. I agree with several points raised so far, but the most critical issue is the 7-day retention window. It needs to be removed.

Being able to reliably fetch historical events is mandatory for any real application. A 7-day limit is a major blocker for app developers. Apps are not temporary experiments. They are meant to live for a long time (forever?), and require durable event access for indexing, recovery, analytics, and user state.

In our case, we are building a privacy pool. By design, we do not want to rely on a backend, because users should not have to trust any backend service. The RPC must be the source of truth. Short-lived events or reliance on third-party infrastructure directly break this model.

I understand and support the goal of decentralization and not forcing the Stellar Foundation to operate heavy indexers indefinitely. However, the ecosystem is (unfortunately) still small !
At this stage, builders need the easiest and most reliable option to get started, which should be the Foundation-provided infrastructure. Once an app grows, switching to another provider is reasonable.

I asked on Discord and was pointed to the Providers page.
After reading it, it’s still unclear:

• which provider is reliable?
• do providers retain events for more than 7 days
• pricing?
• will I need an API key (and therefore a backend)?

Needing to deploy a backend just to fetch historical events significantly slows development and increases complexity. For early-stage builders, this friction is often enough to push them to another chain.

[tupui]

I totally agree on the need for decentralization. We know that a decentralized network is not much if all the necessary infrastructure that must run around is centralized. That list has at least 2 projects from the ecosystem: Lightsail and Lobstr. Though things are very hard for them.

[tamirms]

@teddav @tupui thanks for the feedback, we are actually designing this new API with full history retention in mind! It would be really helpful for us if you could weigh in on the importance of topic filtering. Is the this proposed API sufficient for your needs or do you need more sophisticated topic filtering (if so, please describe how you use topic filtering) ?

[tupui]

Wooo this is amazing to read! I can ping some folks who would have more sensible input than me on that.

[tomerweller]

type TopicClause = { all: string[] }

IIRC topics are ScVal. What are the strings in the topic filter clause? Are these b64 encoded ScVals (similar to the current api)?

[tamirms]

yeah, the strings should be b64 encoded ScVals. I don't use b64 values in the examples because it's hard to decipher. the examples are more like pseudocode

[tomerweller]

Can we allow using standard ScVal json conversions using something like stellar-xdr-json? This can make the API friendlier and less opaque. This also matches the ability to present json in the response. cc @janewang @leighmcculloch

in the past I suggested something like this here

"filters": [
  {
    "type": "contract",
    "contractId": "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
    "topic0": {"type": "symbol", "val": "transfer"},
    "topic1": {"type": "address", "val": "GAIH3ULLFQ4DGSECF2AR555KZ4KNDGEKN4AFI4SU2M7B43MGK3QJZNSR"}
  },
  {
    "type": "contract",
    "contractId": "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
    "topic0": {"type": "symbol", "val": "transfer"},
    "topic2": {"type": "address", "val": "GAIH3ULLFQ4DGSECF2AR555KZ4KNDGEKN4AFI4SU2M7B43MGK3QJZNSR"}
  }
]

[leighmcculloch]

The values in the topics should look like this for xdr-json:

{"symbol":"transfer"}
{"u32":4008401800}
{"u64":"3964442654822814847"}
{"address":"GAIH3ULLFQ4DGSECF2AR555KZ4KNDGEKN4AFI4SU2M7B43MGK3QJZNSR"}

[leighmcculloch]

The reduced nesting simplifies things. And I've had agents trip up on the deep nesting.

I wonder if it might be easy to make a mistake and send topic1 instead of topic2, because on a quick glance I missed that the second one was querying on topic2 and not topic1.

Reducing the nesting, but keeping the topics array might be clearer (see below).

I don't feel strongly about that though, I think topicN works too.

"filters": [
  {
    "type": "contract",
    "contractId": "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
    "topics": [
      {"symbol":"transfer"},
      {"address":"GAIH3ULLFQ4DGSECF2AR555KZ4KNDGEKN4AFI4SU2M7B43MGK3QJZNSR"}
    ]
  },
  {
    "type": "contract",
    "contractId": "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
    "topics": [
      {"symbol":"transfer"},
      "*",
      {"address":"GAIH3ULLFQ4DGSECF2AR555KZ4KNDGEKN4AFI4SU2M7B43MGK3QJZNSR"}
    ]
  }
]

[tamirms]

updated the proposal to accept either JSON or b64 encoded scvals

[Shaptic]

I think it would be valuable to elaborate on the feature subset we plan to support for full history: it seems this v2 variant has both way more features yet also less than the original. If we could compare and contrast it with what's feasible for full history, it may clue us in better as to what can be pared down from this endpoint.

[steven-tomlinson]

This is a timely proposal from my perspective and I look forward to implementing it for use in my sparse history Pakana Node.

[FrankSzendzielarz]

Couple of random thoughts related to above:

• As part of an SDK I developed a year ago, I have a tool that converts the Stellar XDR schemas to protobuf and gRPC and marshalls to JsonRPC. It is used to drive a wrapper that can be hosted as an in-memory server on the client for gRPC interaction with Stellar RPC, or it can be run next to the server (clients of any language can codegen all the SDK objects). Or if you want it you can bake it in or use the tooling to offer a gRPC interface as well. (PS: it also uses AutoFixture to spit out a load of XDR strings for each object to test roundtrip serialisation, so if you want you can have that too, hundreds of test vectors)

• Consider pubsub/push models. If multiple clients are hitting the server watching for something (like a mobile app or video game), you want a push model with topics and subscribers. MQ. Furthermore, you could look into streaming both directions (forward for new ledgers, backwards for scans) by supporting query language integrated observables

• Corollary to above - GraphQL (DoS prevention in complexity config)

[tupui]

Please no GraphQL 😅 I think that would bring unnecessary complexity. Proper RPC API is good enough for that IMO.

[FrankSzendzielarz]

Yeah I dont really know anything about GraphQL lol. Just thought I would mention it because it was not mentioned.

[tomerweller]

Starting a dedicated thread for nesting, following up on @leighmcculloch's comment here.

Arrays of arrays tend to be confusing and there's a lot of potential nesting in the current structure. The following proposal replaces the current TransactionQuery with a FiltersQuery that combines all filters into a strictly flat filters list. Should allow for the same functionality but with less footguns and more predictable performance (though slightly more verbose sometimes).

Definitions

type Filter = { // AND between txhash, contract id and all topic values
  txHash?: string;
  contractId?: string;
  type?: "contract" | "system";
  topics?: ScVal[]; //no further nesting 
};

type FiltersQuery = {
  filters: Filter[];  // UNION between filters
};

Example, XLM transfers to and from a specific address

{
  filters: [
    {  
      contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
      type: "contract", 
      topics: [{ symbol: "transfer"}, { address: "GABC..." }] 
    },
    { 
      contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
      type: "contract", 
      topics: [{ symbol: "transfer"}, "*", { address: "GABC..." }] 
    }
  ]
}

Example with topicN instead of topics array

I find the topic array and the wildcards somewhat confusing so prefer a topicN approach as follows. Though not religious about it.

{
  filters: [
    { 
      contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
      type: "contract", 
      topic0: { symbol: "transfer"},
      topic1: { address: "GABC..."}
    },
    { 
      contractId: "CAS3J7GYLGXMF6TDJBBYYSE3HQ6BBSMLNUQ34T6TZMYMW2EVH34XOWMA",
      type: "contract", 
      topic0: { symbol: "transfer"},
      topic2: { address: "GABC..."}
    }
  ]
}

[tomerweller]

could you elaborate on why you would prefer the limit to be 5 to demonstrate a single filter sep41-transfer-filter? What is the request you had in mind?

I was thinking about a request that covers all token movement for a specific (asset, account) pair:

1. transfer from
2. transfer to
3. mint to
4. burn from
5. clawback from

[tamirms]

In that case you would have to split it across two requests. it's possible that we can optimize our backend design to go up to 5 filters, but I'd prefer to start with limits that we're confident we can handle and then relax them if we're able to improve performance.

[tomerweller]

Alright. This seems to be the case with the original nested topic filter proposal as well, right? since there's a maximum of 3 values per topic

[tamirms]

yes, I think the only advantage of the nested topic proposal is that you can have a 3 topics in combination with three contract ids:

{
  contractIds: ["XLM", "USDC", "EURC"], 
  topics: [
    ["mint", "burn", "clawback"],
    ["myAddress"],
  ]
}

This would find all mints, burns, clawbacks involving my address for the XLM, USDC, and EURC assets.

I think we could make the nested topic API more intuitive with the following shape:

type ScVal =
  | string   // base64-encoded XDR
  | object   // JSON SCVal ({ symbol: "..." }, { address: "..." }, etc.)

type TopicMatch =
  | ScVal      // exact match
  | ScVal[]    // OR: match any of these (max 3)
  // omit a position entirely to match any value

type TopicFilter = {
  topic0?: TopicMatch,
  topic1?: TopicMatch,
  topic2?: TopicMatch,
  topic3?: TopicMatch,
}

The request above would look like:

{
  topics: {
    topic0: [{symbol: "mint"}, {symbol: "burn"}, {symbol: "clawback"}],
    topic1: {address: "myAddress"},
  }
}

Or, we can take your proposal and remove contractId from the filter:

{
  contractIds: ["XLM", "USDC", "EURC"], 
  filters: [
    { 
      type: "contract", 
      topic0: { symbol: "transfer"},
      topic1: { address: "GABC..."}
    },
    { 
      type: "contract", 
      topic0: { symbol: "transfer"},
      topic2: { address: "GABC..."}
    }
  ]
}

This is a request which finds all transfers involving "GABC..." as sender or recipient across the XLM, USDC, EURC assets.

What do you think @tomerweller of taking your proposal and lifting out contract ids ? The limits would be a max of 3 items in contractIds and a max of 3 items in the filters list.

[leighmcculloch]

type ScVal =
  | string   // base64-encoded XDR
  | object   // JSON SCVal ({ symbol: "..." }, { address: "..." }, etc.)

It seems a little risky to use the string type to identify that it is base64 xdr. A couple JSON SCVal's are just a string: "void" and "ledger_key_contract_instance". SCVal's in JSON are either a string or an object.

void is unfortunately valid base64, although it doesn't decode to any valid xdr type that is defined today (using stellar xdr guess) and wouldn't ever decode to a valid SCVal because an SCVal is at minimum 4 bytes long, and void in base64 decodes to 0xbe889d (3 bytes).

ledger_key_contract_instance is not valid base64.

So both strings couldn't be mistaken for a base64 encoded xdr, but others may be added in the future.

type TopicMatch =
  | ScVal      // exact match
  | ScVal[]    // OR: match any of these (max 3)
  // omit a position entirely to match any value

Supporting arrays or exact matches seems non-obvious for the latter case, what could be a hidden feature. But maybe that approach could be adopted instead in the existing API for nested topic lists because the convenient thing about adding it there is that it is backwards compatible whilst making the easy case work:

  filters.topics?: ScVal[] | [ScVal[]]; // One or many topic lists

Or, we can take your proposal and remove contractId from the filter:

{
  contractIds: ["XLM", "USDC", "EURC"], 
  filters: [
    { 
      type: "contract", 
      topic0: { symbol: "transfer"},
      topic1: { address: "GABC..."}
    },
    { 
      type: "contract", 
      topic0: { symbol: "transfer"},
      topic2: { address: "GABC..."}
    }
  ]
}

Would we achieve a similar effect if we kept the structure today, but limited the number of filters to one?

Examples, using the existing rpc format, modifying to use JSON ScVals and new topic wildcard behavior:

Before (Nested filters)

{
  "jsonrpc": "2.0",
  "id": 8675309,
  "method": "getEvents",
  "params": {
    "startLedger": 199616,
    "filters": [
      {
        "type": "contract",
        "contractIds": [ "XLM", "USDC" ],
        "topics": [
          [ { symbol: "transfer"} ]
        ]
      }
    ],
    "pagination": {
      "limit": 2
    }
  }
}

After (Flattened outter)

{
  "jsonrpc": "2.0",
  "id": 8675309,
  "method": "getEvents",
  "params": {
    "startLedger": 199616,
    "type": "contract",
    "contractIds": [ "XLM", "USDC" ],
    "topics": [
      [ { symbol: "transfer"} ]
    ],
    "pagination": {
      "limit": 2
    }
  }
}