Skip to content

Commit fc3c7eb

Browse files
Dav1ddeDav1dde
committed
feat(relay): Add document describing span and transaction quotas
1 parent 4dd1938 commit fc3c7eb

File tree

1 file changed

+230
-0
lines changed

1 file changed

+230
-0
lines changed

‎develop-docs/relay/transaction-span-ratelimits.mdx‎

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
---
2+
title: Transaction and Span Rate Limiting
3+
---
4+
5+
<Alert level="warning" title="Important">
6+
This document describes a desired state, it is not yet fully implemented in Relay.
7+
</Alert>
8+
9+
10+
Relay enforces quotas defined in Sentry and propagates them as rate limits. In most cases,
11+
this is a simple one-to-one mapping of data category to envelope item.
12+
13+
The transaction and span rate limits are more complicated. Spans and Transactions both
14+
have a total and an additional indexed category. They are also closely related, a transaction is a container for spans.
15+
A dropped transaction results in dropped spans.
16+
17+
The following document describes how transaction and span quotas interact with each other within Relay.
18+
19+
<Alert level="info" title="Important">
20+
This section describes how Relay has to interpret quotas, SDKs and other consumers
21+
should read [SDK Development / Rate Limiting](/sdk/rate-limiting/).
22+
</Alert>
23+
24+
25+
Related Documentation:
26+
- [The Indexed Outcome Category](/application/dynamic-sampling/outcomes/)
27+
28+
29+
30+
## Enforcement
31+
32+
The following rules hold true:
33+
- A quota for the transaction category is equivalent to a quota for the spans category.
34+
- A quota for the spans category is equivalent to a quota for the transaction category.
35+
- When a transaction is dropped, all the contained spans should be reported in outcomes.
36+
- If either transactions or spans are rate limited, clients should receive a limit for both categories.
37+
- Indexed quotas only affect the payload for their respective category.
38+
39+
These rules can be visualized using this table:
40+
41+
| | Transaction Payload | Transaction Metric | Span Payload | Span Metric |
42+
| ----------------------- | ------------------- | ------------------ | ------------ | ----------- |
43+
| **Transaction** | ❌ | ❌ | ❌ | ❌ |
44+
| **Transaction Indexed** | ❌ | ✅ | ✅ | ✅ |
45+
| **Span** | ❌ | ❌ | ❌ | ❌ |
46+
| **Span Indexed** | ✅ | ✅ | ❌ | ✅ |
47+
48+
49+
## Outcomes
50+
51+
Outcomes must be generated in Relay for every span and transaction which is dropped. Usually a dropped
52+
span/transaction results in outcomes for their respective total and indexed category, refer to
53+
[The Indexed Outcome Category](/application/dynamic-sampling/outcomes/) for details.
54+
55+
This is straight forward for all indexed rate limits, they only drop the payload which results in a single
56+
negative outcome in the indexed category of the item. A `transaction_indexed` rate limit does not
57+
cause any spans to be dropped and vice versa.
58+
59+
A span quota and the resulting span rate limit is also trivial for standalone spans received by Relay,
60+
the standalone span is dropped, and a single outcome is generated.
61+
62+
### Transaction Outcomes
63+
64+
Transactions are containers for spans until they are extracted in Relay. This span extraction can happen at any
65+
Relay stage: customer managed, PoP-Relay or Processing-Relay. Until spans are extracted from Relay, a dropped transaction
66+
should count the contained spans and generate an outcome with the contained span quantity + 1, for the segment span
67+
which would be generated from the transaction itself.
68+
69+
<Note>
70+
While it is desirable to have span counts correctly extracted from dropped transactions, it may not be feasible
71+
to do so at any stage of the processing pipeline. For example, it may not be possible to do so (malformed transactions)
72+
or simply too expensive to compute.
73+
</Note>
74+
75+
After spans have been extracted, the transaction is no longer a container of span items and just represents itself,
76+
thus, a dropped transaction with spans already extracted only generates outcomes for the total transactions and
77+
indexed transaction categories.
78+
79+
<Note>
80+
Span quotas are equivalent to transactions quotas, so all the above also applies for a span quota.
81+
</Note>
82+
83+
84+
## Examples
85+
86+
### Example: Span Indexed Quota
87+
88+
**Quota**:
89+
90+
```json
91+
{
92+
"categories": ["span_indexed"],
93+
"limit": 0
94+
// ...
95+
}
96+
```
97+
98+
**Transaction**:
99+
100+
```json
101+
{
102+
"type": "transaction",
103+
"spans": [
104+
{ .. },
105+
{ .. },
106+
{ .. }
107+
],
108+
// ...
109+
}
110+
```
111+
112+
An envelope containing a transaction with 3 child spans generates 4 outcomes for rate limited spans in the
113+
`spans_indexed` category. 1 count for the generated segment span from the transaction and 3 counts for the
114+
contained spans. The transaction itself will still be ingested.
115+
116+
**Ingestion**:
117+
118+
| Transaction Payload | Transaction Metrics | Span Payload | Span Metrics |
119+
| ------------------- | ------------------- | ------------ | ------------ |
120+
| ✅ | ✅ | ❌ | ✅ |
121+
122+
**Outcomes**:
123+
124+
| `transaction` | `transaction_indexed` | `span` | `span_indexed` |
125+
| ------------- | --------------------- | ------ | -------------- |
126+
| 0 | 0 | 0 | 4 |
127+
128+
**Rate Limits propagated to SDKs:** None.
129+
130+
131+
### Example: Transaction Quota
132+
133+
**Quota**:
134+
135+
```json
136+
{
137+
"categories": ["transaction"],
138+
"limit": 0
139+
// ...
140+
}
141+
```
142+
143+
**Transaction**:
144+
145+
```json
146+
{
147+
"type": "transaction",
148+
"spans": [
149+
{ .. },
150+
{ .. },
151+
{ .. }
152+
],
153+
// ...
154+
}
155+
```
156+
157+
**Ingestion**:
158+
159+
| Transaction Payload | Transaction Metrics | Span Payload | Span Metrics |
160+
| ------------------- | ------------------- | ------------ | ------------ |
161+
| ❌ | ❌ | ❌ | ❌ |
162+
163+
**Outcomes**:
164+
165+
| `transaction` | `transaction_indexed` | `span` | `span_indexed` |
166+
| ------------- | --------------------- | ------ | -------------- |
167+
| 1 | 1 | 4 | 4 |
168+
169+
**Rate Limits propagated to SDKs:** Transaction, Span.
170+
171+
172+
## FAQ / Reasoning
173+
174+
### Why can transaction limits be propagated as span limits and vice versa?
175+
176+
At first glance this is non-obvious, spans can exist without transactions
177+
and also transactions can exist without (standalone) spans, so why can the quotas
178+
be used interchangeably?
179+
180+
The reasoning lies in the Sentry product and how the information is used within Sentry.
181+
Because of Relay, Sentry can safely assume spans exist if the user/SDK sends transactions
182+
and more and more of the product is built on the basis of spans. From this point of view
183+
transactions and spans convey the same information, it is just represented differently.
184+
This paradigm shift from transaction to spans is also visible in the billing. AM2 and before
185+
only used transactions for billing, whereas AM3 uses spans now. But AM3 still accepts transactions
186+
it just bills on spans. Disabling ingestion for spans on AM3 therefore also would require the same
187+
quota to be set for transactions, but since quotas are tracked separately a transaction quota
188+
may never trigger the span quota or vice versa.
189+
190+
The logical conclusion and simplification is, to just treat span and transaction rate limits
191+
equally (important: treat the limits the same, the quotas are still tracked separately).
192+
Another upside is, enforcing these limits on SDKs does not require extra logic, it is all contained
193+
within Relay.
194+
195+
### Why do span outcomes and transaction outcomes differ?
196+
197+
There can be multiple reasons for this in the entire pipeline, these are just some reasons why there
198+
can be differences caused by Relay:
199+
200+
- Parsing the span count from a transaction is too expensive and may be omitted. This is an extremely
201+
important property of Relay, abuse cases must be handled as fast as possible with as little cost ($ and resources)
202+
as possible. In some cases, it may not be feasible to JSON parse the transaction to extract a span count.
203+
- A envelope item may be malformed, there will be outcomes generated for the inferred data category (span or transaction),
204+
but the span count cannot be recovered from an invalid transaction.
205+
206+
207+
### I want Relay to enforce a quota, which category should I use?
208+
209+
From top to bottom:
210+
211+
- Do you completely disable ingestion? Configure the limit for the `span` and `transaction` data categories.
212+
- Is it billing related? For example, spike protection operates on the category of the billing unit
213+
(AM2: transactions, AM3: spans), use the respective data category for the limit.
214+
- Use the total category (not indexed) which makes most sense to you. When in doubt, ask the Relay team.
215+
216+
### Should I ever use an indexed category in a Quota?
217+
218+
No.
219+
220+
Indexed quotas are only useful to protect downstream infrastructure through abuse quotas.
221+
They are inherently more expensive to enforce, cannot be propagated to clients and are generally a sign
222+
of misconfiguration.
223+
224+
Dynamic-, smart- or client side-sampling should prevent any indexed quota from being enforced.
225+
226+
### What does this mean for standalone spans?
227+
228+
They are not treated differently from extracted spans. After metrics extraction, which may happen in customer
229+
Relays, there is no more distinction. Having no special treatment for standalone spans also means we do not
230+
need any special logic in the SDKs.

0 commit comments

Comments
 (0)