Skip to content

Commit cc35c81

Browse files
authored
Recover Kafka notes and make reference to deployment notes (#13880)
Recover changes from #13834 and point to more detailed instructions in the shared Kafka README file.
1 parent 6a6e4d2 commit cc35c81

File tree

1 file changed

+31
-1
lines changed
  • develop-docs/backend/application-domains

1 file changed

+31
-1
lines changed

develop-docs/backend/application-domains/kafka.mdx

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,34 @@ title: Kafka consumers
33
sidebar_order: 60
44
---
55

6-
Visit https://github.com/getsentry/ops/tree/master/shared_config/kafka for a full, in-depth step-by-step guide.
6+
## Create a new Kafka topic
7+
8+
1. Add the topic to the `KAFKA_TOPIC_TO_CLUSTER` in [src/sentry/conf/server.py](https://github.com/getsentry/sentry/blob/master/src/sentry/conf/server.py):
9+
* e.g. `subscription-results-eap-items`
10+
2. Add the topic to `Topic` in [src/sentry/conf/types/kafka_definition.py](https://github.com/getsentry/sentry/blob/master/src/sentry/conf/types/kafka_definition.py)
11+
12+
## Define or re-use a processing strategy
13+
14+
In most cases a [Streaming Factory](https://getsentry.github.io/arroyo/getstarted.html#create-a-streaming-consumer) is what you want to when defining a consumer (see next section). You can find examples of it in [Sentry's code base](https://github.com/search?q=repo%3Agetsentry%2Fsentry+%28ProcessingStrategyFactory&type=code).
15+
16+
## Define a new Kafka consumer
17+
18+
1. Add a new entry in the `KAFKA_CONSUMERS` key in
19+
[src/sentry/consumers/__init__.py](https://github.com/getsentry/sentry/blob/master/src/sentry/consumers/__init__.py):
20+
```python
21+
KAFKA_CONSUMERS = {
22+
"<your_topic_str_here>": {
23+
"topic": Topic.YOUR_TOPIC,
24+
"strategy_factory": "sentry_package_defining_your_strategy_factory_class",
25+
}
26+
}
27+
```
28+
2. You may need optional properties (e.g. `click_options`, you will need to research them by looking at [ConsumerDefinition](https://github.com/getsentry/sentry/blob/master/src/sentry/conf/types/kafka_definition.py)'s code.
29+
30+
3. Make sure you can run it: `sentry run consumer <your_topic>`
31+
4. You may need to add some devserver options [here](https://github.com/getsentry/sentry/blob/master/src/sentry/runner/commands/devserver.py).
32+
4. Add tests for your consumer
33+
34+
## Deployment
35+
36+
Visit the Ops repo and search for `shared_config/kafka/README.md` for a full, in-depth step-by-step guide.

0 commit comments

Comments
 (0)