Skip to content

Commit 66e2069

Browse files
armenzgantonpirker
authored andcommitted
Point to in-depth Kafka guide (#13857)
The internal document has a lot more details than what I added.
1 parent ad7f9ea commit 66e2069

File tree

1 file changed

+1
-33
lines changed
  • develop-docs/backend/application-domains

1 file changed

+1
-33
lines changed

develop-docs/backend/application-domains/kafka.mdx

Lines changed: 1 addition & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -3,36 +3,4 @@ title: Kafka consumers
33
sidebar_order: 60
44
---
55

6-
## Create a new Kafka topic
7-
8-
1. Add the topic to the `KAFKA_TOPIC_TO_CLUSTER` in [src/sentry/conf/server.py](https://github.com/getsentry/sentry/blob/master/src/sentry/conf/server.py):
9-
* e.g. `subscription-results-eap-items`
10-
2. Add the topic to `Topic` in [src/sentry/conf/types/kafka_definition.py](https://github.com/getsentry/sentry/blob/master/src/sentry/conf/types/kafka_definition.py)
11-
12-
## Define or re-use a processing strategy
13-
14-
In most cases a [Streaming Factory](https://getsentry.github.io/arroyo/getstarted.html#create-a-streaming-consumer) is what you want to when defining a consumer (see next section). You can find examples of it in [Sentry's code base](https://github.com/search?q=repo%3Agetsentry%2Fsentry+%28ProcessingStrategyFactory&type=code).
15-
16-
## Define a new Kafka consumer
17-
18-
1. Add a new entry in the `KAFKA_CONSUMERS` key in
19-
[src/sentry/consumers/__init__.py](https://github.com/getsentry/sentry/blob/master/src/sentry/consumers/__init__.py):
20-
```python
21-
KAFKA_CONSUMERS = {
22-
"<your_topic_str_here>": {
23-
"topic": Topic.YOUR_TOPIC,
24-
"strategy_factory": "sentry_package_defining_your_strategy_factory_class",
25-
}
26-
}
27-
```
28-
2. You may need optional properties (e.g. `click_options`, you will need to research them by looking at [ConsumerDefinition](https://github.com/getsentry/sentry/blob/master/src/sentry/conf/types/kafka_definition.py)'s code.
29-
30-
3. Make sure you can run it: `sentry run consumer <your_topic>`
31-
4. You may need to add some devserver options [here](https://github.com/getsentry/sentry/blob/master/src/sentry/runner/commands/devserver.py).
32-
4. Add tests for your consumer
33-
34-
## Run the consumer in production
35-
1. Discuss it with SRE during their office hours
36-
2. Create ops PR for adding your consumer to
37-
[k8s/services/getsentry/_consumer-deployment.yaml.j2](https://github.com/getsentry/ops/blob/master/k8s/services/getsentry/_consumer-deployment.yaml.j2).
38-
3. In the [Sentry Consumers metrics dashboard](https://www.notion.so/sentry/Kafka-e8b4f93595684c97b01fe831fbceb0dc?pvs=4#1fa8b10e4b5d8044859ecaf559463cb0), add a new saved view for your consumer.
6+
Visit https://github.com/getsentry/ops/tree/master/shared_config/kafka for a full, in-depth step-by-step guide.

0 commit comments

Comments
 (0)