Skip to content

Commit c08b898

Browse files
mccullyamccullya
committed
adding multi-DC SR example
1 parent 5e2d763 commit c08b898

29 files changed

+424
-0
lines changed

‎incubator/schema-registry-leader-follower/README.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Basic Deployment
2+
This example deploys a basic deployment. No RBAC/LDAP. Just a single topic 'foobar' is added as part of the pipeline.
3+
4+
5+
| Feature | Enabled | Note |
6+
|:-----------------|:-------:|:-------------------------|
7+
| Kafka/Zookeeper | ✅ | |
8+
| Control Center | ✅ | |
9+
| Connect | ❌ | |
10+
| Schema Registry | ❌ | |
11+
| KSQL | ❌ | |
12+
| TLS Encryption | ✅ | Self-signed certificates |
13+
| Authentication | ✅ | |
14+
| Authorization | ✅ | via LDAP |
15+
16+
17+
### Deploy CRDs
18+
Deploy the CRDS using the standard way:
19+
```shell
20+
kubectl apply -k ../../base/crds
21+
```
22+
### Deploy Confluent Operator and Confluent Services
23+
Deploy the confluent operator and services:
24+
```shell
25+
kubectl apply -k .
26+
```
27+
28+
### Forward ports locally.
29+
#### Source Kafka
30+
Minikube:source:kafka:9071 -> localhost:9071
31+
#### Source SchemaRegistry
32+
Minikube:source:schemaregistry:9081 -> localhost:9081
33+
#### Source Kafka
34+
Minikube:destination:kafka:9071 -> localhost:9072
35+
#### Source Kafka
36+
Minikube:source:schemaregistry:9081 -> localhost:9082
37+
38+
```shell
39+
40+
nohup kubectl port-forward schemaregistry-0 8081:8081 --namespace master \
41+
&& \
42+
nohup kubectl port-forward schemaregistry-0 8082:8081 --namespace dogs \
43+
&& \
44+
nohup kubectl port-forward schemaregistry-0 8083:8081 --namespace cats
45+
46+
```
47+
kubectl port-forward \
48+
$(kubectl get pods -n source -l statefulset.kubernetes.io/pod-name:kafka-0 -o name) \
49+
:9071 -n source \
50+
&& \
51+
kubectl port-forward \
52+
$(kubectl get pods -n source -l app.kubernetes.io/component=grafana -o name) \
53+
:9071 -n source \
54+
&& \
55+
kubectl port-forward \
56+
$(kubectl get pods -n source -l app.kubernetes.io/component=grafana -o name) \
57+
9071 -n source
58+
&& \
59+
kubectl port-forward \
60+
$(kubectl get pods -n source -l app.kubernetes.io/component=grafana -o name) \
61+
9071 -n source
62+
63+
```
64+
65+
### Query schema Registry
66+
67+
[//]: # (TODO Port forward Production Schema Registry - 8081)
68+
[//]: # (TODO Port forward Production Kafka Broker - 9092)
69+
70+
```
71+
curl -k -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
72+
--data '{"schema": "{\"type\": \"string\"}"}' \
73+
https://localhost:8081/subjects/im-a-source-schema/versions
74+
```
75+
76+
77+
78+
```
79+
curl -k -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
80+
--data '{"schema": "{\"type\": \"string\"}"}' \
81+
https://localhost:8082/subjects/im-a-desgination-schema/versions
82+
```10.109.90.121
83+
84+
`curl -k -X GET https://10.109.90.121:8081/subjects`
85+
86+
`curl -k -X GET https://localhost:8082/subjects`
87+
88+
#TODO - how will

‎incubator/schema-registry-leader-follower/environments/cats/cats-rest-class.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
apiVersion: platform.confluent.io/v1beta1
2+
kind: KafkaRestClass
3+
metadata:
4+
name: cats-kafka-rest
5+
spec:
6+
kafkaClusterRef:
7+
name: kafka
8+
kafkaRest:
9+
endpoint: http://kafka.cats.svc.cluster.local:8090

‎incubator/schema-registry-leader-follower/environments/cats/control-center.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
apiVersion: platform.confluent.io/v1beta1
3+
kind: ControlCenter
4+
metadata:
5+
name: controlcenter
6+
spec:
7+
configOverrides:
8+
server:
9+
- confluent.controlcenter.internal.topics.replication=1
10+
- confluent.controlcenter.command.topic.replication=1
11+
- confluent.monitoring.interceptor.topic.replication=1
12+
- confluent.metrics.topic.replication=1

‎incubator/schema-registry-leader-follower/environments/cats/kafka.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
apiVersion: platform.confluent.io/v1beta1
2+
kind: Kafka
3+
metadata:
4+
name: kafka
5+
spec:
6+
replicas: 1
7+
configOverrides:
8+
server:
9+
- confluent.schema.registry.url=http://schemaregistry:8081
10+
- confluent.balancer.topic.replication.factor=1
11+
- confluent.license.topic.replication.factor=1
12+
- confluent.tier.metadata.replication.factor=1
13+
- confluent.metadata.topic.replication.factor=1
14+
- confluent.metrics.reporter.topic.replicas=1
15+
- confluent.balancer.topic.replication.factor=1
16+
- confluent.security.event.logger.exporter.kafka.topic.replicas=1
17+
- default.replication.factor=1
18+
- event.logger.exporter.kafka.topic.replicas=1
19+
- offsets.topic.replication.factor=1
20+

‎incubator/schema-registry-leader-follower/environments/cats/kustomization.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
namespace: cats
4+
resources:
5+
- namespace.yaml
6+
- ../../../../base/cfk-components/confluent-no-auth-no-tls
7+
- sr-linking-topic-demo.yaml
8+
- schema-registry.yaml
9+
- cats-rest-class.yaml
10+
patchesStrategicMerge:
11+
- kafka.yaml
12+
- zookeeper.yaml
13+
- control-center.yaml

‎incubator/schema-registry-leader-follower/environments/cats/namespace.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: cats

‎incubator/schema-registry-leader-follower/environments/cats/schema-registry.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
apiVersion: platform.confluent.io/v1beta1
2+
kind: SchemaRegistry
3+
metadata:
4+
name: schemaregistry
5+
spec:
6+
replicas: 1
7+
image:
8+
application: confluentinc/cp-schema-registry:7.1.0
9+
init: confluentinc/confluent-init-container:2.3.0
10+
configOverrides:
11+
server:
12+
- kafkastore.topic.replication.factor=1
13+
- leader.eligibility=false
14+
- unclean.leader.election.enable=false
15+
- schema.registry.group.id=id_schemaregistry_master
16+
- kafkastore.topic=_schemas
17+
dependencies:
18+
kafka:
19+
bootstrapEndpoint: http://kafka.master.svc.cluster.local:9071

‎incubator/schema-registry-leader-follower/environments/cats/sr-linking-topic-demo.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: platform.confluent.io/v1beta1
2+
kind: KafkaTopic
3+
metadata:
4+
name: sr-linking-topic-demo-cats
5+
spec:
6+
replicas: 1
7+
partitionCount: 4
8+
configs:
9+
cleanup.policy: "delete"
10+
kafkaRestClassRef:
11+
name: cats-kafka-rest

‎incubator/schema-registry-leader-follower/environments/cats/zookeeper.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: platform.confluent.io/v1beta1
2+
kind: Zookeeper
3+
metadata:
4+
name: zookeeper
5+
spec:
6+
replicas: 1

‎incubator/schema-registry-leader-follower/environments/dogs/control-center.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
apiVersion: platform.confluent.io/v1beta1
3+
kind: ControlCenter
4+
metadata:
5+
name: controlcenter
6+
spec:
7+
configOverrides:
8+
server:
9+
- confluent.controlcenter.internal.topics.replication=1
10+
- confluent.controlcenter.command.topic.replication=1
11+
- confluent.monitoring.interceptor.topic.replication=1
12+
- confluent.metrics.topic.replication=1

0 commit comments

Comments
 (0)