Skip to content

Commit 82f2a60

Browse files
philrzphilrz
committed
Add Connection Guide doc
1 parent e502c52 commit 82f2a60

File tree

2 files changed

+178
-2
lines changed

2 files changed

+178
-2
lines changed

‎Connection-Guide.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# `zync` Kafka Connection Guide: From Basic Setup to SSL/TLS
2+
3+
## Summary
4+
5+
This guide walks through multiple configurations for connecting `zync` to
6+
Kafka, starting with the default plaintext setup and progressing to SSL/TLS
7+
and mutual TLS (mTLS).
8+
9+
## Environment
10+
11+
The following steps were performed on an AWS EC2 instance running Ubuntu Server 24.04.
12+
13+
## Install Software
14+
15+
```
16+
$ sudo apt update && sudo apt upgrade -y
17+
$ sudo apt install make openjdk-21-jdk -y
18+
$ wget https://github.com/brimdata/zync/releases/download/v0.11.0/zync-v0.11.0.linux-amd64.tar.gz && sudo tar -xzf zync-v0.11.0.linux-amd64.tar.gz -C /usr/local/bin
19+
$ wget https://dlcdn.apache.org/kafka/4.1.1/kafka_2.13-4.1.1.tgz && tar -xzf kafka_2.13-4.1.1.tgz
20+
```
21+
22+
## Start Kafka With Default Plaintext Config
23+
24+
These are based on steps from the
25+
[Apache Kafka Quickstart](https://kafka.apache.org/quickstart).
26+
27+
```
28+
$ cd ~/kafka_2.13-4.1.1
29+
$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
30+
$ bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
31+
$ bin/kafka-server-start.sh config/server.properties
32+
```
33+
34+
## Plaintext `zync` Config
35+
36+
Start with a simple `zync` config that connects to Kafka via its default
37+
"plaintext" protocol.
38+
39+
```
40+
$ mkdir ~/.zync
41+
$ cat > ~/.zync/kafka.json << EOF
42+
{
43+
"bootstrap_servers": "127.0.0.1:9092",
44+
"security_protocol": "PLAINTEXT",
45+
"sasl_mechanisms": "PLAIN"
46+
}
47+
EOF
48+
$ echo {} > ~/.zync/schema_registry.json
49+
```
50+
51+
## Smoke Test
52+
53+
Create a Kafka topic.
54+
55+
```
56+
$ cd ~/kafka_2.13-4.1.1
57+
$ bin/kafka-topics.sh --create --topic MyTopic --partitions 1 --bootstrap-server localhost:9092
58+
```
59+
60+
Pass data between `zync` and Kafka.
61+
62+
```
63+
$ echo '{s:"hello,world"}' | zync produce -topic MyTopic -
64+
producing messages to topic "MyTopic"...
65+
waiting for Kafka flush...
66+
1 messages produced to topic "MyTopic"
67+
68+
$ zync consume -topic MyTopic
69+
{key:null,value:{s:"hello,world"}}
70+
```
71+
72+
## Generate Test Certificates for Kafka SSL
73+
74+
We'll use a self-signed Certificate Authority (CA) for test purposes.
75+
76+
```
77+
$ mkdir -p ~/kafka-ssl-test && cd ~/kafka-ssl-test
78+
$ openssl genrsa -out ca-key.pem 2048
79+
$ openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days 365 -subj "/C=US/ST=CA/L=San Francisco/O=Test/OU=Test/CN=TestCA"
80+
$ openssl genrsa -out broker-key.pem 2048
81+
$ openssl req -new -key broker-key.pem -out broker.csr -subj "/C=US/ST=CA/L=San Francisco/O=Test/OU=Test/CN=localhost"
82+
$ openssl x509 -req -in broker.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out broker-cert.pem -days 365 -extfile <(echo "subjectAltName=DNS:localhost,IP:127.0.0.1")
83+
$ openssl genrsa -out client-key.pem 2048
84+
$ openssl req -new -key client-key.pem -out client.csr -subj "/C=US/ST=CA/L=San Francisco/O=Test/OU=Test/CN=zync-client"
85+
$ openssl x509 -req -in client.csr -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out client-cert.pem -days 365
86+
$ openssl pkcs12 -export -in broker-cert.pem -inkey broker-key.pem -out broker.p12 -name localhost -password pass:brokerpass
87+
$ keytool -importkeystore -deststorepass brokerpass -destkeystore broker.keystore.jks -srckeystore broker.p12 -srcstoretype PKCS12 -srcstorepass brokerpass -alias localhost
88+
$ keytool -import -file ca-cert.pem -alias CARoot -keystore broker.truststore.jks -storepass brokerpass -noprompt
89+
```
90+
91+
## Configure Kafka for SSL/TLS
92+
93+
Stop the plaintext Kafka server (Ctrl+C) and restart it with SSL enabled.
94+
95+
```
96+
$ cd ~/kafka_2.13-4.1.1
97+
$ cp config/server.properties config/server.properties.ssl
98+
$ cat >> config/server.properties.ssl << EOF
99+
# Listeners (CONTROLLER stays on 9093, SSL on 9094)
100+
listeners=SSL://localhost:9094,CONTROLLER://localhost:9093
101+
advertised.listeners=SSL://localhost:9094
102+
controller.listener.names=CONTROLLER
103+
inter.broker.listener.name=SSL
104+
105+
# SSL Configuration
106+
ssl.keystore.location=/home/ubuntu/kafka-ssl-test/broker.keystore.jks
107+
ssl.keystore.password=brokerpass
108+
ssl.key.password=brokerpass
109+
110+
ssl.truststore.location=/home/ubuntu/kafka-ssl-test/broker.truststore.jks
111+
ssl.truststore.password=brokerpass
112+
113+
ssl.client.auth=none
114+
115+
ssl.protocol=TLSv1.2
116+
ssl.enabled.protocols=TLSv1.2,TLSv1.3
117+
EOF
118+
$ bin/kafka-server-start.sh config/server.properties.ssl
119+
```
120+
121+
## SSL `zync` Config/Test
122+
123+
If repeated, the `zync produce` from our [smoke test](#smoke-test) will now
124+
hang because `zync` is trying to connect with plaintext to an SSL-only port,
125+
so we update the `zync` config to use SSL.
126+
127+
```
128+
$ cat > ~/.zync/kafka.json << EOF
129+
{
130+
"bootstrap_servers": "localhost:9094",
131+
"security_protocol": "SSL",
132+
"tls_server_ca_cert_file": "/home/ubuntu/kafka-ssl-test/ca-cert.pem"
133+
}
134+
EOF
135+
```
136+
137+
Now the smoke test will succeed.
138+
139+
## Configure Kafka for Mutual TLS (mTLS)
140+
141+
Stop the SSL Kafka server (Ctrl+C) and restart it with client certificate
142+
authentication required.
143+
144+
```
145+
$ cd ~/kafka_2.13-4.1.1
146+
$ sed 's/ssl.client.auth=none/ssl.client.auth=required/' config/server.properties.ssl > config/server.properties.mtls
147+
$ bin/kafka-server-start.sh config/server.properties.mtls
148+
```
149+
150+
## Mutual TLS `zync` Config/Test
151+
152+
If repeated, the `zync produce` from our [smoke test](#smoke-test) will now
153+
hang because the Kafka broker requires client certificate authentication, so
154+
we update the `zync` config to include client certificates.
155+
156+
```
157+
$ cat > ~/.zync/kafka.json << EOF
158+
{
159+
"bootstrap_servers": "localhost:9094",
160+
"security_protocol": "SSL",
161+
"tls_server_ca_cert_file": "/home/ubuntu/kafka-ssl-test/ca-cert.pem",
162+
"tls_client_cert_file": "/home/ubuntu/kafka-ssl-test/client-cert.pem",
163+
"tls_client_key_file": "/home/ubuntu/kafka-ssl-test/client-key.pem"
164+
}
165+
EOF
166+
```
167+
168+
Now the smoke test will succeed.
169+
170+
## Troubleshooting
171+
172+
If `zync produce` hangs, verify:
173+
- Kafka is running and listening on the expected port
174+
- Your `~/.zync/kafka.json` `bootstrap_servers` matches your Kafka listener port
175+
- The certificate paths in your config are correct and accessible

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,8 +99,9 @@ access credentials.
9999

100100
> We currently support no authentication, SASL/PLAIN authentication, and
101101
> TLS client authentication, though it will be easy to add other
102-
> authentication options. Please let us know if you have a requirement
103-
> here.
102+
> authentication options. The [Connection Guide](Connection-Guide.md)
103+
> walks through some sample configurations. Please let us know if you
104+
> have an additional requirement here.
104105
105106
## Description
106107

0 commit comments

Comments
 (0)