You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/docs/aws/services/msk.md
+37-40Lines changed: 37 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,5 @@
1
1
---
2
2
title: "Managed Streaming for Kafka (MSK)"
3
-
linkTitle: "Managed Streaming for Kafka (MSK)"
4
3
description: Get started with Managed Streaming for Kafka (MSK) on LocalStack
5
4
tags: ["Ultimate"]
6
5
persistence: supported with limitations
@@ -13,7 +12,7 @@ MSK offers a centralized platform to facilitate seamless communication between v
13
12
MSK also features automatic scaling and built-in monitoring, allowing users to build robust, high-throughput data pipelines.
14
13
15
14
LocalStack allows you to use the MSK APIs in your local environment to spin up Kafka clusters on the local machine, create topics for exchanging messages, and define event source mappings that trigger Lambda functions when messages are received on a certain topic.
16
-
The supported APIs are available on our [API coverage page]({{< ref "coverage_kafka" >}}), which provides information on the extent of MSK's integration with LocalStack.
15
+
The supported APIs are available on our [API coverage page](), which provides information on the extent of MSK's integration with LocalStack.
17
16
18
17
## Getting started
19
18
@@ -43,13 +42,13 @@ Create the file and add the following content to it:
Execute the following command, replacing `ZookeeperConnectString` with the value you saved after running the [`DescribeCluster`](https://docs.aws.amazon.com/msk/1.0/apireference/clusters.html#DescribeCluster) API:
114
113
115
-
{{< command >}}
116
-
$ bin/kafka-topics.sh \
114
+
```bash
115
+
bin/kafka-topics.sh \
117
116
--create \
118
117
--zookeeper localhost:4510 \
119
118
--replication-factor 1 \
120
119
--partitions 1 \
121
120
--topic LocalMSKTopic
122
-
{{< / command >}}
121
+
```
123
122
124
123
After executing the command, your output should resemble the following:
125
124
@@ -135,13 +134,13 @@ Create a folder named `/tmp` on the client machine, and navigate to the bin fold
135
134
Run the following command, replacing `java_home` with the path of your `java_home`.
136
135
For this instance, the java_home path is `/Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home`.
137
136
138
-
{{< callout >}}
137
+
:::note
139
138
The following step is optional and may not be required, depending on the operating system environment being used.
While you are still in the `bin` folder of the Apache Kafka installation on the client machine, create a text file named `client.properties` with the following contents:
Run the following command, replacing `ClusterArn` with the Amazon Resource Name (ARN) you have.
153
152
154
-
{{< command >}}
155
-
$ awslocal kafka get-bootstrap-brokers \
153
+
```bash
154
+
awslocal kafka get-bootstrap-brokers \
156
155
--cluster-arn ClusterArn
157
-
{{< / command >}}
156
+
```
158
157
159
158
To proceed with the following commands, save the value associated with the string named `BootstrapBrokerStringTls` from the JSON result obtained from the previous command.
160
159
It should look like this:
@@ -167,12 +166,12 @@ It should look like this:
167
166
168
167
Now, navigate to the bin folder and run the next command, replacing `BootstrapBrokerStringTls` with the value you obtained:
169
168
170
-
{{< command >}}
171
-
$ ./kafka-console-producer.sh \
169
+
```bash
170
+
./kafka-console-producer.sh \
172
171
--broker-list BootstrapBrokerStringTls \
173
172
--producer.config client.properties \
174
173
--topic LocalMSKTopic
175
-
{{< / command >}}
174
+
```
176
175
177
176
To send messages to your Apache Kafka cluster, enter any desired message and press Enter.
178
177
You can repeat this process twice or thrice, sending each line as a separate message to the Kafka cluster.
@@ -182,13 +181,13 @@ Keep the connection to the client machine open, and open a separate connection t
182
181
In this new connection, navigate to the `bin` folder and run a command, replacing `BootstrapBrokerStringTls` with the value you saved earlier.
183
182
This command will allow you to interact with the Apache Kafka cluster using the saved value for secure communication.
184
183
185
-
{{< command >}}
186
-
$ ./kafka-console-consumer.sh \
184
+
```bash
185
+
./kafka-console-consumer.sh \
187
186
--bootstrap-server BootstrapBrokerStringTls \
188
187
--consumer.config client.properties \
189
188
--topic LocalMSKTopic \
190
189
--from-beginning
191
-
{{< / command >}}
190
+
```
192
191
193
192
You should start seeing the messages you entered earlier when you used the console producer command.
194
193
These messages are TLS encrypted in transit.
@@ -201,13 +200,13 @@ The configuration for this mapping sets the starting position of the topic to `L
201
200
202
201
Run the following command to use the [`CreateEventSourceMapping`](https://docs.aws.amazon.com/lambda/latest/dg/API_CreateEventSourceMapping.html) API by specifying the Event Source ARN, the topic name, the starting position, and the Lambda function name.
The LocalStack Web Application provides a Resource Browser for managing MSK clusters.
256
255
You can access the Resource Browser by opening the LocalStack Web Application in your browser, navigating to the **Resources** section, and then clicking on **Kafka** under the **Analytics** section.
0 commit comments