Skip to content

Commit 04426f7

Browse files
committed
revamp msl
1 parent e9da307 commit 04426f7

File tree

1 file changed

+37
-40
lines changed
  • src/content/docs/aws/services

1 file changed

+37
-40
lines changed

src/content/docs/aws/services/msk.md

Lines changed: 37 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
---
22
title: "Managed Streaming for Kafka (MSK)"
3-
linkTitle: "Managed Streaming for Kafka (MSK)"
43
description: Get started with Managed Streaming for Kafka (MSK) on LocalStack
54
tags: ["Ultimate"]
65
persistence: supported with limitations
@@ -13,7 +12,7 @@ MSK offers a centralized platform to facilitate seamless communication between v
1312
MSK also features automatic scaling and built-in monitoring, allowing users to build robust, high-throughput data pipelines.
1413

1514
LocalStack allows you to use the MSK APIs in your local environment to spin up Kafka clusters on the local machine, create topics for exchanging messages, and define event source mappings that trigger Lambda functions when messages are received on a certain topic.
16-
The supported APIs are available on our [API coverage page]({{< ref "coverage_kafka" >}}), which provides information on the extent of MSK's integration with LocalStack.
15+
The supported APIs are available on our [API coverage page](), which provides information on the extent of MSK's integration with LocalStack.
1716

1817
## Getting started
1918

@@ -43,13 +42,13 @@ Create the file and add the following content to it:
4342

4443
Run the following command to create the cluster:
4544

46-
{{< command >}}
47-
$ awslocal kafka create-cluster \
45+
```bash
46+
awslocal kafka create-cluster \
4847
--cluster-name "EventsCluster" \
4948
--broker-node-group-info file://brokernodegroupinfo.json \
5049
--kafka-version "2.8.0" \
5150
--number-of-broker-nodes 3
52-
{{< / command >}}
51+
```
5352

5453
The output of the command looks similar to this:
5554

@@ -65,10 +64,10 @@ The cluster creation process might take a few minutes.
6564
You can describe the cluster using the [`DescribeCluster`](https://docs.aws.amazon.com/msk/1.0/apireference/clusters.html#DescribeCluster) API.
6665
Run the following command, replacing `ClusterArn` with the Amazon Resource Name (ARN) you obtained above when you created cluster.
6766

68-
{{< command >}}
69-
$ awslocal kafka describe-cluster \
67+
```bash
68+
awslocal kafka describe-cluster \
7069
--cluster-arn "arn:aws:kafka:us-east-1:000000000000:cluster/EventsCluster/b154d18a-8ecb-4691-96b2-50348357fc2f-25"
71-
{{< / command >}}
70+
```
7271

7372
The output of the command looks similar to this:
7473

@@ -104,22 +103,22 @@ To use LocalStack MSK, you can download and utilize the Kafka command line inter
104103

105104
To download Apache Kafka, execute the following commands.
106105

107-
{{< command >}}
108-
$ wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.12-2.8.0.tgz
109-
$ tar -xzf kafka_2.12-2.8.0.tgz
110-
{{< / command >}}
106+
```bash
107+
wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.12-2.8.0.tgz
108+
tar -xzf kafka_2.12-2.8.0.tgz
109+
```
111110

112111
Navigate to the **kafka_2.12-2.8.0** directory.
113112
Execute the following command, replacing `ZookeeperConnectString` with the value you saved after running the [`DescribeCluster`](https://docs.aws.amazon.com/msk/1.0/apireference/clusters.html#DescribeCluster) API:
114113

115-
{{< command >}}
116-
$ bin/kafka-topics.sh \
114+
```bash
115+
bin/kafka-topics.sh \
117116
--create \
118117
--zookeeper localhost:4510 \
119118
--replication-factor 1 \
120119
--partitions 1 \
121120
--topic LocalMSKTopic
122-
{{< / command >}}
121+
```
123122

124123
After executing the command, your output should resemble the following:
125124

@@ -135,13 +134,13 @@ Create a folder named `/tmp` on the client machine, and navigate to the bin fold
135134
Run the following command, replacing `java_home` with the path of your `java_home`.
136135
For this instance, the java_home path is `/Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home`.
137136

138-
{{< callout >}}
137+
:::note
139138
The following step is optional and may not be required, depending on the operating system environment being used.
140-
{{< /callout >}}
139+
:::
141140

142-
{{< command >}}
143-
$ cp java_home/lib/security/cacerts /tmp/kafka.client.truststore.jks
144-
{{< / command >}}
141+
```bash
142+
cp java_home/lib/security/cacerts /tmp/kafka.client.truststore.jks
143+
```
145144

146145
While you are still in the `bin` folder of the Apache Kafka installation on the client machine, create a text file named `client.properties` with the following contents:
147146

@@ -151,10 +150,10 @@ ssl.truststore.location=/tmp/kafka.client.truststore.jks
151150

152151
Run the following command, replacing `ClusterArn` with the Amazon Resource Name (ARN) you have.
153152

154-
{{< command >}}
155-
$ awslocal kafka get-bootstrap-brokers \
153+
```bash
154+
awslocal kafka get-bootstrap-brokers \
156155
--cluster-arn ClusterArn
157-
{{< / command >}}
156+
```
158157

159158
To proceed with the following commands, save the value associated with the string named `BootstrapBrokerStringTls` from the JSON result obtained from the previous command.
160159
It should look like this:
@@ -167,12 +166,12 @@ It should look like this:
167166

168167
Now, navigate to the bin folder and run the next command, replacing `BootstrapBrokerStringTls` with the value you obtained:
169168

170-
{{< command >}}
171-
$ ./kafka-console-producer.sh \
169+
```bash
170+
./kafka-console-producer.sh \
172171
--broker-list BootstrapBrokerStringTls \
173172
--producer.config client.properties \
174173
--topic LocalMSKTopic
175-
{{< / command >}}
174+
```
176175

177176
To send messages to your Apache Kafka cluster, enter any desired message and press Enter.
178177
You can repeat this process twice or thrice, sending each line as a separate message to the Kafka cluster.
@@ -182,13 +181,13 @@ Keep the connection to the client machine open, and open a separate connection t
182181
In this new connection, navigate to the `bin` folder and run a command, replacing `BootstrapBrokerStringTls` with the value you saved earlier.
183182
This command will allow you to interact with the Apache Kafka cluster using the saved value for secure communication.
184183

185-
{{< command >}}
186-
$ ./kafka-console-consumer.sh \
184+
```bash
185+
./kafka-console-consumer.sh \
187186
--bootstrap-server BootstrapBrokerStringTls \
188187
--consumer.config client.properties \
189188
--topic LocalMSKTopic \
190189
--from-beginning
191-
{{< / command >}}
190+
```
192191

193192
You should start seeing the messages you entered earlier when you used the console producer command.
194193
These messages are TLS encrypted in transit.
@@ -201,13 +200,13 @@ The configuration for this mapping sets the starting position of the topic to `L
201200

202201
Run the following command to use the [`CreateEventSourceMapping`](https://docs.aws.amazon.com/lambda/latest/dg/API_CreateEventSourceMapping.html) API by specifying the Event Source ARN, the topic name, the starting position, and the Lambda function name.
203202

204-
{{< command >}}
205-
$ awslocal lambda create-event-source-mapping \
203+
```bash
204+
awslocal lambda create-event-source-mapping \
206205
--event-source-arn arn:aws:kafka:us-east-1:000000000000:cluster/EventsCluster \
207206
--topics LocalMSKTopic \
208207
--starting-position LATEST \
209208
--function-name my-kafka-function
210-
{{< / command >}}
209+
```
211210

212211
Upon successful completion of the operation to create the Lambda Event Source Mapping, you can expect the following response:
213212

@@ -240,24 +239,22 @@ You can delete the local MSK cluster using the [`DeleteCluster`](https://docs.aw
240239
To do so, you must first obtain the ARN of the cluster you want to delete.
241240
Run the following command to list all the clusters in the region:
242241

243-
{{< command >}}
244-
$ awslocal kafka list-clusters --region us-east-1
245-
{{< / command >}}
242+
```bash
243+
awslocal kafka list-clusters --region us-east-1
244+
```
246245

247246
To initiate the deletion of a cluster, select the corresponding `ClusterARN` from the list of clusters, and then execute the following command:
248247

249-
{{< command >}}
248+
```bash
250249
awslocal kafka delete-cluster --cluster-arn ClusterArn
251-
{{< / command >}}
250+
```
252251

253252
## Resource Browser
254253

255254
The LocalStack Web Application provides a Resource Browser for managing MSK clusters.
256255
You can access the Resource Browser by opening the LocalStack Web Application in your browser, navigating to the **Resources** section, and then clicking on **Kafka** under the **Analytics** section.
257256

258-
<img src="msk-resource-browser.png" alt="MSK Resource Browser" title="MSK Resource Browser" width="900" />
259-
<br>
260-
<br>
257+
![MSK Resource Browser](/images/aws/msk-resource-browser.png)
261258

262259
The Resource Browser allows you to perform the following actions:
263260

0 commit comments

Comments
 (0)