Skip to content

Commit 7021b0a

Browse files
ddierufddieruf
committed
Creating starlight guides
1 parent 7aea921 commit 7021b0a

File tree

4 files changed

+207
-0
lines changed

4 files changed

+207
-0
lines changed

‎modules/use-cases-architectures/pages/starlight/index.adoc

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
= DataStax Starlight Suite of APIs
2+
:description:
3+
:title: Get started with DataStax starlight quite of APIs
4+
:page-aliases: starlight::index.adoc,starlight-suite::index.adoc
5+
:navtitle: DataStax Starlight Suite of APIs
6+
7+
The Starlight suite of APIs are a collection of Apache Pulsar protocol handlers that extend an existing Pulsar cluster. Some establish wire level protocol XXXX while others follow an interface specification. The goal of all the APIs is to create seamless interact with a Pulsar cluster. There are three APIs within the Starlight suite.
8+
9+
== Starlight for JMS
10+
About the API +
11+
xref:use-cases-architectures:starlight/jms/index.adoc[Get started now]
12+
13+
== Starlight for Kafka
14+
About the API +
15+
xref:use-cases-architectures:starlight/kafka/index.adoc[Get started now]
16+
17+
== Starlight for RabbitMQ
18+
About the API +
19+
xref:use-cases-architectures:starlight/rabbitmq/index.adoc[Get started now]

‎modules/use-cases-architectures/pages/starlight/jms/index.adoc

Whitespace-only changes.

‎modules/use-cases-architectures/pages/starlight/kafka/index.adoc

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
= DataStax Starlight for Kafka Pulsar extension
2+
3+
Starlight for Kafka brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the Starlight for Kafka protocol handler to your existing Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code.
4+
5+
Visit the project's full documentation xref:starlight-for-kafka:ROOT:index.adoc[here].
6+
7+
== Architecture reference
8+
9+
|===
10+
a|image:s4k-architecture.png[Starlight for Kafka Architecture]
11+
|===
12+
13+
== Establishing the Kafka protocol handler
14+
15+
Before you can use an existing Kafka client with Pulsar you are going to need the Starlight for Kafka protocol handler installed in the Pulsar cluster. There are 3 popular ways to complete this:
16+
17+
[tabs]
18+
====
19+
Astra Streaming::
20+
+
21+
--
22+
. Sign in to your Astra account and navigate to your streaming tenant.
23+
+
24+
TIP: Don't have a streaming tenant? Follow our "xref:astra-streaming:getting-started:index.adoc[]" guide.
25+
26+
. Go to the "Connect" tab and choose the "Kafka" option.
27+
+
28+
image:astra-streaming-connect-kafka.png[Astra Streaming connect kafka]
29+
30+
. You will see an "Enable Kafka" button. Click to continue.
31+
+
32+
image:enable-kafka-button.png[Astra Streaming enable kafka]
33+
34+
. A message will let you know of the additions (and restrictions) that come with using Starlight for Kafka.
35+
image:enable-kafka-message.png[Astra Streaming enable kafka message]
36+
37+
. Click the "Enable Kafka" button to confirm your understanding.
38+
39+
Your Astra Streaming tenant is ready for prime time! Continue to the next section of the guide to see it in action.
40+
--
41+
Luna Streaming::
42+
+
43+
--
44+
The Starlight for Kafka extension is included in the `luna-streaming-all` image used to deploy your Luna cluster. The Luna helm chart makes deploying extensions quite easy. Follow the "xref:luna-streaming:components:starlight-for-kafka.adoc[]" guide to create a simple Pulsar cluster with the Starlight for Kafka extension ready for use.
45+
--
46+
Self Managed::
47+
+
48+
--
49+
Already got your own Pulsar Cluster? Or maybe your using a standalone cluster? Starlight for Kafka can easily be a part of that cluster! Follow the "xref:starlight-for-kafka:installation:starlight-kafka-quickstart.adoc[]" guide.
50+
--
51+
====
52+
53+
== Messaging with Starlight for Kafka
54+
55+
Starlight for Kafka supports quite a few different use cases. Remember there is a Pulsar cluster between the producer client and consumer client. Which means can interchange the type of producer and consumer that best fits your needs.
56+
57+
The below examples are using an Astra Streaming tenant as the Kafka bootstrap server. They assume you have completed the enablement steps above in the "Astra Streaming" tab.
58+
59+
[tabs]
60+
====
61+
Kafka CLI::
62+
+
63+
--
64+
65+
Download the latest Kafka dist https://www.apache.org/dyn/closer.cgi?path=/kafka/3.3.1/kafka_2.13-3.3.1.tgz[here]. With the tar ball extracted, the producer cli is in the 'bin' folder.
66+
67+
. To get started, let's set a few variables. If you've completed our "xref:astra-streaming:getting-started:index.adoc[Getting started with Astra Streaming]" guide, the below values will be a perfect fit for your existing tenant.
68+
+
69+
[source,shell]
70+
----
71+
SERVICE_URL="pulsar+ssl://pulsar-gcp-uscentral1.streaming.datastax.com:9951"
72+
TENANT="my-stream-<rand>"
73+
NAMESPACE="my-namespace"
74+
TOPIC="my-topic"
75+
----
76+
+
77+
[source,shell]
78+
----
79+
# cd kafka_2.13-3.3.1
80+
./bin/kafka-console-producer.sh --topic "$TENANT/$NAMESPACE/$TOPIC" --bootstrap-server "$SERVICE_URL"
81+
----
82+
83+
. If all goes as planned you will be in Kafka's producer shell. Type in a super memorable message, hit 'enter' to send, and then 'Ctrl-C' to exit the shell.
84+
+
85+
[source,shell]
86+
----
87+
> This is my first S4K message.
88+
----
89+
+
90+
A new message has been produced in the provided tenant/namespace/topic and is ready for consumption.
91+
92+
. Start the Kafka consumer shell.
93+
+
94+
[source,shell]
95+
----
96+
# cd kafka_2.13-3.3.1
97+
./bin/kafka-console-consumer.sh --topic "$TENANT/$NAMESPACE/$TOPIC" --from-beginning --bootstrap-server "$SERVICE_URL"
98+
----
99+
100+
. The consumer should immediately find the new message added before and output its value.
101+
+
102+
[source,shell]
103+
----
104+
This my first message
105+
----
106+
107+
. Press 'Ctrl-C' to exit the consumer shell.
108+
109+
Wow, you did it! Kafka producer and consumer with an Apache Pulsar cluster. How about trying the Java client now?
110+
--
111+
Kafka Client (Java)::
112+
+
113+
--
114+
While on the "Connect" tab in the Astra Streaming portal, the "kafka" area will provide important connection information. You will need that to create a working Kafka client.
115+
116+
image:kafka-client-settings.png[Astra Streaming kafka settings]
117+
118+
TIP: While reviewing the Kafka connection settings in the Astra portal, if you click the clipboard icon you will get those values as well as a working token to paste in code.
119+
120+
. Create a new java project. We use maven in this example, but you can choose other flavors.
121+
+
122+
[source,shell]
123+
----
124+
include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/create-project.sh[]
125+
----
126+
127+
. Open the new project in your favorite IDE or text editor and add the Kafka client dependency to "pom.xml".
128+
+
129+
[source,xml]
130+
----
131+
<dependency>
132+
<groupId>org.apache.kafka</groupId>
133+
<artifactId>kafka-clients</artifactId>
134+
<version>3.3.1</version>
135+
</dependency>
136+
----
137+
138+
. Open the file "src/main/java/org/example/App.java" and replace the entire contents with the below code. Notice there are variable values that need replacing. This is where you can use those Kafka connection values retrieved previously.
139+
+
140+
[source,java]
141+
----
142+
include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=init-app]
143+
----
144+
+
145+
NOTE: Don't worry if your editor shows errors, this isn't a complete program... yet.
146+
147+
. Next bring in the following code to build the configuration that will be used by both the producer and consumer.
148+
+
149+
[source,java]
150+
----
151+
include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=build-config]
152+
----
153+
154+
. Now paste the producer code below into the file. This is a very simple flow that sends a single message and awaits acknowledgment.
155+
+
156+
[source,java]
157+
----
158+
include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=build-producer]
159+
----
160+
161+
. Finally past the consume code below into the file. This creates a basic subscription and retrieves the latest messages on the topic.
162+
+
163+
[source,java]
164+
----
165+
include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=build-consumer]
166+
----
167+
168+
. Now you should have a complete program. Let's see it in action! Build and run the jar with the following terminal commands.
169+
+
170+
[source,shell]
171+
----
172+
mvn clean package assembly:single
173+
java -jar target/StarlightForKafkaClient-1.0-SNAPSHOT-jar-with-dependencies.jar
174+
----
175+
176+
. If all goes as it should your output will be similar to this:
177+
+
178+
[source,shell]
179+
----
180+
Successfully sent message
181+
182+
Found 1 total record(s)
183+
ConsumerRecord(topic = persistent://my-tenant-007/my-namespace/my-topic, partition = 0, leaderEpoch = null, offset = 22, CreateTime = 1673545962124, serialized key size = 8, serialized value size = 11, headers = RecordHeaders(headers = [], isReadOnly = false), key = ���h, value = Hello World)
184+
----
185+
186+
Congrats! You have just used the Kafka client to send and receive messages in Pulsar. Next stop is the moon!
187+
--
188+
====

‎modules/use-cases-architectures/pages/starlight/rabbitmq/index.adoc

Whitespace-only changes.

0 commit comments

Comments
 (0)