You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Starlight suite of APIs are a collection of Apache Pulsar protocol handlers that extend an existing Pulsar cluster. Some establish wire level protocol XXXX while others follow an interface specification. The goal of all the APIs is to create seamless interact with a Pulsar cluster. There are three APIs within the Starlight suite.
8
+
The Starlight suite of extensions is a collection of Apache Pulsar protocol handlers that extend an existing Pulsar cluster. The goal of all the extensions are to create a native, seamless interaction with a Pulsar cluster using existing tooling and clients.
8
9
9
10
== Starlight for Kafka
10
-
About the API +
11
-
xref:use-cases-architectures:starlight/kafka/index.adoc[Get started now]
11
+
12
+
Starlight for Kafka brings native Apache Kafka® protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers.
13
+
14
+
xref:use-cases-architectures:starlight/kafka/index.adoc[Get started now] | xref:starlight-for-kafka:ROOT:index.adoc[Configuring] | https://github.com/datastax/starlight-for-kafka[Source Code]
15
+
16
+
// == Starlight for RabbitMQ
17
+
//
18
+
// Starlight for RabbitMQ™ combines the industry-standard AMQP 0.9.1 (RabbitMQ) API with the cloud-native and horizontally scalable Pulsar streaming platform, providing a powerful way to modernize your RabbitMQ infrastructure, improve performance, and reduce costs.
19
+
//
20
+
// xref:use-cases-architectures:starlight/rabbitmq/index.adoc[Get started now] | xref:starlight-for-rabbitmq:ROOT:index.adoc[Configuring] | https://github.com/datastax/starlight-for-rabbitmq[Source Code^]
21
+
//
22
+
// == Starlight for JMS
23
+
//
24
+
// Starlight for JMS allows enterprises to take advantage of the scalability and resiliency of a modern streaming platform to run their existing JMS applications. Because Pulsar is open-source and cloud-native, Starlight for JMS enables enterprises to move their JMS applications to run on-premises and in any cloud environment.
25
+
//
26
+
// xref:use-cases-architectures:starlight/jms/index.adoc[Get started now] | xref:starlight-for-jms:ROOT:index.adoc[Configuring] | https://github.com/datastax/starlight-for-jms[Source Code^]
Copy file name to clipboardExpand all lines: modules/use-cases-architectures/pages/starlight/kafka/index.adoc
+49-30Lines changed: 49 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,24 +1,29 @@
1
-
= DataStax Starlight for Kafka Pulsar extension
1
+
= Getting started with the Starlight for Kafka extension
2
+
3
+
:description: Learn how to get started using the Starlight for Kafka extension with Pulsar and get hands on with Kafka producer and consumer interacting with a topic.
4
+
:title: Getting started with the Starlight for Kafka extension
5
+
:navtitle: Kafka
2
6
3
7
Starlight for Kafka brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the Starlight for Kafka protocol handler to your existing Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code.
4
8
5
-
Visit the project's full documentation xref:starlight-for-kafka:ROOT:index.adoc[here].
9
+
If source code is your thing, visit the https://github.com/datastax/starlight-for-kafka[project's repo on GitHub^]{external-link-icon}.
6
10
7
11
== Architecture reference
8
12
9
-
|===
10
-
a|image:s4k-architecture.png[Starlight for Kafka Architecture]
11
-
|===
13
+
image:s4k-architecture.png[Starlight for Kafka Architecture]
12
14
13
15
== Establishing the Kafka protocol handler
14
16
15
-
Before you can use an existing Kafka client with Pulsar you are going to need the Starlight for Kafka protocol handler installed in the Pulsar cluster. There are 3 popular ways to complete this:
17
+
Before you can use a Kafka client can interact with your Pulsar cluster, you need the Starlight for Kafka protocol handler installed in the cluster. Installation looks a bit different depending on where your Pulsar cluster is running. Choose the option that best fits your needs.
16
18
17
19
[tabs]
18
20
====
19
21
Astra Streaming::
20
22
+
21
23
--
24
+
25
+
If you want a working Kafka extension as quickly as possible, this is your best bet. This is also a good option for those that already have a streaming tenant and are looking to extend it.
26
+
22
27
. Sign in to your Astra account and navigate to your streaming tenant.
23
28
+
24
29
TIP: Don't have a streaming tenant? Follow our "xref:astra-streaming:getting-started:index.adoc[]" guide.
@@ -41,7 +46,7 @@ Your Astra Streaming tenant is ready for prime time! Continue to the next sectio
41
46
Luna Streaming::
42
47
+
43
48
--
44
-
The Starlight for Kafka extension is included in the `luna-streaming-all` image used to deploy your Luna cluster. The Luna helm chart makes deploying extensions quite easy. Follow the "xref:luna-streaming:components:starlight-for-kafka.adoc[]" guide to create a simple Pulsar cluster with the Starlight for Kafka extension ready for use.
49
+
The Starlight for Kafka extension is included in the `luna-streaming-all` image used to deploy a Luna cluster. The Luna helm chart makes deploying the Kafka extension quite easy. Follow the "xref:luna-streaming:components:starlight-for-kafka.adoc[]" guide to create a simple Pulsar cluster with the Starlight for Kafka extension ready for use.
45
50
--
46
51
Self Managed::
47
52
+
@@ -52,41 +57,50 @@ Already got your own Pulsar Cluster? Or maybe your using a standalone cluster? S
52
57
53
58
== Messaging with Starlight for Kafka
54
59
55
-
Starlight for Kafka supports quite a few different use cases. Remember there is a Pulsar cluster between the producer client and consumer client. Which means can interchange the type of producer and consumer that best fits your needs.
60
+
Starlight for Kafka supports quite a few different use cases. With a Pulsar cluster between producers and consumers you can interchange the type of producer and consumer to fit your needs. *The below examples are using an Astra Streaming tenant as the Kafka bootstrap server.* If you are using Luna or self-managed, switch the bootstrap server URL for your own.
61
+
62
+
=== Retrieve Kafka connection properties in Astra Streaming
63
+
64
+
While on the "Connect" tab in the Astra Streaming portal, the "kafka" area will provide important connection information. You will need that to create a working Kafka client or using the CLI.
56
65
57
-
The below examples are using an Astra Streaming tenant as the Kafka bootstrap server. They assume you have completed the enablement steps above in the "Astra Streaming" tab.
TIP: While reviewing the Kafka connection settings in the Astra portal, if you click the clipboard icon you will get those values as well as a working token to paste in code.
69
+
70
+
=== Produce and consume a message
58
71
59
72
[tabs]
60
73
====
61
74
Kafka CLI::
62
75
+
63
76
--
64
-
65
-
Download the latest Kafka dist https://www.apache.org/dyn/closer.cgi?path=/kafka/3.3.1/kafka_2.13-3.3.1.tgz[here]. With the tar ball extracted, the producer cli is in the 'bin' folder.
77
+
Download the latest Kafka dist https://www.apache.org/dyn/closer.cgi?path=/kafka/3.3.1/kafka_2.13-3.3.1.tgz[here^]{external-link-icon}. With the tar ball extracted, the producer and consumer cli's are in the 'bin' folder.
66
78
67
79
. To get started, let's set a few variables. If you've completed our "xref:astra-streaming:getting-started:index.adoc[Getting started with Astra Streaming]" guide, the below values will be a perfect fit for your existing tenant.
. If all goes as planned you will be in Kafka's producer shell. Type in a super memorable message, hit 'enter' to send, and then 'Ctrl-C' to exit the shell.
97
+
. Type in a super memorable messages and hit 'enter' to send. Press 'Ctrl-C' to exit the shell.
84
98
+
85
99
[source,shell]
86
100
----
87
101
> This is my first S4K message.
88
102
----
89
-
+
103
+
90
104
A new message has been produced in the provided tenant/namespace/topic and is ready for consumption.
91
105
92
106
. Start the Kafka consumer shell.
@@ -97,11 +111,11 @@ A new message has been produced in the provided tenant/namespace/topic and is re
. The consumer should immediately find the new message added before and output its value.
114
+
. The consumer should immediately find the new message and output its value.
101
115
+
102
116
[source,shell]
103
117
----
104
-
This my first message
118
+
This is my first S4K message.
105
119
----
106
120
107
121
. Press 'Ctrl-C' to exit the consumer shell.
@@ -111,13 +125,11 @@ Wow, you did it! Kafka producer and consumer with an Apache Pulsar cluster. How
111
125
Kafka Client (Java)::
112
126
+
113
127
--
114
-
While on the "Connect" tab in the Astra Streaming portal, the "kafka" area will provide important connection information. You will need that to create a working Kafka client.
128
+
This example uses maven as the project structure. If you prefer gradle or another, this code should still be a good fit.
TIP: Visit our https://github.com/datastax/astra-streaming-examples[examples repo^]{external-link-icon} to see the complete source of this example.
117
131
118
-
TIP: While reviewing the Kafka connection settings in the Astra portal, if you click the clipboard icon you will get those values as well as a working token to paste in code.
119
-
120
-
. Create a new java project. We use maven in this example, but you can choose other flavors.
. Open the file "src/main/java/org/example/App.java" and replace the entire contents with the below code. Notice there are variable values that need replacing. This is where you can use those Kafka connection values retrieved previously.
150
+
. Open the file "src/main/java/org/example/App.java" and replace the entire contents with the below code. Notice there are class variables that need replacing. Apply the values previously retrieved in Astra Streaming.
0 commit comments