|
| 1 | += Getting started with the Starlight for Kafka extension |
| 2 | + |
| 3 | +:description: Learn how to get started using the Starlight for Kafka extension with Pulsar and get hands on with Kafka producer and consumer interacting with a topic. |
| 4 | +:title: Getting started with the Starlight for Kafka extension |
| 5 | +:navtitle: Kafka |
| 6 | + |
| 7 | +Starlight for Kafka brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the Starlight for Kafka protocol handler to your existing Pulsar cluster, you can migrate your existing Kafka applications and services to Pulsar without modifying the code. |
| 8 | + |
| 9 | +If source code is your thing, visit the https://github.com/datastax/starlight-for-kafka[project's repo on GitHub^]{external-link-icon}. |
| 10 | + |
| 11 | +== Architecture reference |
| 12 | + |
| 13 | +image:s4k-architecture.png[Starlight for Kafka Architecture] |
| 14 | + |
| 15 | +== Establishing the Kafka protocol handler |
| 16 | + |
| 17 | +Before a Kafka client can interact with your Pulsar cluster, you need the Starlight for Kafka protocol handler installed in the cluster. |
| 18 | +Installation looks a bit different depending on where your Pulsar cluster is running. |
| 19 | +Choose the option that best fits your needs. |
| 20 | + |
| 21 | +[tabs] |
| 22 | +==== |
| 23 | +Astra Streaming:: |
| 24 | ++ |
| 25 | +-- |
| 26 | +
|
| 27 | +If you want a working Kafka extension as quickly as possible, this is your best bet. |
| 28 | +This is also a good option for those that already have a streaming tenant and are looking to extend it. |
| 29 | +
|
| 30 | +. Sign in to your Astra account and navigate to your streaming tenant. |
| 31 | ++ |
| 32 | +TIP: Don't have a streaming tenant? Follow our "xref:astra-streaming:getting-started:index.adoc[]" guide. |
| 33 | +
|
| 34 | +. Go to the "Connect" tab and choose the "Kafka" option. |
| 35 | ++ |
| 36 | +image:astra-streaming-connect-kafka.png[Astra Streaming connect kafka] |
| 37 | +
|
| 38 | +. You will see an "Enable Kafka" button. Click to continue. |
| 39 | ++ |
| 40 | +image:enable-kafka-button.png[Astra Streaming enable kafka] |
| 41 | +
|
| 42 | +. A message will let you know of the additions (and restrictions) that come with using Starlight for Kafka. |
| 43 | +image:enable-kafka-message.png[Astra Streaming enable kafka message] |
| 44 | +
|
| 45 | +. Select the "Enable Kafka" button to confirm your understanding. |
| 46 | +
|
| 47 | +Your Astra Streaming tenant is ready for prime time! Continue to the next section of the guide to see it in action. |
| 48 | +-- |
| 49 | +Luna Streaming:: |
| 50 | ++ |
| 51 | +-- |
| 52 | +The Starlight for Kafka extension is included in the `luna-streaming-all` image used to deploy a Luna cluster. The Luna helm chart makes deploying the Kafka extension quite easy. Follow the "xref:luna-streaming:components:starlight-for-kafka.adoc[]" guide to create a simple Pulsar cluster with the Starlight for Kafka extension ready for use. |
| 53 | +-- |
| 54 | +Self Managed:: |
| 55 | ++ |
| 56 | +-- |
| 57 | +Already have your own Pulsar cluster? Or maybe you're using a standalone cluster? Starlight for Kafka can easily be a part of that cluster! Follow the "xref:starlight-for-kafka:installation:starlight-kafka-quickstart.adoc[]" guide. |
| 58 | +-- |
| 59 | +==== |
| 60 | + |
| 61 | +== Messaging with Starlight for Kafka |
| 62 | + |
| 63 | +Starlight for Kafka supports quite a few different use cases. With a Pulsar cluster between producers and consumers you can interchange the type of producer and consumer to fit your needs. |
| 64 | + |
| 65 | +*The below examples are using an Astra Streaming tenant as the Kafka bootstrap server.* If you are using Luna Streaming or a self-managed tenant, switch the bootstrap server URL for your own. |
| 66 | + |
| 67 | +=== Retrieve Kafka connection properties in Astra Streaming |
| 68 | + |
| 69 | +In the Astra Streaming portal "Connect" tab, the "kafka" area provides important connection information. |
| 70 | +You will need this connection information to create a working Kafka client or use the CLI. |
| 71 | + |
| 72 | +image:kafka-client-settings.png[Astra Streaming kafka settings] |
| 73 | + |
| 74 | +TIP: Click the clipboard icon to copy the Kafka connection values, as well as a working token to paste in code. |
| 75 | + |
| 76 | +=== Produce and consume a message |
| 77 | + |
| 78 | +[tabs] |
| 79 | +==== |
| 80 | +Kafka CLI:: |
| 81 | ++ |
| 82 | +-- |
| 83 | +Download the latest Kafka distribution https://www.apache.org/dyn/closer.cgi?path=/kafka/3.3.1/kafka_2.13-3.3.1.tgz[here^]{external-link-icon}. |
| 84 | +With the tarball extracted, the producer and consumer CLIs are in the 'bin' folder. |
| 85 | +
|
| 86 | +. To get started, let's set a few variables. If you've completed our "xref:astra-streaming:getting-started:index.adoc[Getting started with Astra Streaming]" guide, the below values will be a perfect fit for your existing tenant. |
| 87 | ++ |
| 88 | +[source,shell] |
| 89 | +---- |
| 90 | +SERVICE_URL="<REPLACE_WITH_BOOTSTRAP_SERVER_URL>" |
| 91 | +TENANT="<REPLACE_WITH_TENANT_NAME>" |
| 92 | +NAMESPACE="<REPLACE_WITH_NAMESPACE>" |
| 93 | +TOPIC="<REPLACE_WITH_TOPIC>" |
| 94 | +---- |
| 95 | +
|
| 96 | +. Now let's enter those variables in Kafka's producer shell. |
| 97 | ++ |
| 98 | +[source,shell] |
| 99 | +---- |
| 100 | +# cd kafka_2.13-3.3.1 |
| 101 | +./bin/kafka-console-producer.sh --topic "$TENANT/$NAMESPACE/$TOPIC" --bootstrap-server "$SERVICE_URL" |
| 102 | +---- |
| 103 | +
|
| 104 | +. Type in a super memorable message and hit 'enter' to send. Press 'Ctrl-C' to exit the shell. |
| 105 | ++ |
| 106 | +[source,shell] |
| 107 | +---- |
| 108 | +> This is my first S4K message. |
| 109 | +---- |
| 110 | +A new message has been produced in the provided tenant/namespace/topic and is ready for consumption. |
| 111 | +. Start the Kafka consumer shell. |
| 112 | ++ |
| 113 | +[source,shell] |
| 114 | +---- |
| 115 | +# cd kafka_2.13-3.3.1 |
| 116 | +./bin/kafka-console-consumer.sh --topic "$TENANT/$NAMESPACE/$TOPIC" --from-beginning --bootstrap-server "$SERVICE_URL" |
| 117 | +---- |
| 118 | +
|
| 119 | +. The consumer should immediately find the new message and output its value. |
| 120 | ++ |
| 121 | +[source,shell] |
| 122 | +---- |
| 123 | +This is my first S4K message. |
| 124 | +---- |
| 125 | +
|
| 126 | +. Press 'Ctrl-C' to exit the consumer shell. |
| 127 | +
|
| 128 | +Wow, you did it! A Kafka producer and consumer with an Apache Pulsar cluster. How about trying the Java client now? |
| 129 | +-- |
| 130 | +Kafka Client (Java):: |
| 131 | ++ |
| 132 | +-- |
| 133 | +This example uses Maven for the project structure. |
| 134 | +If you prefer Gradle or another tool, this code should still be a good fit. |
| 135 | +
|
| 136 | +TIP: Visit our https://github.com/datastax/astra-streaming-examples[examples repo^]{external-link-icon} to see the complete source for this example. |
| 137 | +
|
| 138 | +. Create a new Maven project. |
| 139 | ++ |
| 140 | +[source,shell] |
| 141 | +---- |
| 142 | +include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/create-project.sh[] |
| 143 | +---- |
| 144 | +
|
| 145 | +. Open the new project in your favorite IDE or text editor and add the Kafka client dependency to "pom.xml". |
| 146 | ++ |
| 147 | +[source,xml] |
| 148 | +---- |
| 149 | +<dependency> |
| 150 | + <groupId>org.apache.kafka</groupId> |
| 151 | + <artifactId>kafka-clients</artifactId> |
| 152 | + <version>3.3.1</version> |
| 153 | +</dependency> |
| 154 | +---- |
| 155 | +
|
| 156 | +. Open the file "src/main/java/org/example/App.java" and replace the entire contents with the below code. Notice there are class variables that need replacing. Apply the values previously retrieved in Astra Streaming. |
| 157 | ++ |
| 158 | +[source,java] |
| 159 | +---- |
| 160 | +include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=init-app] |
| 161 | +---- |
| 162 | ++ |
| 163 | +NOTE: Don't worry if your editor shows errors, this isn't a complete program... yet. |
| 164 | +
|
| 165 | +. Bring in the following code to build the configuration that will be used by both the producer and consumer. |
| 166 | ++ |
| 167 | +[source,java] |
| 168 | +---- |
| 169 | +include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=build-config] |
| 170 | +---- |
| 171 | +
|
| 172 | +. Now paste the producer code into the file. This is a very simple flow that sends a single message and awaits acknowledgment. |
| 173 | ++ |
| 174 | +[source,java] |
| 175 | +---- |
| 176 | +include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=build-producer] |
| 177 | +---- |
| 178 | +
|
| 179 | +. Paste the consumer code into the file. This creates a basic subscription and retrieves the latest messages on the topic. |
| 180 | ++ |
| 181 | +[source,java] |
| 182 | +---- |
| 183 | +include::{astra-streaming-examples-repo}/java/starlight-for-kafka/kafka-client/StarlightForKafkaClient/src/main/java/org/example/App.java[tag=build-consumer] |
| 184 | +---- |
| 185 | +
|
| 186 | +. Now you should have a complete program. Let's see it in action! Build and run the jar with the following terminal commands. |
| 187 | ++ |
| 188 | +[source,shell] |
| 189 | +---- |
| 190 | +mvn clean package assembly:single |
| 191 | +java -jar target/StarlightForKafkaClient-1.0-SNAPSHOT-jar-with-dependencies.jar |
| 192 | +---- |
| 193 | +
|
| 194 | +. If all goes as it should, your output will be similar to this: |
| 195 | ++ |
| 196 | +[source,shell] |
| 197 | +---- |
| 198 | +Successfully sent message |
| 199 | +
|
| 200 | +Found 1 total record(s) |
| 201 | +ConsumerRecord(topic = persistent://my-tenant-007/my-namespace/my-topic, partition = 0, leaderEpoch = null, offset = 22, CreateTime = 1673545962124, serialized key size = 8, serialized value size = 11, headers = RecordHeaders(headers = [], isReadOnly = false), key = xxxxx, value = Hello World) |
| 202 | +---- |
| 203 | +
|
| 204 | +Congrats! You have just used the Kafka client to send and receive messages in Pulsar. Next stop is the moon! |
| 205 | +-- |
| 206 | +==== |
| 207 | + |
| 208 | +The Starlight for Kafka documentation provides more specifics about the below topics and more. Visit those for more detail. |
| 209 | + |
| 210 | +* xref:starlight-for-kafka:operations:starlight-kafka-kstreams.adoc[] |
| 211 | +* xref:starlight-for-kafka:operations:starlight-kafka-implementation.adoc[] |
| 212 | +* xref:starlight-for-kafka:operations:starlight-kafka-monitor.adoc[Monitoring] |
| 213 | +* xref:starlight-for-kafka:operations:starlight-kafka-security.adoc[] |
0 commit comments