You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This guide is part of a series of guides that creates a real-time data pipeline. Read more about the series xref:streaming-learning:use-cases-architectures:real-time-data-pipeline/index.adoc[here].
10
10
11
-
== Creating message topics to stream click data
11
+
== Creating message topics to capture the stream of click data
12
12
13
13
. Navigate to your Astra portal home and choose "Create a stream"
. From the Astra portal home click “Create a Database”.
44
44
+
@@ -91,7 +91,7 @@ describe click_data;
91
91
+
92
92
The output will be 3 “create” CQL statements for the keyspace, the click_data.all_clicks table, and the click_data.product_clicks table
93
93
94
-
== Create a sink between your topics and the store
94
+
== Connecting the topics to the store
95
95
96
96
. From the Astra portal home, click on the “webstore-clicks” streaming tenant.
97
97
+
@@ -191,4 +191,4 @@ To debug you can click the sink name, scroll to the bottom terminal output area
191
191
192
192
== Next step
193
193
194
-
Now it's time see the magic! xref:real-time-data-pipeline/03-put-it-all-together.adoc[Run the pipelines >>]
194
+
With the Astra objects in place, now it's time to get the Decodable processing set up. xref:real-time-data-pipeline/03-put-it-all-together.adoc[Setup Decodable >>]
1.0, February 1, 2022: Creating the real-time data pipeline Decodable objects
4
4
@@ -10,7 +10,7 @@ This guide is part of a series of guides that creates a real-time data pipeline.
10
10
11
11
== The Astra Streaming connection info
12
12
13
-
To create the connection you are going to need a few things from your Astra Streaming tenant. First from the Astra portal home navigate to the webstore-clicks streaming tenant.
13
+
To connect Astra to Decodable you are going to need a few things from your Astra Streaming tenant. First from the Astra portal home navigate to the webstore-clicks streaming tenant.
14
14
15
15
image:decodable-data-pipeline/02/image4.png[]
16
16
@@ -22,21 +22,21 @@ You will also need a Pulsar token. Click the “Token Manager” link and “Cre
22
22
23
23
image:decodable-data-pipeline/02/image31.png[]
24
24
25
-
Navigate back to the “Connect” tab. We’ll come back here in a bit.
25
+
Close the popup and navigate back to the “Connect” tab. We’ll come back here in a bit.
26
26
27
27
== Creating a Decodable connection to Astra Streaming for all web clicks
28
28
29
-
In a different browser tab, sign in to your Decodable account and navigate to the “Connections” tab.
29
+
In a different browser tab, sign in to your Decodable account and navigate to the “Connections” tab. We are going to create a new connection and stream, that will direct all web clicks to the correct topic in Astra.
30
30
31
-
. We are going to create a new connection and stream, that will direct all web clicks to the correct topic in Astra. Click the “New Connection” button
31
+
. Click the “New Connection” button
32
32
+
33
33
image:decodable-data-pipeline/02/image25.png[]
34
34
35
-
. In the “Choose a Connector” view, locate the “DataStax Astra Streaming” connector and click “Connect”.
35
+
. In the “Choose a Connector” view, locate the DataStax Astra Streaming connector and click “Connect”.
36
36
+
37
37
image:decodable-data-pipeline/02/image14.png[]
38
38
39
-
. Use the Tenant Details from Astra to fill in the details about the connection as follows:
39
+
. Use the Tenant Details from Astra (in your other browser tab) to fill in the details about the connection as follows:
== Creating Decodable connections to Astra Streaming for product web clicks
110
108
111
-
Now let’s create another connection for the specific product clicks. Navigate back to the “Connections” area and click “New Connection”.
109
+
Create another connection to stream product clicks.
110
+
111
+
. Navigate back to the “Connections” area and click “New Connection”.
112
112
113
113
. Just as previously done, locate the Astra Streaming connection and use the Tenant Details from Astra to fill in the details about the connection as follows. Note the change in topic name.
Now we have a pipeline ready to filter by product.
300
-
303
+
. Now we have a pipeline ready to filter by product.
304
+
+
301
305
image:decodable-data-pipeline/02/image30.png[]
302
306
303
307
== Next step
304
308
305
-
With the Astra objects in place, now it's time to get the Decodable processing set up. xref:real-time-data-pipeline/03-put-it-all-together.adoc[Setup Decodable >>]
309
+
Now it's time see the magic! xref:real-time-data-pipeline/03-put-it-all-together.adoc[Run the pipelines >>]
. Now we can go to our Astra Tenant “webstore-clicks”, and navigate to the “Namespace and Topics” area, then expand the “Production” namespace, and click the “all-clicks” topic. Notice in the “Data In” area 1 message was produced and in the “Data Out” area 1 message was consumed.
73
+
. Now we can go to our Astra Streaming tenant “webstore-clicks” and navigate to the “Namespace and Topics” area. Expand the “Production” namespace and click the “all-clicks” topic. Notice that “Data In” has 1 message and “Data Out” has 1 message. That means the topic took the data in and a consumer acknowledged receipt of the message.
74
74
+
75
75
image:decodable-data-pipeline/03/image6.png[]
76
76
77
-
. Now on to the “Sinks” tab in Astra and click the “all-clicks” sink. In “Instance Stats” you see “Reads” has a value of 1 and “Writes” has a value of 1. This confirms our AstraDB sink received a message from the topic and wrote the data to our Cassandra table.
77
+
. On to the “Sinks” tab in Astra and click the “all-clicks” sink. In “Instance Stats” you see “Reads” has a value of 1 and “Writes” has a value of 1. This confirms the Sink consumed a message from the topic and wrote the data to the store.
78
78
+
79
79
image:decodable-data-pipeline/03/image5.png[]
80
80
81
-
. Let’s look at the final data in AstraDB. Navigate to the Astra home and click the “webstore-clicks” Serverless Database. Choose the “CQL Console” tab and copy/paste the following command in the terminal.
81
+
. Let’s look at the final data in Astra DB. Navigate to the Astra home and click the “webstore-clicks” Serverless Database. Choose the “CQL Console” tab and copy/paste the following command in the terminal.
82
82
+
83
83
[source,sql]
84
84
----
@@ -89,10 +89,12 @@ Your should see a single record output.
89
89
+
90
90
image:decodable-data-pipeline/03/image8.png[]
91
91
92
-
{emoji-tada}{emoji-tada} Queue the loud cheers and high-fives! Our pipeline ingested raw web click data, normalized it, and persisted the parsed data to the database!
92
+
{emoji-tada}{emoji-tada} Queue the loud cheers and high-fives! Our pipeline ingested raw web click data, normalized it, and persisted the parsed data to the database! Woot woot!!
93
93
94
94
== Follow the flow of the product clicks data
95
95
96
+
Similar to how you followed the above flow follow this flow to confirm the filtered messages were stored.
97
+
96
98
. Navigate to your Decodable pipeline named “Webstore-Product-Clicks-Pipeline”.
97
99
.. The “Input Metrics” should be 1 record and the “Output Metrics” should be 1 record.
Let’s see what this can do! To put a load on the pipeline we’ll need a way to continuously post data to our endpoint. Below are a few examples. Use the download button below to download a zip of a static(html) site ecommerce catalog, that silently posts click data to an endpoint. The site is a copy of https://www.blazemeter.com/[BlazeMeter^]’s{external-link-icon} https://www.demoblaze.com/[Demoblaze site^]{external-link-icon}.
120
122
121
-
You’ll need 2 pieces of information to attempt any of the examples, the Endpoint URL and an authorization token. Learn more about retrieving both of those in https://docs.decodable.co/docs/connector-reference-rest#authentication[Decodable documentation^]{external-link-icon}.
123
+
You’ll need 2 pieces of information the Endpoint URL and an authorization token. Learn more about retrieving both of those in https://docs.decodable.co/docs/connector-reference-rest#authentication[Decodable documentation^]{external-link-icon}.
122
124
123
125
Once you extract the zip, open the folder in your text editor of IDE of choice and look in the script.js file. There are 2 placeholders for the data retrieved above.
124
126
125
127
Open the phones.html file in your browser (yes, as a local file) and begin clicking on products. Each click should be a new post to your Decodable endpoint.
126
128
127
-
image::icons/button-get-started.png["Download web clicks website",130,xref=attachment$web-clicks-website.zip]
0 commit comments