You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,14 +2,14 @@
2
2
3
3
This is a repository for real world [DataSQRL](https://github.com/DataSQRL/sqrl) use cases and examples.
4
4
5
-
***[Finance Credit Card Chatbot](finance-credit-card-chatbot/)**: Build a data pipeline that enriches and analyzes credit card transaction in real time and feeds the data into a GenAI chatbot to answer customer's questions about their transactions and spending. The extended example shows how to build a credit card rewards program and GenAI agent that sells credit cards.
6
-
***[Clickstream AI Recommendation](clickstream-ai-recommendation/)**: Build a personalized recommendation engine based on clickstream data and vector content embeddings generated by an LLM.
7
-
***[Healthcare Study](healthcare-study-monitoring/)**: Build a data pipeline for enriching healthcare data and querying it in realtime through an API, for data analytics in Iceberg, and publishing it to Kafka.
5
+
***[Finance Credit Card Chatbot](finance-credit-card-chatbot)**: Build a data pipeline that enriches and analyzes credit card transaction in real time and feeds the data into a GenAI chatbot to answer customer's questions about their transactions and spending. The extended example shows how to build a credit card rewards program and GenAI agent that sells credit cards.
6
+
***[Clickstream AI Recommendation](clickstream-ai-recommendation)**: Build a personalized recommendation engine based on clickstream data and vector content embeddings generated by an LLM.
7
+
***[Healthcare Study](healthcare-study)**: Build a data pipeline for enriching healthcare data and querying it in realtime through an API, for data analytics in Iceberg, and publishing it to Kafka.
8
8
***[Law Enforcement](law-enforcement)**: Build a realtime data pipeline for capturing and tracking warrants and Bolos.
9
9
***[Oil & Gas IoT Automation Agent](oil-gas-agent-automation)**: Build a realtime data enrichment pipeline that triggers an agent to analyze abnormal events for automated troubleshooting.
10
-
***[IoT Sensor Metrics](iot-sensor-metrics/)**: Build an event-driven microservice that ingests sensor metrics, processes them in realtime, and produces alerts and dashboards for users.
11
-
***[Logistics Shipping](logistics-shipping-geodata/)**: Build a data pipeline that processes logistics data to provide real-time tracking and shipment information for customers.
12
-
***[User Defined Function](user-defined-function/)**: This small tutorial shows how to include your call a custom function in your SQRL script.
10
+
***[IoT Sensor Metrics](iot-sensor-metrics)**: Build an event-driven microservice that ingests sensor metrics, processes them in realtime, and produces alerts and dashboards for users.
11
+
***[Logistics Shipping](logistics-shipping-geodata)**: Build a data pipeline that processes logistics data to provide real-time tracking and shipment information for customers.
12
+
***[User Defined Function](user-defined-function)**: This small tutorial shows how to include your call a custom function in your SQRL script.
13
13
14
14
## Running the Examples
15
15
@@ -41,4 +41,4 @@ docker run -it --rm -v $PWD:/build datasqrl/cmd:latest compile [ARGUMENTS GO HER
41
41
42
42
Check out the main [DataSQRL repository](https://github.com/DataSQRL/sqrl/) for more information on the compiler and runtime used in these examples.
43
43
44
-
Take a look at the [DataSQRL documentation](https://datasqrl.github.io/sqrl) to learn how to build your own project with DataSQRL.
44
+
Take a look at the [DataSQRL documentation](https://datasqrl.github.io/sqrl) to learn how to build your own project with DataSQRL.
This project contains two example use cases that process credit card transaction, customer, and merchant data:
4
4
5
-
1.**Transaction Analytics**: A data pipeline which enriches the credit card transactions with customer and merchant information to give customers an overview of their transactions as well as some analytics on their spending.
6
-
1. To run this data pipeline with file data, use the `creditcard_analytics_package_test.json` manifest file.
7
-
2. To run this data pipeline with Kafka as the data source, use the `creditcard_analytics_package_kafka.json` manifest file.
8
-
2.**Credit Card Rewards**: A data pipeline that implements a credit card rewards program. Merchants sign up for cash-back rewards on certain credit card types during certain periods and customer get a cash reward when they make a purchase at the merchant during that time. The data pipeline processes the rewards and give the customer insight into the reward they earned.
9
-
1. To run this data pipeline with file data, use the `creditcard_rewards_package_test.json` manifest file.
10
-
2. To run this data pipeline with Kafka as the data source, use the `creditcard_rewards_package_kafka.json` manifest file.
11
-
12
-
See below for detailed instructions on how to run each data pipeline. Note, that the instructions are for the *Transaction Analytics* use case. Replace the manifest files to run the *Credit Card Rewards* use case (i.e. `rewards` instead of `analytics` in the package JSON filename) - the instructions are otherwise identical.
5
+
*[**Transaction Analytics**](credit-card-analytics): A data pipeline which enriches the credit card
6
+
transactions with customer and merchant information to give customers an overview of their transactions as well as
7
+
some analytics on their spending.
8
+
* To run this data pipeline with file data, use the `creditcard_analytics_package_test.json` manifest file.
9
+
* To run this data pipeline with Kafka as the data source, use the `creditcard_analytics_package_kafka.json` manifest file.
10
+
*[**Credit Card Rewards**](credit-card-rewards): A data pipeline that implements a credit card rewards program.
11
+
Merchants sign up for cash-back rewards on certain credit card types during certain periods and customer get a cash
12
+
reward when they make a purchase at the merchant during that time. The data pipeline processes the rewards and give
13
+
the customer insight into the reward they earned.
14
+
* To run this data pipeline with file data, use the `creditcard_rewards_package_test.json` manifest file.
15
+
* To run this data pipeline with Kafka as the data source, use the `creditcard_rewards_package_kafka.json` manifest file.
16
+
17
+
See below for detailed instructions on how to run each data pipeline.
18
+
Note, that the instructions are for the *Transaction Analytics* use case.
19
+
Replace the manifest files to run the *Credit Card Rewards* use case (i.e. `rewards` instead of `analytics` in the
20
+
package JSON filename) - the instructions are otherwise identical.
13
21
14
22
## 1. Run the API with File data source
15
23
16
-
To run this example, invoke the following command in this directory on Unix based systems to compile the project
24
+
To run this example, invoke the following command in the example directory on Unix based systems to compile the project:
17
25
```bash
26
+
cd credit-card-analytics
18
27
docker run -it -p 8081:8081 -p 8888:8888 --rm -v $PWD:/build datasqrl/cmd:latest run -c creditcard_analytics_package_test.json
19
28
```
20
29
@@ -25,12 +34,14 @@ When you are done, you can stop the pipeline by hitting CTRL-C.
25
34
26
35
## 2. Run the API with Kafka data source
27
36
28
-
The instructions above run the data pipeline with data ingested from local files. While this is great for testing and local development, most production use cases need to ingest data from an external data source like Kafka.
37
+
The instructions above run the data pipeline with data ingested from local files. While this is great for testing and local development,
38
+
most production use cases need to ingest data from an external data source like Kafka.
29
39
30
40
To use Kafka as the data source, follow these steps:
31
41
32
-
Invoke the following command in this directory:
42
+
Invoke the following command in the example directory:
33
43
```bash
44
+
cd credit-card-analytics
34
45
docker run -it -p 8081:8081 -p 8888:8888 -p 9092:9092 --rm -v $PWD:/build datasqrl/cmd:latest run -c creditcard_analytics_package_kafka.json
35
46
```
36
47
@@ -51,28 +62,30 @@ The first load should be pretty quick. The transactions are then loaded at a rat
51
62
52
63
To see how the data enters the topics and the [Flink UI](http://localhost:8081/) to see the processing status.
53
64
54
-
As above, you can [open GraphiQL](http://localhost:8888/graphiql/) to access the API and query for data. Note, that the time windows are very long, so you won't be seeing any output there for the short period of time we are inserting data. You can adjust the time windows or keep loading data for a long time ;-).
65
+
As above, you can [open GraphiQL](http://localhost:8888/graphiql/) to access the API and query for data. Note, that the time windows are very long,
66
+
so you won't be seeing any output there for the short period of time we are inserting data.
67
+
You can adjust the time windows or keep loading data for a long time ;-).
55
68
56
69
When you are done, you can stop the pipeline by hitting CTRL-C.
57
70
58
-
## 3. Run the AI Data Agent
59
-
60
-
Both use cases can be extended with a Generative AI data agent:
61
-
1.**Transaction Analytics**: The data agent can answer customers' questions about their credit card transaction history and spending habits. The agent is defined in the folder `analytics-agent`.
62
-
2.**Credit Card Rewards**: The data agent can show customers the rewards they earned and sell customers on different types of credit cards to maximize their rewards. The agent is defined in the folder `rewards-agent`.
71
+
## 3. Securing the API endpoints
63
72
64
-
To run the data agent as a chatbot, you follow these steps:
73
+
To secure the public API endpoints, both examples include a JWT-specific package configuration that can be combined with
74
+
any of the main package configurations to secure the endpoints, regardless of other data source–specific configuration files.
65
75
66
-
1. Run the agent in docker:
76
+
For example, to secure the API with the File data source, invoke the following command in the specific example directory:
docker run -it -p 8081:8081 -p 8888:8888 --rm -v $PWD:/build datasqrl/cmd:latest run -c creditcard_analytics_package_test.json creditcard_analytics_package_jwt.json
69
80
```
70
-
* Replace `{ADD_YOUR_KEY}` with your OpenAI API key.
71
-
* To run the agent for the credit card rewards use case, replace the folder `analytics-agent` with `rewards-agent` and the GraphQL file `creditcard_analytics.graphqls` with `creditcard_rewards.graphqls`.
72
-
2. Open the [data agent chat](http://localhost:8080/) and enter a customer id (1-9) to "log in" as that customer. Then ask away. Questions like "what credit card would you recommend for me?" or "How many rewards did I earn?" or "How many rewards could I have earned?"
73
81
74
-
The example above uses OpenAI as the LLM model provider. To use a different LLM model provider, you can change the configuration file (i.e. the first argument that ends with `config.json`):
75
-
*`creditcard.bedrock-llama.config.json`: Uses Llama3 on AWS Bedrock.
76
-
*`creditcard.groq-llama.config.json`: Uses Llama3 on Groq.
82
+
## 4. Run the MCP Inspector
83
+
84
+
Both use cases can utilize the MCP inspector. If API is running, open a new terminal and invoke the following command:
85
+
```bash
86
+
npx @modelcontextprotocol/inspector
87
+
```
88
+
Assuming `npm` is installed, first time, this will install a new `npm` package, then it will start the inspector.
89
+
In the browser window that pop up, pick **Streamable HTTP**, and the SQRL MCP server should be available at `http://localhost:8888/mcp`.
77
90
78
-
You may have to update the model provider configuration to match your cloud configuration for Bedrock or Google Vertex.
91
+
All public API endpoints will be listed under. **Tools**. For more information about MCP inspector, [see its documentation](https://modelcontextprotocol.io/docs/tools/inspector).
0 commit comments