Skip to content

Commit 4cb545b

Browse files
v-dirichardsv-dirichards
authored
Merge pull request #268083 from sreekzz/patch-1
New Page start-sql-client-cli-in-gateway-mode
2 parents 63cd2aa + 83e59c8 commit 4cb545b

File tree

6 files changed

+212
-1
lines changed

6 files changed

+212
-1
lines changed

‎articles/hdinsight-aks/TOC.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,9 @@ items:
178178
- name: Hive dialect in Flink
179179
href: ./flink/hive-dialect-flink.md
180180
- name: Table API and SQL
181-
href: ./flink/flink-table-api-and-sql.md
181+
href: ./flink/flink-table-api-and-sql.md
182+
- name: SQL gateway
183+
href: ./flink/start-sql-client-cli-gateway-mode.md
182184
- name: How-to guides
183185
items:
184186
- name: General

‎articles/hdinsight-aks/flink/media/start-sql-client-cli-in-gateway-mode/allow-public-ip.png

80.4 KB
Loading

‎articles/hdinsight-aks/flink/media/start-sql-client-cli-in-gateway-mode/check-sql-gateway.png

157 KB
Loading

‎articles/hdinsight-aks/flink/media/start-sql-client-cli-in-gateway-mode/get-cluster-endpoint.png

23.2 KB
Loading

‎articles/hdinsight-aks/flink/media/start-sql-client-cli-in-gateway-mode/jar-files-in-azure-portal.png

104 KB
Loading

‎articles/hdinsight-aks/flink/start-sql-client-cli-gateway-mode.md

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
---
2+
title: Start SQL Client CLI in gateway mode in Apache Flink Cluster 1.17.0 on HDInsight on AKS.
3+
description: Learn how to start SQL Client CLI in gateway mode in Apache Flink Cluster 1.17.0 on HDInsight on AKS.
4+
ms.service: hdinsight-aks
5+
ms.topic: how-to
6+
ms.date: 03/07/2024
7+
---
8+
9+
# Start SQL Client CLI in gateway mode
10+
11+
[!INCLUDE [feature-in-preview](../includes/feature-in-preview.md)]
12+
13+
This tutorial guides you how to start the SQL Client CLI in gateway mode in Apache Flink Cluster 1.17.0 on HDInsight on AKS. In the gateway mode, the CLI submits the SQL to the specified remote gateway to execute statements.
14+
15+
```
16+
./bin/sql-client.sh gateway --endpoint <gateway address>
17+
```
18+
19+
> [!NOTE]
20+
> In Apache Flink Cluster on HDInsight on AKS, any external connection will go via 443 port. But internally, it will reroute the request to sql-gateway service listening to port 8083.
21+
22+
Check sql gateway service in AKS side:
23+
24+
:::image type="image" source="./media/start-sql-client-cli-in-gateway-mode/check-sql-gateway.png" alt-text="Screenshot showing how to check SQL gateway." border="true" lightbox="./media/start-sql-client-cli-in-gateway-mode/check-sql-gateway.png":::
25+
26+
## What is SQL Client in Flink?
27+
28+
Flink’s Table & SQL API makes it possible to work with queries that written in the SQL language, but these queries need embed within a table program written in either Java or Scala. Moreover, these programs need to be packaged with a build tool before being submitted to a cluster. This feature limits the usage of Flink to Java/Scala programmers.
29+
30+
The SQL Client aims to provide an easy way of writing, debugging, and submitting table programs to a Flink cluster without a single line of Java or Scala code. The SQL Client CLI allows for retrieving and visualizing real-time results from the running distributed application on the command line.
31+
32+
For more information, see [how to enter the Flink SQL CLI client on webssh](https://supportability.visualstudio.com/AzureHDinsight/_wiki/wikis/AzureHDinsight/833819/-Hilo-sql_client-on-webssh-To-Flink-SQL).
33+
34+
## What is SQL Gateway in Flink
35+
36+
The SQL Gateway is a service that enables multiple clients from the remote to execute SQL in concurrency. It provides an easy way to submit the Flink Job, look up the metadata, and analyze the data online.
37+
38+
For more information, see [SQL Gateway](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql-gateway/overview).
39+
40+
## Start the SQL Client CLI in gateway mode in Flink-cli
41+
42+
In Apache Flink Cluster on HDInsight on AKS, start the SQL Client CLI in gateway mode by running command:
43+
44+
```
45+
./bin/sql-client.sh gateway --endpoint host:port
46+
47+
or
48+
49+
./bin/sql-client.sh gateway --endpoint fqdn:443
50+
```
51+
52+
Get cluster endpoint(host or fqdn) on Azure portal.
53+
54+
:::image type="image" source="./media/start-sql-client-cli-in-gateway-mode/get-cluster-endpoint.png" alt-text="Screenshot showing cluster endpoint." border="true" lightbox="./media/start-sql-client-cli-in-gateway-mode/get-cluster-endpoint.png":::
55+
56+
## Testing
57+
58+
### Preparation
59+
#### Download Flink CLI
60+
61+
1. Download Flink CLI from https://aka.ms/hdionaksflink117clilinux in local Windows machine.
62+
63+
#### Install Windows Subsystem for Linux to make this work on local Windows machine.
64+
65+
1. Open Windows command and run (replace JAVA_HOME and flink-cli path with your own) to download flink-cli:
66+
67+
```
68+
Windows Subsystem for Linux --distribution Ubuntu
69+
export JAVA_HOME=/mnt/c/Work/99_tools/zulu11.56.19-ca-jdk11.0.15-linux_x64
70+
cd <folder>
71+
wget https://hdiconfigactions.blob.core.windows.net/hiloflink17blob/flink-cli.tgz
72+
tar -xvf flink-cli.tgz
73+
```
74+
1. Set endpoint, tenant ID, and port 443 in flink-conf.yaml
75+
```
76+
user@MININT-481C9TJ:/mnt/c/Users/user/flink-cli$ cd conf
77+
user@MININT-481C9TJ:/mnt/c/Users/user/flink-cli/conf$ ls -l
78+
total 8
79+
-rwxrwxrwx 1 user user 2451 Feb 26 20:33 flink-conf.yaml
80+
-rwxrwxrwx 1 user user 2946 Feb 23 14:13 log4j-cli.properties
81+
82+
user@MININT-481C9TJ:/mnt/c/Users/user/flink-cli/conf$ cat flink-conf.yaml
83+
84+
rest.address: <flink cluster endpoint on Azure portal>
85+
azure.tenant.id: <tenant ID>
86+
rest.port: 443
87+
```
88+
1. Allowlist Local Windows public IP with port 443 with VPN enabled into HDInsight on AKS cluster Subnet's Network security inbound.
89+
90+
:::image type="image" source="./media/start-sql-client-cli-in-gateway-mode/allow-public-ip.png" alt-text="Screenshot showing how to allow public IP address." border="true" lightbox="./media/start-sql-client-cli-in-gateway-mode/allow-public-ip.png":::
91+
92+
1. Run the sql-client.sh in gateway mode on Flink-cli to Flink SQL.
93+
94+
```
95+
bin/sql-client.sh gateway --endpoint <fqdn>:443
96+
```
97+
98+
Example
99+
```
100+
user@MININT-481C9TJ:/mnt/c/Users/user/flink-cli$ bin/sql-client.sh gateway --endpoint <fqdn:443>
101+
102+
▒▓██▓██▒
103+
▓████▒▒█▓▒▓███▓▒
104+
▓███▓░░ ▒▒▒▓██▒ ▒
105+
░██▒ ▒▒▓▓█▓▓▒░ ▒████
106+
██▒ ░▒▓███▒ ▒█▒█▒
107+
░▓█ ███ ▓░▒██
108+
▓█ ▒▒▒▒▒▓██▓░▒░▓▓█
109+
█░ █ ▒▒░ ███▓▓█ ▒█▒▒▒
110+
████░ ▒▓█▓ ██▒▒▒ ▓███▒
111+
░▒█▓▓██ ▓█▒ ▓█▒▓██▓ ░█░
112+
▓░▒▓████▒ ██ ▒█ █▓░▒█▒░▒█▒
113+
███▓░██▓ ▓█ █ █▓ ▒▓█▓▓█▒
114+
░██▓ ░█░ █ █▒ ▒█████▓▒ ██▓░▒
115+
███░ ░ █░ ▓ ░█ █████▒░░ ░█░▓ ▓░
116+
██▓█ ▒▒▓▒ ▓███████▓░ ▒█▒ ▒▓ ▓██▓
117+
▒██▓ ▓█ █▓█ ░▒█████▓▓▒░ ██▒▒ █ ▒ ▓█▒
118+
▓█▓ ▓█ ██▓ ░▓▓▓▓▓▓▓▒ ▒██▓ ░█▒
119+
▓█ █ ▓███▓▒░ ░▓▓▓███▓ ░▒░ ▓█
120+
██▓ ██▒ ░▒▓▓███▓▓▓▓▓██████▓▒ ▓███ █
121+
▓███▒ ███ ░▓▓▒░░ ░▓████▓░ ░▒▓▒ █▓
122+
█▓▒▒▓▓██ ░▒▒░░░▒▒▒▒▓██▓░ █▓
123+
██ ▓░▒█ ▓▓▓▓▒░░ ▒█▓ ▒▓▓██▓ ▓▒ ▒▒▓
124+
▓█▓ ▓▒█ █▓░ ░▒▓▓██▒ ░▓█▒ ▒▒▒░▒▒▓█████▒
125+
██░ ▓█▒█▒ ▒▓▓▒ ▓█ █░ ░░░░ ░█▒
126+
▓█ ▒█▓ ░ █░ ▒█ █▓
127+
█▓ ██ █░ ▓▓ ▒█▓▓▓▒█░
128+
█▓ ░▓██░ ▓▒ ▓█▓▒░░░▒▓█░ ▒█
129+
██ ▓█▓░ ▒ ░▒█▒██▒ ▓▓
130+
▓█▒ ▒█▓▒░ ▒▒ █▒█▓▒▒░░▒██
131+
░██▒ ▒▓▓▒ ▓██▓▒█▒ ░▓▓▓▓▒█▓
132+
░▓██▒ ▓░ ▒█▓█ ░░▒▒▒
133+
▒▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░▓▓ ▓░▒█░
134+
135+
______ _ _ _ _____ ____ _ _____ _ _ _ BETA
136+
| ____| (_) | | / ____|/ __ \| | / ____| (_) | |
137+
| |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_
138+
| __| | | | '_ \| |/ / \___ \| | | | | | | | | |/ _ \ '_ \| __|
139+
| | | | | | | | < ____) | |__| | |____ | |____| | | __/ | | | |_
140+
|_| |_|_|_| |_|_|\_\ |_____/ \___\_\______| \_____|_|_|\___|_| |_|\__|
141+
142+
Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to exit.
143+
144+
Command history file path: /home/user/.flink-sql-history
145+
```
146+
147+
1. Before querying any table with external source, prepare the related jars.
148+
Following examples query kafka table, mysql table in Flink SQL. Download the jar and put it in Flink cluster attached Azure Data Lake Storage gen2 storage.
149+
150+
Jars in Azure Data Lake Storage gen2 in Azure portal:
151+
152+
:::image type="image" source="./media/start-sql-client-cli-in-gateway-mode/jar-files-in-azure-portal.png" alt-text="Screenshot showing jar files in Azure portal." border="true" lightbox="./media/start-sql-client-cli-in-gateway-mode/jar-files-in-azure-portal.png":::
153+
154+
1. Use the table already created and put it into Hive metastore for management, then run the query.
155+
156+
``` SQL
157+
CREATE CATALOG myhive WITH (
158+
'type' = 'hive'
159+
);
160+
161+
USE CATALOG myhive;
162+
163+
// ADD jar into environment
164+
ADD JAR 'abfs://<container>@<storage name>.dfs.core.windows.net/jar/flink-connector-jdbc-3.1.0-1.17.jar';
165+
ADD JAR 'abfs://<container>@<storage name>.dfs.core.windows.net/jar/mysql-connector-j-8.0.33.jar';
166+
ADD JAR 'abfs://<container>@<storage name>.dfs.core.windows.net/jar/kafka-clients-3.2.0.jar';
167+
ADD JAR 'abfs://<container>@<storage name>.dfs.core.windows.net/jar/flink-connector-kafka-1.17.0.jar';
168+
169+
Flink SQL> show jars;
170+
----------------------------------------------------------------------------------------------+
171+
| jars |
172+
+----------------------------------------------------------------------------------------------+
173+
| abfs://<container>@<storage name>.dfs.core.windows.net/jar/flink-connector-kafka-1.17.0.jar |
174+
| abfs://<container>@<storage name>.dfs.core.windows.net/jar/flink-connector-jdbc-3.1.0-1.17.jar |
175+
| abfs://<container>@<storage name>.dfs.core.windows.net/jar/kafka-clients-3.2.0.jar |
176+
| abfs://<container>@<storage name>.dfs.core.windows.net/jar/mysql-connector-j-8.0.33.jar |
177+
+----------------------------------------------------------------------------------------------+
178+
4 rows in set
179+
180+
Flink SQL> SET 'sql-client.execution.result-mode' = 'tableau';
181+
[INFO] Execute statement succeed.
182+
183+
Flink SQL> show tables;
184+
+----------------------+
185+
| table name |
186+
+----------------------+
187+
| flightsintervaldata1 |
188+
| kafka_user_orders |
189+
| kafkatable |
190+
| mysql_user_orders |
191+
| orders |
192+
+----------------------+
193+
5 rows in set
194+
195+
// mysql cdc table
196+
Flink SQL> select * from mysql_user_orders;
197+
+----+-------------+----------------------------+-------------+--------------------------------+--------------+-------------+--------------+
198+
| op | order_id | order_date | customer_id | customer_name | price | product_id | order_status |
199+
+----+-------------+----------------------------+-------------+--------------------------------+--------------+-------------+--------------+
200+
| +I | 10001 | 2023-07-16 10:08:22.000000 | 1 | Jark | 50.00000 |
201+
102 | FALSE |
202+
| +I | 10002 | 2023-07-16 10:11:09.000000 | 2 | Sally | 15.00000 |
203+
105 | FALSE |
204+
| +I | 10003 | 2023-07-16 10:11:09.000000 | 3 | Sally | 25.00000 |
205+
206+
```
207+
208+
### Reference
209+
[Apache Flink® Command-Line Interface (CLI) on HDInsight on AKS clusters](./use-flink-cli-to-submit-jobs.md)

0 commit comments

Comments
 (0)