Skip to content

Commit 93b52db

Browse files
ttorblettorble
authored
Merge pull request #263853 from olegsavin/users/olegsa/trino11updates
Trino v1.1: remove phoenix connector, HMS config simplified
2 parents 8223771 + dc4db2d commit 93b52db

File tree

3 files changed

+142
-28
lines changed

3 files changed

+142
-28
lines changed

‎articles/hdinsight-aks/hdinsight-on-aks-autoscale-clusters.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ The following table describes the cluster types that are compatible with the Aut
4040
|Workload |Load Based |Schedule Based|
4141
|-|-|-|
4242
|Flink |Planned |Yes|
43-
|Trino |Planned |Yes**|
43+
|Trino |Yes** |Yes**|
4444
|Spark |Yes** |Yes**|
4545

4646
**Graceful decommissioning is configurable.

‎articles/hdinsight-aks/trino/trino-connect-to-metastore.md

Lines changed: 141 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -35,26 +35,62 @@ The following example covers the addition of Hive catalog and metastore database
3535

3636
**There are few important sections you need to add to your cluster ARM template to configure the Hive catalog and Hive metastore database:**
3737

38-
- `secretsProfile` – It specifies Azure Key Vault and list of secrets to be used in Trino cluster, required to connect to external Hive metastore.
39-
- `serviceConfigsProfiles` - It includes configuration for Trino catalogs. For more information, see [Add catalogs to existing cluster](trino-add-catalogs.md).
40-
- `trinoProfile.catalogOptions.hive` – List of Hive or iceberg or delta catalogs with parameters of external Hive metastore database for each catalog. To use external metastore database, catalog must be present in this list.
41-
38+
### Metastore configuration
39+
Configure external Hive metastore database in `config.properties` file:
40+
```json
41+
{
42+
"fileName": "config.properties",
43+
"values": {
44+
"hive.metastore.hdi.metastoreDbConnectionURL": "jdbc:sqlserver://mysqlserver1.database.windows.net;database=myhmsdb1;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=30",
45+
"hive.metastore.hdi.metastoreDbConnectionUserName": "trinoadmin",
46+
"hive.metastore.hdi.metastoreDbConnectionPasswordSecret": "hms-db-pwd",
47+
"hive.metastore.hdi.metastoreWarehouseDir": "abfs://[email protected]/hive/warehouse"
48+
}
49+
}
50+
```
51+
| Property| Description| Example|
52+
|---|---|---|
53+
|hive.metastore.hdi.metastoreDbConnectionURL|JDBC connection string to database.|jdbc:sqlserver://mysqlserver1.database.windows.net;database=myhmsdb1;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=30|
54+
|hive.metastore.hdi.metastoreDbConnectionUserName|SQL user name to connect to database.|trinoadmin|
55+
|hive.metastore.hdi.metastoreDbConnectionPasswordSecret|Secret referenceName configured in secretsProfile with password.|hms-db-pwd|
56+
|hive.metastore.hdi.metastoreWarehouseDir|ABFS URI to location in storage where data is stored.|`abfs://[email protected]/hive/warehouse`|
4257

58+
### Metastore authentication
59+
Configure authentication to external Hive metastore database specifying Azure Key Vault secrets.
60+
> [!NOTE]
61+
> `referenceName` should match value provided in `hive.metastore.hdi.metastoreDbConnectionPasswordSecret`
62+
```json
63+
"secretsProfile": {
64+
"keyVaultResourceId": "/subscriptions/{USER_SUBSCRIPTION_ID}/resourceGroups/{USER_RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/{USER_KEYVAULT_NAME}",
65+
"secrets": [
66+
{
67+
"referenceName": "hms-db-pwd",
68+
"type": "secret",
69+
"keyVaultObjectName": "hms-db-pwd"
70+
} ]
71+
},
72+
```
4373
| Property| Description| Example|
4474
|---|---|---|
4575
|secretsProfile.keyVaultResourceId|Azure resource ID string to Azure Key Vault where secrets for Hive metastore are stored.|/subscriptions/0000000-0000-0000-0000-000000000000/resourceGroups/trino-rg/providers/Microsoft.KeyVault/vaults/trinoakv|
4676
|secretsProfile.secrets[*].referenceName|Unique reference name of the secret to use later in clusterProfile.|Secret1_ref|
4777
|secretsProfile.secrets[*].type|Type of object in Azure Key Vault, only “secret” is supported.|secret|
4878
|secretsProfile.secrets[*].keyVaultObjectName|Name of secret object in Azure Key Vault containing actual secret value.|secret1|
49-
|trinoProfile.catalogOptions.hive|List of Hive or iceberg or delta catalogs with parameters of external Hive metastore database, require parameters for each. To use external metastore database, catalog must be present in this list.
50-
|trinoProfile.catalogOptions.hive[*].catalogName|Name of Trino catalog configured in `serviceConfigsProfiles`, which configured to use external Hive metastore database.|hive1|
51-
|trinoProfile.catalogOptions.hive[*].metastoreDbConnectionURL|JDBC connection string to database.|jdbc:sqlserver://mysqlserver1.database.windows.net;database=myhmsdb1;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=30|
52-
|trinoProfile.catalogOptions.hive[*].metastoreDbConnectionUserName|SQL user name to connect to database.|trinoadmin|
53-
|trinoProfile.catalogOptions.hive[*].metastoreDbConnectionPasswordSecret|Secret referenceName configured in secretsProfile with password.|hms-db-pwd|
54-
|trinoProfile.catalogOptions.hive[*].metastoreWarehouseDir|ABFS URI to location in storage where data is stored.|`abfs://[email protected]/hive/warehouse`|
5579

56-
To configure external Hive metastore to an existing Trino cluster, add the required sections in your cluster ARM template by referring to the following example:
80+
### Catalog configuration
81+
In order for a Trino catalog to use external Hive metastore it should specify `hive.metastore=hdi` property. For more information, see [Add catalogs to existing cluster](trino-add-catalogs.md):
82+
```
83+
{
84+
"fileName": "hive1.properties",
85+
"values": {
86+
"connector.name": "hive",
87+
"hive.metastore": "hdi"
88+
}
89+
}
90+
```
5791

92+
### Complete example
93+
To configure external Hive metastore to an existing Trino cluster, add the required sections in your cluster ARM template by referring to the following example:
5894

5995
```json
6096
{
@@ -85,33 +121,35 @@ To configure external Hive metastore to an existing Trino cluster, add the requi
85121
{
86122
"serviceName": "trino",
87123
"configs": [
124+
{
125+
"component": "common",
126+
"files": [
127+
{
128+
"fileName": "config.properties",
129+
"values": {
130+
"hive.metastore.hdi.metastoreDbConnectionURL": "jdbc:sqlserver://mysqlserver1.database.windows.net;database=myhmsdb1;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=30",
131+
"hive.metastore.hdi.metastoreDbConnectionUserName": "trinoadmin",
132+
"hive.metastore.hdi.metastoreDbConnectionPasswordSecret": "hms-db-pwd",
133+
"hive.metastore.hdi.metastoreWarehouseDir": "abfs://[email protected]/hive/warehouse"
134+
}
135+
}
136+
]
137+
},
88138
{
89139
"component": "catalogs",
90140
"files": [
91141
{
92142
"fileName": "hive1.properties",
93143
"values": {
94-
"connector.name": "hive"
144+
"connector.name": "hive",
145+
"hive.metastore": "hdi"
95146
}
96147
}
97148
]
98149
}
99150
]
100151
}
101-
],
102-
"trinoProfile": {
103-
"catalogOptions": {
104-
"hive": [
105-
{
106-
"catalogName": "hive1",
107-
"metastoreDbConnectionURL": "jdbc:sqlserver://mysqlserver1.database.windows.net;database=myhmsdb1;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=30",
108-
"metastoreDbConnectionUserName": "trinoadmin",
109-
"metastoreDbConnectionPasswordSecret": "hms-db-pwd",
110-
"metastoreWarehouseDir": "abfs://[email protected]/hive/warehouse"
111-
}
112-
]
113-
}
114-
}
152+
]
115153
}
116154
}
117155
}
@@ -135,3 +173,80 @@ create schema hive1.schema1;
135173
create table hive1.schema1.tpchorders as select * from tpch.tiny.orders;
136174
select * from hive1.schema1.tpchorders limit 100;
137175
```
176+
177+
## Alternative configuration
178+
Alternatively external Hive metastore database parameters can be specified in `trinoProfile.catalogOptions.hive` together with `hive.metastore=hdi` catalog property:
179+
180+
| Property| Description| Example|
181+
|---|---|---|
182+
|trinoProfile.catalogOptions.hive|List of Hive or iceberg or delta catalogs with parameters of external Hive metastore database, require parameters for each. To use external metastore database, catalog must be present in this list.
183+
|trinoProfile.catalogOptions.hive[*].catalogName|Name of Trino catalog configured in `serviceConfigsProfiles`, which configured to use external Hive metastore database.|hive1|
184+
|trinoProfile.catalogOptions.hive[*].metastoreDbConnectionURL|JDBC connection string to database.|jdbc:sqlserver://mysqlserver1.database.windows.net;database=myhmsdb1;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=30|
185+
|trinoProfile.catalogOptions.hive[*].metastoreDbConnectionUserName|SQL user name to connect to database.|trinoadmin|
186+
|trinoProfile.catalogOptions.hive[*].metastoreDbConnectionPasswordSecret|Secret referenceName configured in secretsProfile with password.|hms-db-pwd|
187+
|trinoProfile.catalogOptions.hive[*].metastoreWarehouseDir|ABFS URI to location in storage where data is stored.|`abfs://[email protected]/hive/warehouse`|
188+
189+
### Complete example
190+
```json
191+
{
192+
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
193+
"contentVersion": "1.0.0.0",
194+
"parameters": {},
195+
"resources": [
196+
{
197+
"type": "microsoft.hdinsight/clusterpools/clusters",
198+
"apiVersion": "<api-version>",
199+
"name": "<cluster-pool-name>/<cluster-name>",
200+
"location": "<region, e.g. westeurope>",
201+
"tags": {},
202+
"properties": {
203+
"clusterType": "Trino",
204+
205+
"clusterProfile": {
206+
"secretsProfile": {
207+
"keyVaultResourceId": "/subscriptions/{USER_SUBSCRIPTION_ID}/resourceGroups/{USER_RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/{USER_KEYVAULT_NAME}",
208+
"secrets": [
209+
{
210+
"referenceName": "hms-db-pwd",
211+
"type": "secret",
212+
"keyVaultObjectName": "hms-db-pwd"
213+
} ]
214+
},
215+
"serviceConfigsProfiles": [
216+
{
217+
"serviceName": "trino",
218+
"configs": [
219+
{
220+
"component": "catalogs",
221+
"files": [
222+
{
223+
"fileName": "hive1.properties",
224+
"values": {
225+
"connector.name": "hive",
226+
"hive.metastore": "hdi"
227+
}
228+
}
229+
]
230+
}
231+
]
232+
}
233+
],
234+
"trinoProfile": {
235+
"catalogOptions": {
236+
"hive": [
237+
{
238+
"catalogName": "hive1",
239+
"metastoreDbConnectionURL": "jdbc:sqlserver://mysqlserver1.database.windows.net;database=myhmsdb1;encrypt=true;trustServerCertificate=true;create=false;loginTimeout=30",
240+
"metastoreDbConnectionUserName": "trinoadmin",
241+
"metastoreDbConnectionPasswordSecret": "hms-db-pwd",
242+
"metastoreWarehouseDir": "abfs://[email protected]/hive/warehouse"
243+
}
244+
]
245+
}
246+
}
247+
}
248+
}
249+
}
250+
]
251+
}
252+
```

‎articles/hdinsight-aks/trino/trino-connectors.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,6 @@ Trino in HDInsight on AKS enables seamless integration with data sources. You ca
3232
* [MongoDB](https://trino.io/docs/410/connector/mongodb.html)
3333
* [MySQL](https://trino.io/docs/410/connector/mysql.html)
3434
* [Oracle](https://trino.io/docs/410/connector/oracle.html)
35-
* [Phoenix](https://trino.io/docs/410/connector/phoenix.html)
3635
* [PostgreSQL](https://trino.io/docs/410/connector/postgresql.html)
3736
* [Prometheus](https://trino.io/docs/410/connector/prometheus.html)
3837
* [Redis](https://trino.io/docs/410/connector/redis.html)

0 commit comments

Comments
 (0)