|
| 1 | +--- |
| 2 | +title: Use connections |
| 3 | +titleSuffix: Azure Machine Learning |
| 4 | +description: Learn how to use connections to connect to External data sources for training with Azure Machine Learning. |
| 5 | +services: machine-learning |
| 6 | +ms.service: machine-learning |
| 7 | +ms.subservice: mldata |
| 8 | +ms.topic: how-to |
| 9 | +ms.author: ambadal |
| 10 | +author: AmarBadal |
| 11 | +ms.reviewer: franksolomon |
| 12 | +ms.date: 04/11/2023 |
| 13 | +ms.custom: data4ml |
| 14 | + |
| 15 | +# Customer intent: As an experienced data scientist with Python skills, I have data located in external sources outside of Azure. I need to make that data available to the Azure Machine Learning platform, to train my machine learning models. |
| 16 | +--- |
| 17 | + |
| 18 | +# Create connections |
| 19 | + |
| 20 | +[!INCLUDE [dev v2](../../includes/machine-learning-dev-v2.md)] |
| 21 | + |
| 22 | +In this article, learn how to connect to data sources located outside of Azure, to make that data available to Azure Machine Learning services. For this data availability, Azure supports connections to these external sources: |
| 23 | +- Snowflake DB |
| 24 | +- Amazon S3 |
| 25 | +- Azure SQL DB |
| 26 | + |
| 27 | +## Prerequisites |
| 28 | + |
| 29 | +- An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning](https://azure.microsoft.com/free/). |
| 30 | + |
| 31 | +- The [Azure Machine Learning SDK for Python](https://aka.ms/sdk-v2-install). |
| 32 | + |
| 33 | +- An Azure Machine Learning workspace. |
| 34 | + |
| 35 | +> [!NOTE] |
| 36 | +> An Azure Machine Learning connection securely stores the credentials passed during connection creation in the Workspace Azure Key Vault. A connection references the credentials from the key vault storage location for further use. You won't need to directly deal with the credentials after they are stored in the key vault. You have the option to store the credentials in the YAML file. A CLI command or SDK can override them. We recommend that you **avoid** credential storage in a YAML file, because a security breach could lead to a credential leak. |
| 37 | +
|
| 38 | +## Create a Snowflake DB connection |
| 39 | + |
| 40 | +# [CLI: Username/password](#tab/cli-username-password) |
| 41 | +This YAML file creates a Snowflake DB connection. Be sure to update the appropriate values: |
| 42 | + |
| 43 | +```yaml |
| 44 | +# my_snowflakedb_connection.yaml |
| 45 | +$schema: http://azureml/sdk-2-0/Connection.json |
| 46 | +type: snowflake |
| 47 | +name: my_snowflakedb_connection # add your datastore name here |
| 48 | + |
| 49 | +target: jdbc:snowflake://<myaccount>.snowflakecomputing.com/?db=<mydb>&warehouse=<mywarehouse>&role=<myrole> |
| 50 | +# add the Snowflake account, database, warehouse name and role name here. If no role name provided it will default to PUBLIC |
| 51 | +credentials: |
| 52 | + type: username_password |
| 53 | + username: <username> # add the Snowflake database user name here or leave this blank and type in CLI command line |
| 54 | + password: <password> # add the Snowflake database password here or leave this blank and type in CLI command line |
| 55 | +``` |
| 56 | +
|
| 57 | +Create the Azure Machine Learning connection in the CLI: |
| 58 | +
|
| 59 | +### Option 1: Use the username and password in YAML file |
| 60 | +
|
| 61 | +```azurecli |
| 62 | +az ml connection create --file my_snowflakedb_connection.yaml |
| 63 | +``` |
| 64 | + |
| 65 | +### Option 2: Override the username and password at the command line |
| 66 | + |
| 67 | +```azurecli |
| 68 | +az ml connection create --file my_snowflakedb_connection.yaml --set credentials.username="XXXXX" credentials.password="XXXXX" |
| 69 | +``` |
| 70 | + |
| 71 | +# [Python SDK: username/ password](#tab/sdk-username-password) |
| 72 | + |
| 73 | +### Option 1: Load connection from YAML file |
| 74 | + |
| 75 | +```python |
| 76 | +from azure.ai.ml import MLClient, load_workspace_connection |
| 77 | + |
| 78 | +ml_client = MLClient.from_config() |
| 79 | + |
| 80 | +wps_connection = load_workspace_connection(source="./my_snowflakedb_connection.yaml") |
| 81 | +wps_connection.credentials.username="XXXXX" |
| 82 | +wps_connection.credentials.password="XXXXXXXX" |
| 83 | +ml_client.connections.create_or_update(workspace_connection=wps_connection) |
| 84 | + |
| 85 | +``` |
| 86 | + |
| 87 | +### Option 2: Use WorkspaceConnection() in a Python script |
| 88 | + |
| 89 | +```python |
| 90 | +from azure.ai.ml import MLClient |
| 91 | +from azure.ai.ml.entities import WorkspaceConnection |
| 92 | +from azure.ai.ml.entities import UsernamePasswordConfiguration |
| 93 | + |
| 94 | +target= "jdbc:snowflake://<myaccount>.snowflakecomputing.com/?db=<mydb>&warehouse=<mywarehouse>&role=<myrole>" |
| 95 | +# add the Snowflake account, database, warehouse name and role name here. If no role name provided it will default to PUBLIC |
| 96 | + |
| 97 | +wps_connection = WorkspaceConnection(type="snowflake", |
| 98 | +target= target, |
| 99 | +credentials= UsernamePasswordConfiguration(username="XXXXX", password="XXXXXX") |
| 100 | +) |
| 101 | + |
| 102 | +ml_client.connections.create_or_update(workspace_connection=wps_connection) |
| 103 | + |
| 104 | +``` |
| 105 | + |
| 106 | +--- |
| 107 | + |
| 108 | +## Create an Azure SQL DB connection |
| 109 | + |
| 110 | +# [CLI: Username/password](#tab/cli-sql-username-password) |
| 111 | + |
| 112 | +This YAML script creates an Azure SQL DB connection. Be sure to update the appropriate values: |
| 113 | + |
| 114 | +```yaml |
| 115 | +# my_sqldb_connection.yaml |
| 116 | +$schema: http://azureml/sdk-2-0/Connection.json |
| 117 | + |
| 118 | +type: azuresqldb |
| 119 | +name: my_sqldb_connection |
| 120 | + |
| 121 | +target: Server=tcp:<myservername>,<port>;Database=<mydatabase>;Trusted_Connection=False;Encrypt=True;Connection Timeout=30 |
| 122 | +# add the sql servername, port addresss and database |
| 123 | +credentials: |
| 124 | + type: sql_auth |
| 125 | + username: <username> # add the sql database user name here or leave this blank and type in CLI command line |
| 126 | + password: <password> # add the sql database password here or leave this blank and type in CLI command line |
| 127 | +``` |
| 128 | +
|
| 129 | +Create the Azure Machine Learning connection in the CLI: |
| 130 | +
|
| 131 | +### Option 1: Use the username / password from YAML file |
| 132 | +
|
| 133 | +```azurecli |
| 134 | +az ml connection create --file my_sqldb_connection.yaml |
| 135 | +``` |
| 136 | + |
| 137 | +### Option 2: Override the username and password in YAML file |
| 138 | + |
| 139 | +```azurecli |
| 140 | +az ml connection create --file my_sqldb_connection.yaml --set credentials.username="XXXXX" credentials.password="XXXXX" |
| 141 | +``` |
| 142 | + |
| 143 | +# [Python SDK: username/ password](#tab/sdk-sql-username-password) |
| 144 | + |
| 145 | +### Option 1: Load connection from YAML file |
| 146 | + |
| 147 | +```python |
| 148 | +from azure.ai.ml import MLClient, load_workspace_connection |
| 149 | + |
| 150 | +ml_client = MLClient.from_config() |
| 151 | + |
| 152 | +wps_connection = load_workspace_connection(source="./my_sqldb_connection.yaml") |
| 153 | +wps_connection.credentials.username="XXXXXX" |
| 154 | +wps_connection.credentials.password="XXXXXxXXX" |
| 155 | +ml_client.connections.create_or_update(workspace_connection=wps_connection) |
| 156 | + |
| 157 | +``` |
| 158 | + |
| 159 | +### Option 2: Using WorkspaceConnection() |
| 160 | + |
| 161 | +```python |
| 162 | +from azure.ai.ml import MLClient |
| 163 | +from azure.ai.ml.entities import WorkspaceConnection |
| 164 | +from azure.ai.ml.entities import UsernamePasswordConfiguration |
| 165 | + |
| 166 | +target= "Server=tcp:<myservername>,<port>;Database=<mydatabase>;Trusted_Connection=False;Encrypt=True;Connection Timeout=30" |
| 167 | +# add the sql servername, port addresss and database |
| 168 | + |
| 169 | +wps_connection = WorkspaceConnection(type="azure_sql_db", |
| 170 | +target= target, |
| 171 | +credentials= UsernamePasswordConfiguration(username="XXXXX", password="XXXXXX") |
| 172 | +) |
| 173 | + |
| 174 | +ml_client.connections.create_or_update(workspace_connection=wps_connection) |
| 175 | + |
| 176 | +``` |
| 177 | + |
| 178 | +--- |
| 179 | + |
| 180 | +## Create Amazon S3 connection |
| 181 | + |
| 182 | +# [CLI: Access key](#tab/cli-s3-access-key) |
| 183 | + |
| 184 | +Create an Amazon S3 connection with the following YAML file. Be sure to update the appropriate values: |
| 185 | + |
| 186 | +```yaml |
| 187 | +# my_s3_connection.yaml |
| 188 | +$schema: http://azureml/sdk-2-0/Connection.json |
| 189 | + |
| 190 | +type: s3 |
| 191 | +name: my_s3_connection |
| 192 | + |
| 193 | +target: https://<mybucket>.amazonaws.com # add the s3 bucket details |
| 194 | +credentials: |
| 195 | + type: access_key |
| 196 | + access_key_id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # add access key id |
| 197 | + secret_access_key: XxXxXxXXXXXXXxXxXxxXxxXXXXXXXXxXxxXXxXXXXXXXxxxXxXXxXXXXXxXXxXXXxXxXxxxXXxXXxXXXXXxXxxXX # add access key secret |
| 198 | +``` |
| 199 | +
|
| 200 | +Create the Azure Machine Learning connection in the CLI: |
| 201 | +
|
| 202 | +```azurecli |
| 203 | +az ml connection create --file my_s3_connection.yaml |
| 204 | +``` |
| 205 | + |
| 206 | +# [Python SDK: Access key](#tab/sdk-s3-access-key) |
| 207 | + |
| 208 | +### Option 1: Load connection from YAML file |
| 209 | + |
| 210 | +```python |
| 211 | +from azure.ai.ml import MLClient, load_workspace_connection |
| 212 | + |
| 213 | +ml_client = MLClient.from_config() |
| 214 | + |
| 215 | + |
| 216 | +wps_connection = load_workspace_connection(source="./my_s3_connection.yaml") |
| 217 | +ml_client.connections.create_or_update(workspace_connection=wps_connection) |
| 218 | + |
| 219 | +``` |
| 220 | + |
| 221 | +### Option 2: Use WorkspaceConnection() in a Python script |
| 222 | + |
| 223 | +```python |
| 224 | +from azure.ai.ml import MLClient |
| 225 | +from azure.ai.ml.entities import WorkspaceConnection |
| 226 | +from azure.ai.ml.entities import AccessKeyConfiguration |
| 227 | + |
| 228 | +target = "https://<mybucket>.amazonaws.com" # add the s3 bucket details |
| 229 | +wps_connection = WorkspaceConnection(type="s3", |
| 230 | +target= target, |
| 231 | +credentials= AccessKeyConfiguration(access_key_id="XXXXXX",acsecret_access_key="XXXXXXXX") |
| 232 | +) |
| 233 | + |
| 234 | +ml_client.connections.create_or_update(workspace_connection=wps_connection) |
| 235 | + |
| 236 | +``` |
| 237 | +--- |
| 238 | + |
| 239 | +## Next steps |
| 240 | + |
| 241 | +- [Import data assets](how-to-import-data-assets.md#import-data-assets) |
0 commit comments