Merge pull request #211569 from Blackmist/v2-secrets

v2 version

Author: JamesJBarnett

articles/machine-learning/how-to-use-secrets-in-runs.md

@@ -1,76 +1,84 @@
 ---
-title: Authentication secrets in training
+title: Authentication secrets
 titleSuffix: Azure Machine Learning
-description: Learn how to pass secrets to training jobs in secure fashion using the Azure Key Vault for your workspace.
+description: Learn how to pass secrets to training jobs in secure fashion using Azure Key Vault.
 services: machine-learning
 author: rastala
 ms.author: roastala
 ms.reviewer: larryfr
 ms.service: machine-learning
 ms.subservice: enterprise-readiness
-ms.date: 10/21/2021
+ms.date: 09/16/2022
 ms.topic: how-to
-ms.custom: sdkv1, event-tier1-build-2022
+ms.custom: sdkv2
 ---
 
-# Use authentication credential secrets in Azure Machine Learning training jobs
+# Use authentication credential secrets in Azure Machine Learning jobs
 
-[!INCLUDE [sdk v1](../../includes/machine-learning-sdk-v1.md)]
+[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
+> [!div class="op_single_selector" title1="Select the version of the Azure Machine Learning Python SDK you are using:"]
+> * [v1](v1/how-to-use-secrets-in-runs.md)
+> * [v2 (current version)](how-to-use-secrets-in-runs.md)
 
-In this article, you learn how to use secrets in training jobs securely. Authentication information such as your user name and password are secrets. For example, if you connect to an external database in order to query training data, you would need to pass your username and password to the remote job context. Coding such values into training scripts in cleartext is insecure as it would expose the secret. 
+Authentication information such as your user name and password are secrets. For example, if you connect to an external database in order to query training data, you would need to pass your username and password to the remote job context. Coding such values into training scripts in clear text is insecure as it would potentially expose the secret.
 
-Instead, your Azure Machine Learning workspace has an associated resource called a [Azure Key Vault](../key-vault/general/overview.md). Use this Key Vault to pass secrets to remote jobs securely through a set of APIs in the Azure Machine Learning Python SDK.
+The Azure Key Vault allows you to securely store and retrieve secrets. In this article, learn how you can retrieve secrets stored in a key vault from a training job running on a compute cluster.
 
-The standard flow for using secrets is:
- 1. On local computer, log in to Azure and connect to your workspace.
- 2. On local computer, set a secret in Workspace Key Vault.
- 3. Submit a remote job.
- 4. Within the remote job, get the secret from Key Vault and use it.
+> [!IMPORTANT]
+> The Azure Machine Learning Python SDK v2 and Azure CLI extension v2 for machine learning do not provide the capability to set or get secrets. Instead, the information in this article uses the [Azure Key Vault Secrets client library for Python](/python/api/overview/azure/keyvault-secrets-readme).
 
-## Set secrets
+## Prerequisites
 
-In the Azure Machine Learning, the [Keyvault](/python/api/azureml-core/azureml.core.keyvault.keyvault) class contains methods for setting secrets. In your local Python session, first obtain a reference to your workspace Key Vault, and then use the [`set_secret()`](/python/api/azureml-core/azureml.core.keyvault.keyvault#set-secret-name--value-) method to set a secret by name and value. The __set_secret__ method updates the secret value if the name already exists.
+Before following the steps in this article, make sure you have the following prerequisites:
 
-```python
-from azureml.core import Workspace
-from azureml.core import Keyvault
-import os
+> [!TIP]
+> Many of the prerequisites in this section require __Contributor__, __Owner__, or equivalent access to your Azure subscription, or the Azure Resource Group that contains the resources. You may need to contact your Azure administrator and have them perform these actions.
 
+* An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning](https://azure.microsoft.com/free/).
+ 
+* An Azure Machine Learning workspace. If you don't have one, use the steps in the [Quickstart: Create workspace resources](quickstart-create-resources.md) article to create one.
 
-ws = Workspace.from_config()
-my_secret = os.environ.get("MY_SECRET")
-keyvault = ws.get_default_keyvault()
-keyvault.set_secret(name="mysecret", value = my_secret)
-```
+* An Azure Key Vault. If you used the [Quickstart: Create workspace resources](quickstart-create-resources.md) article to create your workspace, a key vault was created for you. You can also create a separate key vault instance using the information in the [Quickstart: Create a key vault](/azure/key-vault/general/quick-create-portal) article.
 
-Do not put the secret value in your Python code as it is insecure to store it in file as cleartext. Instead, obtain the secret value from an environment variable, for example Azure DevOps build secret, or from interactive user input.
+    > [!TIP]
+    > You do not have to use same key vault as the workspace.
 
-You can list secret names using the [`list_secrets()`](/python/api/azureml-core/azureml.core.keyvault.keyvault#list-secrets--) method and there is also a batch version,[set_secrets()](/python/api/azureml-core/azureml.core.keyvault.keyvault#set-secrets-secrets-batch-) that allows you to set multiple secrets at a time.
+* An Azure Machine Learning compute cluster configured to use a [managed identity](how-to-create-attach-compute-cluster.md?tabs=azure-studio#set-up-managed-identity). The cluster can be configured for either a system-assigned or user-assigned managed identity.
 
-> [!IMPORTANT]
-> Using `list_secrets()` will only list secrets created through `set_secret()` or `set_secrets()` using the Azure ML SDK. It will not list secrets created by something other than the SDK. For example, a secret created using the Azure portal or Azure PowerShell will not be listed.
-> 
-> You can use [`get_secret()`](#get-secrets) to get a secret value from the key vault, regardless of how it was created. So you can retrieve secrets that are not listed by `list_secrets()`.
+* Grant the managed identity for the compute cluster access to the secrets stored in key vault. The method used to grant access depends on how your key vault is configured:
+
+    * [Azure role-based access control (Azure RBAC)](/azure/key-vault/general/rbac-guide): When configured for Azure RBAC, add the managed identity to the __Key Vault Secrets User__ role on your key vault.
+    * [Azure Key Vault access policy](/azure/key-vault/general/assign-access-policy): When configured to use access policies, add a new policy that grants the __get__ operation for secrets and assign it to the managed identity.
+
+* A stored secret value in the key vault. This value can then be retrieved using a key. For more information, see [Quickstart: Set and retrieve a secret from Azure Key Vault](/azure/key-vault/secrets/quick-create-python).
+
+    > [!TIP]
+    > The quickstart link is to the steps for using the Azure Key Vault Python SDK. In the table of contents in the left navigation area are links to other ways to set a key.
+
+## Getting secrets
+
+1. Add the `azure-keyvault-secrets` and `azure-identity` packages to the [Azure Machine Learning environment](concept-environments.md) used when training the model. For example, by adding them to the conda file used to build the environment.
 
-## Get secrets
+    The environment is used to build the Docker image that the training job runs in on the compute cluster.
 
-In your local code, you can use the [`get_secret()`](/python/api/azureml-core/azureml.core.keyvault.keyvault#get-secret-name-) method to get the secret value by name.
+1. From your training code, use the [Azure Identity SDK](/python/api/overview/azure/identity-readme) and [Key Vault client library](/python/api/overview/azure/keyvault-secrets-readme) to get the managed identity credentials and authenticate to key vault:
 
-For jobs submitted the [`Experiment.submit`](/python/api/azureml-core/azureml.core.experiment.experiment#submit-config--tags-none----kwargs-)  , use the [`get_secret()`](/python/api/azureml-core/azureml.core.run.run#get-secret-name-) method with the [`Run`](/python/api/azureml-core/azureml.core.run%28class%29) class. Because a submitted run is aware of its workspace, this method shortcuts the Workspace instantiation and returns the secret value directly.
+    ```python
+    from azure.identity import DefaultAzureCredential
+    from azure.keyvault.secret import SecretClient
 
-```python
-# Code in submitted job
-from azureml.core import Experiment, Run
+    credential = DefaultAzureCredential()
 
-run = Run.get_context()
-secret_value = run.get_secret(name="mysecret")
-```
+    secret_client = SecretClient(vault_url="https://my-key-vault.vault.azure.net/", credential=credential)
+    ```
 
-Be careful not to expose the secret value by writing or printing it out.
+1. After authenticating, use the Key Vault client library to retrieve a secret by providing the associated key:
 
-There is also a batch version, [get_secrets()](/python/api/azureml-core/azureml.core.run.run#get-secrets-secrets-) for accessing multiple secrets at once.
+    ```python
+    secret = secret_client.get_secret("secret-name")
+    print(secret.value)
+    ```
 
 ## Next steps
 
- * [View example notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/manage-azureml-service/authentication-in-azureml/authentication-in-azureml.ipynb)
- * [Learn about enterprise security with Azure Machine Learning](concept-enterprise-security.md)
+For an example of submitting a training job using the Azure Machine Learning Python SDK v2 (preview), see [Train models with the Python SDK v2](how-to-train-sdk.md).

articles/machine-learning/toc.yml

@@ -405,7 +405,7 @@
           - name: Train with custom Docker image
             href: how-to-train-with-custom-image.md
           - name: Use Key Vault when training
-            displayName: secrets keyvault
+            displayName: secrets keyvault job
             href: how-to-use-secrets-in-runs.md
         - name: Train with the CLI v2
           href: how-to-train-cli.md

articles/machine-learning/v1/how-to-use-secrets-in-runs.md

@@ -0,0 +1,79 @@
+---
+title: Authentication secrets in training
+titleSuffix: Azure Machine Learning
+description: Learn how to pass secrets to training jobs in secure fashion using the Azure Key Vault for your workspace.
+services: machine-learning
+author: rastala
+ms.author: roastala
+ms.reviewer: larryfr
+ms.service: machine-learning
+ms.subservice: enterprise-readiness
+ms.date: 10/21/2021
+ms.topic: how-to
+ms.custom: sdkv1, event-tier1-build-2022
+---
+
+# Use authentication credential secrets in Azure Machine Learning training jobs
+
+[!INCLUDE [sdk v1](../../../includes/machine-learning-sdk-v1.md)]
+> [!div class="op_single_selector" title1="Select the version of the Azure Machine Learning Python SDK you are using:"]
+> * [v1](how-to-use-secrets-in-runs.md)
+> * [v2 (current version)](../how-to-use-secrets-in-runs.md)
+
+In this article, you learn how to use secrets in training jobs securely. Authentication information such as your user name and password are secrets. For example, if you connect to an external database in order to query training data, you would need to pass your username and password to the remote job context. Coding such values into training scripts in cleartext is insecure as it would expose the secret. 
+
+Instead, your Azure Machine Learning workspace has an associated resource called a [Azure Key Vault](/azure/key-vault/general/overview). Use this Key Vault to pass secrets to remote jobs securely through a set of APIs in the Azure Machine Learning Python SDK.
+
+The standard flow for using secrets is:
+ 1. On local computer, log in to Azure and connect to your workspace.
+ 2. On local computer, set a secret in Workspace Key Vault.
+ 3. Submit a remote job.
+ 4. Within the remote job, get the secret from Key Vault and use it.
+
+## Set secrets
+
+In the Azure Machine Learning, the [Keyvault](/python/api/azureml-core/azureml.core.keyvault.keyvault) class contains methods for setting secrets. In your local Python session, first obtain a reference to your workspace Key Vault, and then use the [`set_secret()`](/python/api/azureml-core/azureml.core.keyvault.keyvault#set-secret-name--value-) method to set a secret by name and value. The __set_secret__ method updates the secret value if the name already exists.
+
+```python
+from azureml.core import Workspace
+from azureml.core import Keyvault
+import os
+
+
+ws = Workspace.from_config()
+my_secret = os.environ.get("MY_SECRET")
+keyvault = ws.get_default_keyvault()
+keyvault.set_secret(name="mysecret", value = my_secret)
+```
+
+Do not put the secret value in your Python code as it is insecure to store it in file as cleartext. Instead, obtain the secret value from an environment variable, for example Azure DevOps build secret, or from interactive user input.
+
+You can list secret names using the [`list_secrets()`](/python/api/azureml-core/azureml.core.keyvault.keyvault#list-secrets--) method and there is also a batch version,[set_secrets()](/python/api/azureml-core/azureml.core.keyvault.keyvault#set-secrets-secrets-batch-) that allows you to set multiple secrets at a time.
+
+> [!IMPORTANT]
+> Using `list_secrets()` will only list secrets created through `set_secret()` or `set_secrets()` using the Azure ML SDK. It will not list secrets created by something other than the SDK. For example, a secret created using the Azure portal or Azure PowerShell will not be listed.
+> 
+> You can use [`get_secret()`](#get-secrets) to get a secret value from the key vault, regardless of how it was created. So you can retrieve secrets that are not listed by `list_secrets()`.
+
+## Get secrets
+
+In your local code, you can use the [`get_secret()`](/python/api/azureml-core/azureml.core.keyvault.keyvault#get-secret-name-) method to get the secret value by name.
+
+For jobs submitted the [`Experiment.submit`](/python/api/azureml-core/azureml.core.experiment.experiment#submit-config--tags-none----kwargs-)  , use the [`get_secret()`](/python/api/azureml-core/azureml.core.run.run#get-secret-name-) method with the [`Run`](/python/api/azureml-core/azureml.core.run%28class%29) class. Because a submitted run is aware of its workspace, this method shortcuts the Workspace instantiation and returns the secret value directly.
+
+```python
+# Code in submitted job
+from azureml.core import Experiment, Run
+
+run = Run.get_context()
+secret_value = run.get_secret(name="mysecret")
+```
+
+Be careful not to expose the secret value by writing or printing it out.
+
+There is also a batch version, [get_secrets()](/python/api/azureml-core/azureml.core.run.run#get-secrets-secrets-) for accessing multiple secrets at once.
+
+## Next steps
+
+ * [View example notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/manage-azureml-service/authentication-in-azureml/authentication-in-azureml.ipynb)
+ * [Learn about enterprise security with Azure Machine Learning](../concept-enterprise-security.md)

articles/machine-learning/v1/toc.yml

@@ -190,6 +190,9 @@
         href: how-to-train-pytorch.md
       - name: Migrate from Estimators to ScriptRunConfig
         href: how-to-migrate-from-estimators-to-scriptrunconfig.md
+      - name: Use Key Vault when training
+        displayName: secrets keyvault
+        href: how-to-use-secrets-in-runs.md
     - name: Automated machine learning
       displayName: automl, auto ml
       items: