Merge pull request #255130 from msakande/secret-injection-online-deployment

Updates for secret injection

Author: JamesJBarnett

articles/machine-learning/concept-endpoints-online-auth.md

@@ -24,10 +24,14 @@ A _user identity_ is a Microsoft Entra ID that you can use to create an endpoint
 
 An _endpoint identity_ is a Microsoft Entra ID that runs the user container in deployments. In other words, if the identity is associated with the endpoint and used for the user container for the deployment, then it's called an endpoint identity. The endpoint identity would also need proper permissions for the user container to interact with resources as needed. For example, the endpoint identity would need the proper permissions to pull images from the Azure Container Registry or to interact with other Azure services.
 
+In general, the user identity and endpoint identity would have separate permission requirements. For more information on managing identities and permissions, see [How to authenticate clients for online endpoints](how-to-authenticate-online-endpoint.md). For more information on the special case of automatically adding extra permission for secrets, see [Additional permissions for user identity](#additional-permissions-for-user-identity-when-enforcing-access-to-default-secret-stores).
+
+
 ## Limitation
 
 Microsoft Entra ID authentication (`aad_token`) is supported for managed online endpoints __only__. For Kubernetes online endpoints, you can use either a key or an Azure Machine Learning token (`aml_token`).
 
+
 ## Permissions needed for user identity
 
 When you sign in to your Azure tenant with your Microsoft account (for example, using `az login`), you complete the user authentication step (commonly known as _authn_) and your identity as a user is determined. Now, say you want to create an online endpoint under a workspace, you'll need the proper permission to do so. This is where authorization (commonly known as _authz_) comes in.
@@ -60,6 +64,22 @@ For control plane operations, your user identity needs to have a proper Azure ro
 > [!NOTE]
 > You can fetch your Microsoft Entra token (`aad_token`) directly from Microsoft Entra ID once you're signed in, and you don't need extra Azure RBAC permission on the workspace.
 
+#### Additional permissions for user identity when enforcing access to default secret stores
+
+If you intend to use the [secret injection](concept-secret-injection.md) feature, and while creating your endpoints, you set the flag to enforce access to the default secret stores, your _user identity_ needs to have the permission to read secrets from workspace connections.
+
+When the endpoint is created with a system-assigned identity (SAI) _and_ the flag is set to enforce access to the default secret stores, your user identity needs to have permissions to read secrets from workspace connections when creating the endpoint and creating the deployment(s) under the endpoint. This restriction ensures that only a _user identity_ with the permission to read secrets can grant the endpoint identity the permission to read secrets.
+
+- If a user identity doesn't have the permissions to read secrets from workspace connections, but it tries to create the _endpoint_ with an SAI and the endpoint's flag set to enforce access to the default secret stores, the endpoint creation is rejected.
+
+- Similarly, if a user identity doesn't have the permissions to read secrets from workspace connections, but tries to create a _deployment_ under the endpoint with an SAI and the endpoint's flag set to enforce access to the default secret stores, the deployment creation is rejected.
+
+When (1) the endpoint is created with a UAI, _or_ (2) the flag is _not_ set to enforce access to the default secret stores even if the endpoint uses an SAI, your user identity doesn't need to have permissions to read secrets from workspace connections. In this case, the endpoint identity won't be automatically granted the permission to read secrets, but you can still manually grant the endpoint identity this permission by assigning proper roles if needed. Regardless of whether the role assignment was done automatically or manually, the secret retrieval and injection will still be triggered if you mapped the environment variables with secret references in the deployment definition, and it will use the endpoint identity to do so.
+
+For more information on managing authorization to an Azure Machine Learning workspace, see [Manage access to Azure Machine Learning](how-to-assign-roles.md).
+
+For more information on secret injection, see [Secret injection in online endpoints](concept-secret-injection.md).
+
 
 ### Data plane operations
 
@@ -96,18 +116,20 @@ An online deployment runs your user container with the _endpoint identity_, that
 
 ### Automatic role assignment for endpoint identity
 
-Online endpoints require Azure Container Registry (ACR) pull permission on the ACR associated with the workspace. They also require Storage Blob Data Reader permission on the default datastore of the workspace. By default, these permissions are automatically granted to the endpoint identity if the endpoint identity is a system-assigned identity.
+If the endpoint identity is a system-assigned identity, some roles are assigned to the endpoint identity for convenience.
+
+Role | Description | Condition for the automatic role assignment
+-- | -- | --
+`AcrPull` | Allows the endpoint identity to pull images from the Azure Container Registry (ACR) associated with the workspace. | The endpoint identity is a system-assigned identity (SAI).
+`Storage Blob Data Reader` | Allows the endpoint identity to read blobs from the default datastore of the workspace. | The endpoint identity is a system-assigned identity (SAI).
+`AzureML Metrics Writer (preview)` | Allows the endpoint identity to write metrics to the workspace. | The endpoint identity is a system-assigned identity (SAI).
+`Azure Machine Learning Workspace Connection Secrets Reader` <sup>1</sup> | Allows the endpoint identity to read secrets from workspace connections. | The endpoint identity is a system-assigned identity (SAI). The endpoint is created with a flag to enforce access to the default secret stores. The _user identity_ that creates the endpoint has the same permission to read secrets from workspace connections. <sup>2</sup>
 
-Also, when creating an endpoint, if you set the flag to enforce access to the default secret stores, the endpoint identity is automatically granted the permission to read secrets from workspace connections.
+<sup>1</sup> For more information on the `Azure Machine Learning Workspace Connection Secrets Reader` role, see [Assign permissions to the identity](how-to-authenticate-online-endpoint.md#assign-permissions-to-the-identity).
 
-There's no automatic role assignment if the endpoint identity is a user-assigned identity.
+<sup>2</sup> Even if the endpoint identity is SAI, if the enforce flag is not set or the user identity doesn't have the permission, there's no automatic role assignment for this role. For more information, see [How to deploy online endpoint with secret injection](how-to-deploy-online-endpoint-with-secret-injection.md#create-an-endpoint).
 
-In more detail:
-- If you use a system-assigned identity (SAI) for the endpoint, roles with fundamental permissions (such as Azure Container Registry pull permission, and Storage Blob Data Reader) are automatically assigned to the endpoint identity. Also, you can set a flag on the endpoint to allow its SAI have the permission to read secrets from workspace connections. To have this permission, the `Azure Machine Learning Workspace Connection Secret Reader` role would be automatically assigned to the endpoint identity. For this role to be automatically assigned to the endpoint identity, the following conditions must be met:
-  - Your _user identity_, that is, the identity that creates the endpoint, has the permissions to read secrets from workspace connections when creating the endpoint.
-  - The endpoint uses an SAI.
-  - The endpoint is defined with a flag to enforce access to default secret stores (workspace connections under the current workspace) when creating the endpoint.
-- If your endpoint uses a UAI, or it uses the Key Vault as the secret store with an SAI. In these cases, you need to manually assign to the endpoint identity the role with the proper permissions to read secrets from the Key Vault.
+If the endpoint identity is a user-assigned identity, there's no automatic role assignment. In this case, you need to manually assign roles to the endpoint identity as needed.
 
 
 ## Choosing the permissions and scope for authorization

articles/machine-learning/concept-endpoints-online.md

@@ -11,9 +11,9 @@ ms.author: sehan
 ms.reviewer: mopeakande
 reviewer: msakande
 ms.custom: devplatv2
-ms.date: 09/13/2023
+ms.date: 10/24/2023
 
-#Customer intent: As an MLOps administrator, I want to understand what a managed endpoint is and why I need it.
+#Customer intent: As an ML pro, I want to understand what an online endpoint is and why I need it.
 ---
 
 # Online endpoints and deployments for real-time inference
@@ -245,6 +245,12 @@ Monitoring for Azure Machine Learning endpoints is possible via integration with
 
 For more information on monitoring, see [Monitor online endpoints](how-to-monitor-online-endpoints.md).
 
+### Secret injection in online deployments (preview)
+
+Secret injection in the context of an online deployment is a process of retrieving secrets (such as API keys) from secret stores, and injecting them into your user container that runs inside an online deployment. Secrets will eventually be accessible via environment variables, thereby providing a secure way for them to be consumed by the inference server that runs your scoring script or by the inferencing stack that you bring with a BYOC (bring your own container) deployment approach.
+
+There are two ways to inject secrets. You can inject secrets yourself, using managed identities, or you can use the secret injection feature. To learn more about the ways to inject secrets, see [Secret injection in online endpoints (preview)](concept-secret-injection.md).
+
 
 ## Next steps
 

articles/machine-learning/concept-secret-injection.md

@@ -0,0 +1,96 @@
+---
+title: What is secret injection in online endpoints (preview)?
+titleSuffix: Azure Machine Learning
+description: Learn about secret injection as it applies to online endpoints in Azure Machine Learning.
+services: machine-learning
+ms.service: machine-learning
+ms.subservice: inferencing
+ms.topic: concept-article
+author: dem108
+ms.author: sehan
+ms.reviewer: mopeakande
+reviewer: msakande
+ms.custom: ignite-2023
+ms.date: 01/10/2024
+
+#CustomerIntent: As an ML Pro, I want to retrieve and inject secrets into the deployment environment easily so that deployments I create can consume the secrets in a secured manner.
+---
+
+# Secret injection in online endpoints (preview)
+
+[!INCLUDE [dev v2](includes/machine-learning-dev-v2.md)]
+
+Secret injection in the context of an online endpoint is a process of retrieving secrets (such as API keys) from secret stores, and injecting them into your user container that runs inside an online deployment. Secrets are eventually accessed securely via environment variables, which are used by the inference server that runs your scoring script or by the inferencing stack that you bring with a BYOC (bring your own container) deployment approach.
+
+[!INCLUDE [machine-learning-preview-generic-disclaimer](includes/machine-learning-preview-generic-disclaimer.md)]
+
+## Problem statement
+
+When you create an online deployment, you might want to use secrets from within the deployment to access external services. Some of these external services include Microsoft Azure OpenAI service, Azure AI Services, and Azure AI Content Safety.
+
+To use the secrets, you have to find a way to securely pass them to your user container that runs inside the deployment. We don't recommend that you include secrets as part of the deployment definition, since this practice would expose the secrets in the deployment definition. 
+
+A better approach is to store the secrets in secret stores and then retrieve them securely from within the deployment. However, this approach poses its own challenge: how the deployment should authenticate itself to the secret stores to retrieve secrets. Because the online deployment runs your user container using the _endpoint identity_, which is a [managed identity](/entra/identity/managed-identities-azure-resources/overview), you can use [Azure RBAC](../role-based-access-control/overview.md) to control the endpoint identity's permissions and allow the endpoint to retrieve secrets from the secret stores.
+Using this approach requires you to do the following tasks:
+
+- Assign the right roles to the endpoint identity so that it can read secrets from the secret stores.
+- Implement the scoring logic for the deployment so that it uses the endpoint's managed identity to retrieve the secrets from the secret stores.
+
+While this approach of using a managed identity is a secure way to retrieve and inject secrets, [secret injection via the secret injection feature](#secret-injection-via-the-secret-injection-feature) further simplifies the process of retrieving secrets for [workspace connections](prompt-flow/concept-connections.md) and [key vaults](../key-vault/general/overview.md).
+
+
+## Managed identity associated with the endpoint
+
+
+An online deployment runs your user container with the managed identity associated with the endpoint. This managed identity, called the _endpoint identity_, is a [Microsoft Entra ID](/entra/fundamentals/whatis) that supports [Azure RBAC](../role-based-access-control/overview.md). Therefore, you can assign Azure roles to the identity to control permissions that are required to perform operations. The endpoint identity can be either a system-assigned identity (SAI) or a user-assigned identity (UAI). You can decide which of these kinds of identities to use when you create the endpoint.
+
+- For a _system-assigned identity_, the identity is created automatically when you create the endpoint, and roles with fundamental permissions (such as the Azure Container Registry pull permission and the storage blob data reader) are automatically assigned.
+- For a _user-assigned identity_, you need to create the identity first, and then associate it with the endpoint when you create the endpoint. You're also responsible for assigning proper roles to the UAI as needed.
+
+For more information on using managed identities of an endpoint, see [How to access resources from endpoints with managed identities](how-to-access-resources-from-endpoints-managed-identities.md), and the example for [using managed identities to interact with external services](https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/online/managed/managed-identities).
+
+
+## Role assignment to the endpoint identity
+
+The following roles are required by the secret stores:
+
+- For __secrets stored in workspace connections under your workspace__: `Workspace Connections` provides a [List Secrets API (preview)](/rest/api/azureml/2023-08-01-preview/workspace-connections/list-secrets) that requires the identity that calls the API to have `Azure Machine Learning Workspace Connection Secrets Reader` role (or equivalent) assigned to the identity.
+- For __secrets stored in an external Microsoft Azure Key Vault__: Key Vault provides a [Get Secret Versions API](/rest/api/keyvault/secrets/get-secret-versions/get-secret-versions) that requires the identity that calls the API to have `Key Vault Secrets User` role (or equivalent) assigned to the identity.
+
+
+## Implementation of secret injection
+
+Once secrets (such as API keys) are retrieved from secret stores, there are two ways to inject them into a user container that runs inside the online deployment:
+
+- Inject secrets yourself, using managed identities.
+- Inject secrets, using the secret injection feature.
+
+Both of these approaches involve two steps:
+
+1. First, retrieve secrets from the secret stores, using the endpoint identity.
+1. Second, inject the secrets into your user container.
+
+### Secret injection via the use of managed identities
+
+In your deployment definition, you need to use the endpoint identity to call the APIs from secret stores. You can implement this logic either in your scoring script or in shell scripts that you run in your BYOC container. To implement secret injection via the use of managed identities, see the [example for using managed identities to interact with external services](https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/online/managed/managed-identities).
+
+### Secret injection via the secret injection feature
+
+To use the secret injection feature, in your deployment definition, map the secrets (that you want to refer to) from workspace connections or the Key Vault onto the environment variables. This approach doesn't require you to write any code in your scoring script or in shell scripts that you run in your BYOC container. To map the secrets from workspace connections or the Key Vault onto the environment variables, the following conditions must be met:
+
+- During endpoint creation, if an online endpoint was defined to enforce access to default secret stores (workspace connections under the current workspace), your user identity that creates the deployment under the endpoint should have the permissions to read secrets from workspace connections.
+- The endpoint identity that the deployment uses should have permissions to read secrets from either workspace connections or the Key Vault, as referenced in the deployment definition.
+
+> [!NOTE]
+> - If the endpoint was successfully created with an SAI and the flag set to enforce access to default secret stores, then the endpoint would automatically have the permission for workspace connections. 
+> - In the case where the endpoint used a UAI, or the flag to enforce access to default secret stores wasn't set, then the endpoint identity might not have the permission for workspace connections. In such a situation, you need to manually assign the role for the workspace connections to the endpoint identity.
+> - The endpoint identity won't automatically receive permission for the external Key Vault. If you're using the Key Vault as a secret store, you'll need to manually assign the role for the Key Vault to the endpoint identity.
+
+For more information on using secret injection, see [Deploy machine learning models to online endpoints with secret injection (preview)](how-to-deploy-online-endpoint-with-secret-injection.md).
+
+
+## Related content
+
+- [Deploy machine learning models to online endpoints with secret injection (preview)](how-to-deploy-online-endpoint-with-secret-injection.md)
+- [Authentication for managed online endpoints](concept-endpoints-online-auth.md)
+- [Online endpoints](concept-endpoints-online.md)