Skip to content

Commit f609758

Browse files
Merge pull request #275671 from dem108/patch-30
Explain deployment creation fail fast when workspace managed VNet is …
2 parents a81d769 + bf1b94f commit f609758

File tree

3 files changed

+24
-10
lines changed

3 files changed

+24
-10
lines changed

articles/machine-learning/how-to-managed-network.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -809,9 +809,11 @@ To enable the [serverless Spark jobs](how-to-submit-spark-jobs.md) for the manag
809809

810810
## Manually provision a managed VNet
811811

812-
The managed VNet is automatically provisioned when you create a compute resource. When you rely on automatic provisioning, it can take around __30 minutes__ to create the first compute resource as it is also provisioning the network. If you configured FQDN outbound rules (only available with allow only approved mode), the first FQDN rule adds around __10 minutes__ to the provisioning time. If you have a large set of outbound rules to be provisioned in the managed network, it can take longer for provisioning to complete. The increased provisioning time can cause your first compute creation, or your first managed online endpoint deployment, to time out.
812+
The managed VNet is automatically provisioned when you create a compute instance. When you rely on automatic provisioning, it can take around __30 minutes__ to create the first compute instance as it is also provisioning the network. If you configured FQDN outbound rules (only available with allow only approved mode), the first FQDN rule adds around __10 minutes__ to the provisioning time. If you have a large set of outbound rules to be provisioned in the managed network, it can take longer for provisioning to complete. The increased provisioning time can cause your first compute instance creation to time out.
813813

814-
To reduce the wait time and avoid potential timeout errors, we recommend manually provisioning the managed network. Then wait until the provisioning completes before you create a compute resource or managed online endpoint deployment.
814+
To reduce the wait time and avoid potential timeout errors, we recommend manually provisioning the managed network. Then wait until the provisioning completes before you create a compute instance.
815+
816+
To create an online deployment, you must manually provision the managed network, or create a compute instance first which will automatically provision it.
815817

816818
# [Azure CLI](#tab/azure-cli)
817819

articles/machine-learning/how-to-secure-online-endpoint.md

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ ms.custom: devx-track-azurecli, moe-wsvnet
1717

1818
[!INCLUDE [machine-learning-dev-v2](includes/machine-learning-dev-v2.md)]
1919

20-
In this article, you'll use network isolation to secure a managed online endpoint. You'll create a managed online endpoint that uses an Azure Machine Learning workspace's private endpoint for secure inbound communication. You'll also configure the workspace with a **managed virtual network** that **allows only approved outbound** communication for deployments. Finally, you'll create a deployment that uses the private endpoints of the workspace's managed virtual network for outbound communication.
20+
In this article, you'll use network isolation to secure a managed online endpoint. You'll create a managed online endpoint that uses an Azure Machine Learning workspace's private endpoint for secure **inbound** communication. You'll also configure the workspace with a **managed virtual network** that **allows only approved outbound** communication for deployments. Finally, you'll create a deployment that uses the private endpoints of the workspace's managed virtual network for outbound communication.
2121

2222
For examples that use the legacy method for network isolation, see the deployment files [deploy-moe-vnet-legacy.sh](https://github.com/Azure/azureml-examples/blob/main/cli/deploy-moe-vnet-legacy.sh) (for deployment using a generic model) and [deploy-moe-vnet-mlflow-legacy.sh](https://github.com/Azure/azureml-examples/blob/main/cli/deploy-moe-vnet-mlflow-legacy.sh) (for deployment using an MLflow model) in the azureml-examples GitHub repo.
2323

@@ -40,14 +40,18 @@ For examples that use the legacy method for network isolation, see the deploymen
4040
4141
* If you want to use a [user-assigned managed identity](../active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities.md?pivots=identity-mi-methods-azp) to create and manage online endpoints and online deployments, the identity should have the proper permissions. For details about the required permissions, see [Set up service authentication](./how-to-identity-based-service-authentication.md#workspace). For example, you need to assign the proper RBAC permission for Azure Key Vault on the identity.
4242
43-
#### Migrate from legacy network isolation method to managed virtual network
43+
### Migrate from legacy network isolation method to workspace managed virtual network
4444
45-
If you've used the [legacy method](concept-secure-online-endpoint.md#secure-outbound-access-with-legacy-network-isolation-method) previously for network isolation of managed online endpoints, and you want to migrate to using a workspace managed virtual network to secure your endpoints, follow these steps:
45+
If you've used the [legacy method](concept-secure-online-endpoint.md#secure-outbound-access-with-legacy-network-isolation-method) previously for network isolation of managed online endpoints, and you want to migrate to using a workspace managed virtual network to secure your endpoints, you can follow these steps:
4646
47-
1. Delete all computes in your workspace.
48-
1. Enable managed virtual network for your workspace. For more information on how to configure a managed network for your workspace, see [Workspace Managed Virtual Network Isolation](how-to-managed-network.md).
49-
1. Configure private endpoints for outbound communication to private resources that your managed online endpoints need to access. These private resources include a storage account, Azure Key Vault, and Azure Container Registry (ACR).
50-
1. (Optional) If you're integrating with a user registry, configure private endpoints for outbound communication to your registry, its storage account, and its ACR.
47+
1. Create a new workspace and enable managed virtual network. For more information on how to configure a managed network for your workspace, see [Workspace Managed Virtual Network Isolation](how-to-managed-network.md).
48+
1. (Optional) On the workspace network setting, add outbound rules with the type of private endpoints if your deployments need to access additional private resources, other than Storage account, Azure Key Vault, and Azure Container Registry (ACR) associated with the workspace (which are added by default).
49+
1. (Optional) If you intend to use Azure Machine Learning registries, configure private endpoints for outbound communication to your registry, its storage account, and its Azure Container Registry.
50+
1. Create online endpoints / deployments in the new workspace. You may leverage Azure Machine Learning registries to directly deploy from them. For more information, see [Deploy from Registry](how-to-share-models-pipelines-across-workspaces-with-registries.md#deploy-model-from-registry-to-online-endpoint-in-workspace).
51+
1. Update applications invoking endpoints to use the scoring URIs of the new online endpoints.
52+
1. Delete online endpoints from old workspace after validation.
53+
54+
If you don't need to maintain computes or keep online endpoints and deployments in the old workspace to serve without downtime, you can simply delete all computes in the existing workspace, and update the workspace to enable workspace managed virtual network.
5155
5256
## Limitations
5357
@@ -76,6 +80,9 @@ If you've used the [legacy method](concept-secure-online-endpoint.md#secure-outb
7680
7781
When the workspace is configured with a private endpoint, the Azure Container Registry for the workspace must be configured for __Premium__ tier to allow access via the private endpoint. For more information, see [Azure Container Registry service tiers](../container-registry/container-registry-skus.md). Also, the workspace should be set with the `image_build_compute` property, as deployment creation involves building of images. See [Configure image builds](how-to-managed-network.md#configure-image-builds) for more.
7882
83+
> [!IMPORTANT]
84+
> When workspace managed virtual network is set up for a workspace for the first time, the network is not provisioned yet. Before proceeding to create online deployments, provision the network by following the guideline [Manually provision a managed network](how-to-managed-network.md#manually-provision-a-managed-vnet). Creating online deployments will be rejected until the managed network is provisioned.
85+
7986
1. Configure the defaults for the CLI so that you can avoid passing in the values for your workspace and resource group multiple times.
8087
8188
```azurecli

articles/machine-learning/how-to-troubleshoot-online-endpoints.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ There are two supported tracing headers:
193193
- `x-request-id` is reserved for server tracing. We override this header to ensure it's a valid GUID.
194194

195195
> [!Note]
196-
> When you create a support ticket for a failed request, attach the failed request ID to expedite the investigation.
196+
> When you create a support ticket for a failed request, attach the failed request ID to expedite the investigation. Alternatively, provide the name of the region and the endpoint name.
197197
198198
- `x-ms-client-request-id` is available for client tracing scenarios. This header is sanitized to only accept alphanumeric characters, hyphens and underscores, and is truncated to a maximum of 40 characters.
199199

@@ -229,6 +229,7 @@ The following list is of common deployment errors that are reported as part of t
229229
* [ResourceNotFound](#error-resourcenotfound)
230230
* [Azure Resource Manager can't find a required resource](#resource-manager-cannot-find-a-resource)
231231
* [Azure Container Registry is private or otherwise inaccessible](#container-registry-authorization-error)
232+
* [WorkspaceManagedNetworkNotReady](#error-workspacemanagednetworknotready)
232233
* [OperationCanceled](#error-operationcanceled)
233234
* [Operation was canceled by another operation that has a higher priority](#operation-canceled-by-another-higher-priority-operation)
234235
* [Operation was canceled due to a previous operation waiting for lock confirmation](#operation-canceled-waiting-for-lock-confirmation)
@@ -531,6 +532,10 @@ At deployment time, your online endpoint's system identity pulls the image from
531532

532533
For more diagnostic information, see [How To Use the Workspace Diagnostic API](../machine-learning/how-to-workspace-diagnostic-api.md).
533534

535+
### ERROR: WorkspaceManagedNetworkNotReady
536+
537+
This error occurs when you tried to create an online deployment under the workspace which enabled workspace managed VNet but the managed VNet is not provisioned yet. Workspace managed VNet should be provisioned before you create an online deployment. Follow instructions [Manually provision workspace managed VNet](how-to-managed-network.md#manually-provision-a-managed-vnet) to manually provision the workspace managed VNet. Once completed, you may start creating online deployments. For more information, see [Network isolation with managed online endpoint](concept-secure-online-endpoint.md) and [Secure your managed online endpoints with network isolation](how-to-secure-online-endpoint.md).
538+
534539
### ERROR: OperationCanceled
535540

536541
The following list is of reasons you might run into this error when using either managed online endpoint or Kubernetes online endpoint:

0 commit comments

Comments
 (0)