Merge pull request #256932 from ericd-mst-github/erd-aib-isolated-image-builds

Isolated image builds

Author: v-dirichards

articles/virtual-machines/TOC.yml

@@ -657,6 +657,9 @@
           href: ./windows/image-builder-virtual-desktop.md
       - name: Security
         items:
+        - name: Security mechanisms
+          displayName: Image builder, images, building
+          href: ./security-isolated-image-builds-image-builder.md
         - name: Security controls by Azure Policy
           displayName: Image builder, images, building
           href: ./security-controls-policy-image-builder.md

articles/virtual-machines/image-builder-api-update-release-notes.md

@@ -5,7 +5,7 @@ author: kof-f
 ms.service: virtual-machines
 ms.topic: conceptual
 ms.workload: infrastructure
-ms.date: 10/05/2023
+ms.date: 11/01/2023
 ms.reviewer: erd
 ms.subservice: image-builder
 ms.custom: references_regions
@@ -21,6 +21,10 @@ This article contains all major API changes and feature updates for the Azure VM
 
 ## Updates
 
+### November 2023
+Azure Image Builder is enabling Isolated Image Builds using Azure Container Instances in a phased manner. The rollout is expected to be completed by early 2024. Your existing image templates will continue to work and there is no change in the way you create or build new image templates. 
+
+You might observe a different set of transient Azure resources appear temporarily in the staging resource group but that does not impact your actual builds or the way you interact with Azure Image Builder. For more information, please see [Isolated Image Builds](./security-isolated-image-builds-image-builder.md).
 
 ### April 2023
 New portal functionality has been added for Azure Image Builder. Search “Image Templates” in Azure portal, then click “Create”. You can also [get started here](https://ms.portal.azure.com/#create/Microsoft.ImageTemplate) with building and validating custom images inside the portal.

articles/virtual-machines/linux/image-builder-troubleshoot.md

@@ -4,7 +4,7 @@ description: This article helps you troubleshoot common problems and errors you
 author: kof-f
 ms.author: kofiforson
 ms.reviewer: erd
-ms.date: 09/18/2023
+ms.date: 11/01/2023
 ms.topic: troubleshooting
 ms.service: virtual-machines
 ms.subservice: image-builder
@@ -21,13 +21,20 @@ Use this article to troubleshoot and resolve common issues that you might encoun
 
 When you're creating a build, do the following:
 
-- The VM Image Builder service communicates to the build VM by using WinRM or Secure Shell (SSH). Do *not* disable these settings as part of the build.
-- VM Image Builder creates resources as part of the build. Be sure to verify that Azure Policy doesn't prevent VM Image Builder from creating or using necessary resources.
+- The VM Image Builder service communicates to the build VM by using WinRM or Secure Shell (SSH). Don't* disable these settings as part of the build.
+- VM Image Builder creates resources in the staging resource group as part of the builds. Be sure to verify that Azure Policy doesn't prevent VM Image Builder from creating or using necessary resources.
   - Create an IT_ resource group.
   - Create a storage account without a firewall.
+  - Deploy [Azure Container Instances](../../container-instances/container-instances-overview.md).
+  - Deploy [Azure Virtual Network resources](../../virtual-network/virtual-networks-overview.md) (and subnets therein).
+  - Deploy [Azure Private Endpoint](../../private-link/private-endpoint-overview.md) resources.
+  - Deploy [Azure Files](../../storage/files/storage-files-introduction.md).
 - Verify that Azure Policy doesn't install unintended features on the build VM, such as Azure Extensions.
 - Ensure that VM Image Builder has the correct permissions to read/write images and to connect to the storage account. For more information, review the permissions documentation for the [Azure CLI](./image-builder-permissions-cli.md) or [Azure PowerShell](./image-builder-permissions-powershell.md).
-- VM Image Builder will fail the build if the scripts or inline commands fail with errors (non-zero exit codes). Ensure that you've tested the custom scripts and verified that they run without error (exit code 0) or require user input. For more information, see [Create an Azure Virtual Desktop image by using VM Image Builder and PowerShell](../windows/image-builder-virtual-desktop.md#tips-for-building-windows-images).
+- VM Image Builder fails the build if the scripts or inline commands fail with errors (nonzero exit codes). Ensure that you've tested the custom scripts and verified that they run without error (exit code 0) or require user input. For more information, see [Create an Azure Virtual Desktop image by using VM Image Builder and PowerShell](../windows/image-builder-virtual-desktop.md#tips-for-building-windows-images).
+- Ensure your subscription has sufficient [quota](../../container-instances/container-instances-resource-and-quota-limits.md) of Azure Container Instances.
+    - Each image build might deploy up to one temporary Azure Container Instance resource (of four standard cores) in the staging resource group. These resources are required for [Isolated image builds](../security-isolated-image-builds-image-builder.md).
+
 
 VM Image Builder failures can happen in two areas:
 
@@ -144,7 +151,7 @@ Microsoft.VirtualMachineImages/imageTemplates 'helloImageTemplateforSIG01' faile
 
 #### Cause
 
-In most cases, the resource deployment failure error occurs because of missing permissions. This error may also be caused by a conflict with the staging resource group.
+In most cases, the resource deployment failure error occurs because of missing permissions. This error might also be caused by a conflict with the staging resource group.
 
 #### Solution
 
@@ -728,6 +735,57 @@ The cause might be a timing issue because of the D1_V2 VM size. If customization
 
 To avoid the timing issue, you can increase the VM size or you can add a 60-second PowerShell sleep customization.
 
+### Azure Container Instances quota exceeded
+
+#### Error
+"Azure Container Instances quota exceeded"
+
+#### Cause
+Your subscription doesn't have enough Azure Container Instances (ACI) quota for Azure Image Builder to successfully build an image.
+
+#### Solution
+You can do the following to make ACI quota available for Azure Image Builder:
+- Lookup other usage of Azure Container Instances in your subscription and remove any unneeded instances to make quota available for Azure Image Builder.
+- Azure Image Builder deploys ACI only temporarily while a build is taking place. These instances are deleted once the build completes. If too many concurrent image builds are taking place in your subscription, then you can consider delaying some of the image builds. This reduces concurrent usage of ACI in your subscription. If your image templates are set up for automatic image builds using triggers, then such failed builds will automatically be retried by Azure Image Builder.
+- If the current ACI limits for your subscription are too low to support your image building scenarios, then you can request an increase in your [ACI quota](../../container-instances/container-instances-resource-and-quota-limits.md#next-steps).
+
+> [!NOTE]
+> ACI resources are required for [Isolated Image Builds](../security-isolated-image-builds-image-builder.md).
+
+### Too many Azure Container Instances deployed within a period of time
+
+#### Error
+"Too many Azure Container Instances deployed within a period of time"
+
+#### Cause
+Your subscription doesn't have enough Azure Container Instances (ACI) quota for Azure Image Builder to successfully build images concurrently.
+
+#### Solution
+You can do the following:
+- Retry your failed builds in a less concurrent manner.
+- If the current ACI limits for your subscription are too low to support your image building scenarios, then you can request an increase in your [ACI quota](../../container-instances/container-instances-resource-and-quota-limits.md#next-steps).
+
+### Isolated Image Build failure
+
+#### Error
+Azure Image Builder builds are failing due to Isolated Image Build.
+
+#### Cause
+Azure Image Builder builds can fail for reasons listed elsewhere in this document. However, there's a small chance that a build fails due to Isolated Image Builds depending on your scenario, subscription quotas, or some unforeseen service error. For more information, see [Isolated Image Builds](../security-isolated-image-builds-image-builder.md).
+
+#### Solution
+If you determine that a build is failing due to Isolated Image Builds, you can do the following:
+- Ensure there's no [Azure Policy](../../governance/policy/overview.md) blocking the deployment of resources mentioned in the Prerequisites section, specifically Azure Container Instances, Azure Virtual Networks, and Azure Private Endpoints.
+- Ensure your subscription has sufficient quota of Azure Container Instances to support all your concurrent image builds. For more information, see, Azure Container Instances [quota exceeded](./image-builder-troubleshoot.md#azure-container-instances-quota-exceeded).
+
+Azure Image Builder is currently in the process of deploying Isolated Image Builds. Specific image templates are not tied to Isolated Image Builds and the same image template might or might not utilize Isolated Image Builds during different builds. You can do the following to temporarily run your build without Isolated Image Builds.
+- Retry your build. Since Image Templates are not tied to the Isolated Image Builds feature, retrying a build has a high probability of rerunning without Isolated Image Builds.
+
+ If none of these solutions mitigate failing image builds, then you can contact Azure support to temporarily opt your subscription out of Isolated Image Builds. For more information, see [Create an Azure support request](../../azure-portal/supportability/how-to-create-azure-support-request.md).
+
+> [!NOTE]
+> Isolated Image Builds will eventually be enabled in all regions and templates. So, the above mitigations should be considered temporary and the underlying cause of build failures must be addressed.
+
 ### The build is canceled after the context cancelation context is canceled
 
 #### Error
@@ -798,7 +856,7 @@ Making these observations is especially important in build failures, where these
 
 #### Error
 
-When images are stuck in template deletion, the customization log may show the below error:
+When images are stuck in template deletion, the customization log might show the below error:
 
 ```output
 error deleting resource id /subscriptions/<subscriptionID>/resourceGroups/<rgName>/providers/Microsoft.Network/networkInterfaces/<networkInterfacName>: resources.Client#DeleteByID: Failure sending request: StatusCode=400 -- 

articles/virtual-machines/security-isolated-image-builds-image-builder.md

@@ -0,0 +1,45 @@
+---
+title: Isolated Image Builds for Azure VM Image Builder
+description: Isolated Image Builds is achieved by transitioning core process of VM image customization/validation from shared infrastructure to dedicated Azure Container Instances resources in your subscription providing compute and network isolation. 
+ms.date: 11/01/2023
+ms.topic: sample
+author: kof-f
+ms.author: erd
+ms.reviewer: erd
+ms.service: virtual-machines
+ms.subservice: image-builder
+
+---
+
+# What is Isolated Image Builds for Azure Image Builder?
+
+Isolated Image Builds is a feature of Azure Image Builder (AIB). It transitions the core process of VM image customization/validation from shared infrastructure to dedicated Azure Container Instances (ACI) resources in your subscription, providing compute and network isolation.
+
+## Advantages of Isolated Image Builds
+
+Isolated Image Builds enable defense-in-depth by limiting network access of your build VM to just your subscription. Isolated Image Builds also provide you with more transparency by allowing your inspection of the processing done by Image Builder to customize/validate your VM image. Further, Isolated Image Builds eases viewing of live build logs. Specifically:
+
+1. **Compute Isolation:** Isolated Image Builds perform major portion of image building processing in Azure Container Instances resources in your subscription instead of on AIB's shared platform resources. ACI provides hypervisor isolation for each container group to ensure containers run in isolation without sharing a kernel.
+2. **Network Isolation:**  Isolated Image Builds remove all direct network WinRM/ssh communication between your build VM and Image Builder service. 
+    - If you are provisioning an Image Builder template without your own Virtual Network then a Public IP Address resource will no more be provisioned in your staging resource group at image build time.
+    - If you are provisioning an Image Builder template with an existing Virtual Network in your subscription then a Private Link based communication channel will no more be setup between your Build VM and AIB's backend platform resources. Instead, the communication channel will be setup between the Azure Container Instance and the Build VM resources - both of which reside in the staging resource group in your subscription.
+3. **Transparency:** AIB is built on HashiCorp [Packer](https://www.packer.io/). Isolated Image Builds executes Packer in the ACI in your subscription, which allows you to inspect the ACI resource and its containers. Similarly, having the entire network communication pipeline in your subscription allows you to inspect all the network resources, their settings, and their allowances.
+4. **Better viewing of live logs:** AIB writes customization logs to a storage account in the staging resource group in your subscription. Isolated Image Builds provides with another way to follow the same logs directly in the Azure portal which can be done by navigating to Image Builder's container in the ACI resource.
+
+## Backward compatibility
+
+This is a platform level change and doesn't affect AIB's interfaces. So, your existing Image Template and Trigger resources continue to function and there's no change in the way you'll deploy new resources of these types. Similarly, customization logs continue to be available in the storage account.
+
+You might observe a few new resources temporarily appear in the staging resource group (for example, Azure Container Instance, and Private Endpoint) while some other resource will no longer appear (for example, Public IP Address). Just as earlier, these temporary resources will exist only for the duration of the build and will be deleted by Image Builder thereafter.
+
+Your image builds will automatically be migrated to Isolated Image Builds and you need to take no action to opt-in.
+
+> [!NOTE]
+> Image Builder is in the process of rolling this change out to all locations and customers. Some of these details might change as the process is fine-tuned based on service telemetry and feedback. Please refer to the [troubleshooting guide](./linux/image-builder-troubleshoot.md#troubleshoot-build-failures) for more information.
+
+## Next steps
+
+- [Azure VM Image Builder overview](./image-builder-overview.md)
+- [Getting started with Azure Container Instances](../container-instances/container-instances-overview.md)
+- [Securing your Azure resources](../security/fundamentals/overview.md)
+- [Troubleshooting guide for Azure VM Image Builder](./linux/image-builder-troubleshoot.md#troubleshoot-build-failures)