Update configure scaling with predownload information

Author: bryanrichter

docs/managed-devops-pools/configure-scaling.md

@@ -15,7 +15,7 @@ Managed DevOps Pools can be configured as stateless or stateful.
 * [Stateless pools](#stateless-pools) - Provide a fresh agent for every job.
 * [Stateful pools](#stateful-pools) - Allow sharing of agents between multiple jobs.
 
-The default setting for a Managed DevOps pool is stateless (**Fresh agent every time**) but in some cases teams might want to reuse agents in order to reuse the packages or files created during the previous pipeline run. Build workload is a common scenario where teams want to preserve state and reuse agents. You can achieve stateful pools through Managed DevOps Pools while balancing it with security best practices. By default an agent can be reused for a maximum of 7 days but you can configure it to be recycled sooner. 
+The default setting for a Managed DevOps pool is stateless (**Fresh agent every time**) but in some cases teams might want to reuse agents in order to reuse the packages or files created during the previous pipeline run. Build workload is a common scenario where teams want to preserve state and reuse agents. You can achieve stateful pools through Managed DevOps Pools while balancing it with security best practices. By default an agent can be reused for a maximum of 7 days but you can configure it to be recycled sooner.
 
 > [!NOTE]
 > Stateless pools or use of the agent state setting **Fresh agent every time** are recommended by security experts as a defense against supply chain attacks.
@@ -24,6 +24,8 @@ The default setting for a Managed DevOps pool is stateless (**Fresh agent every
 
 When a stateless agent is configured, a new agent is procured for each job, and is discarded after the job completes.
 
+For the lifecycle of stateless agents and an explanation on how they are used in Azure DevOps pipelines (including potential delays in allocation), please see [below](#lifecycle-of-agents-and-potential-delays-in-allocation)
+
 #### [Azure portal](#tab/azure-portal/)
 
 :::image type="content" source="media/agent-performance/stateless-agent.png" alt-text="Screenshot of a stateless agent.":::
@@ -884,6 +886,21 @@ If you don't know your usage patterns and want to rely on automatic forecasting
 
 * * *
 
+* * *
+
+## Lifecycle of Agents and potential delays in allocation
+
+When using a Stateless scheme, ready agents require the Azure DevOps Task Agent to both be installed and set up in order to run an Azure DevOps pipeline. When provisioning new agents, our service will attempt to download the latest Azure DevOps VSTS Task agent offered [to the public](https://github.com/microsoft/azure-pipelines-agent/releases) in order to have it already downloaded on buffered agents. Startup, connection, and beginning the job can take anywhere from 10 seconds to a minute depending on the pool's SKU speed, image used, and networking load. Additionally, certain settings in an Azure DevOps's job can cause a redownload and running of a different agent, and regressions + rollbacks of the Task Agent can also cause a redownload of the agent. Ready agents will always have a potential delay, as Managed DevOps Pools uses this agent in an "ephemeral" manner, meaning we startup and run the task agent one time per job.
+
+If you are seeing delays in ready agents picking up jobs from Azure DevOps, the following are important to consider:
+
+* Do you have ready agents? - The most common issue is a misunderstanding of when agents should be preprovisioned. When the amount of jobs queued is greater than the buffer on a pool, or jobs are queued when the buffer is set to be empty, then machines must be spun up from scratch.
+* Are you using the buffer with multiple images properly? - If you are not specifying which image to use in your pipeline using the ImageOverride demand, jobs will be targeting the first image. This means, depending on your Scaling settings, you might not have as many agents available as you'd expect as some are allocated to other images.
+* Are Proxy/VNet/Firewall settings slowing down your pool? - Potential slowness from any network setting will result in agents taking longer to start the agent and connect it to Azure DevOps.
+* Are you overriding the agent version? - By default, Managed DevOps pools will run on the most recent Azure DevOps task agent version. Settings in the Pipeline yaml (such as the "Agent.Version" demand) and Azure DevOps org can force pipelines to use older versions of the task agent, requiring a redownload once a machine has been allocated.
+
+* * *
+
 ## See also
 
 * [Configure pool settings](./configure-pool-settings.md)