You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Azure DevOps Services uses multi-tenancy to reduce costs and improve performance. This design leaves users vulnerable to performance issues and even outages when other users of their shared resources have spikes in their consumption. So, Azure DevOps limits the resources individuals can consume, and the amount of requests they can make to certain commands. When these limits are exceeded, future requests might be either delayed or blocked.
20
+
Azure DevOps Services uses multi-tenancy to reduce costs and improve performance. This design can cause performance issues or outages when other users of shared resources have spikes in consumption. To help prevent this, Azure DevOps limits the resources each user can consume, and the number of requests they can make to certain commands. If you exceed these limits, future requests can be delayed or blocked.
21
21
22
-
For more information, see[Git limits](../../repos/git/limits.md) and [Best practices to avoid hitting rate limits](integration-bestpractices.md).
22
+
Learn more in[Git limits](../../repos/git/limits.md) and [Best practices to avoid hitting rate limits](integration-bestpractices.md).
23
23
24
24
## Global consumption limit
25
25
26
-
Azure DevOps currently has a global consumption limit, which delays requests from individual users beyond a threshold when shared resources are in danger of being overwhelmed. This limit is focused exclusively on avoiding outages when shared resources are close to being overwhelmed.
27
-
Individual users typically only get delayed requests when one of the following incidents occurs:
26
+
Azure DevOps has a global consumption limit that delays requests from individual users when shared resources are at risk of being overwhelmed. This limit helps avoid outages when shared resources are close to being overwhelmed.
27
+
Individual users typically experience delayed requests only when one of the following incidents occurs:
28
28
29
-
- One of their shared resources is at risk of being overwhelmed
30
-
- Their personal usage exceeds 200 times the consumption of a typical user within a (sliding) five-minute window
29
+
- One of their shared resources is at risk of being overwhelmed.
30
+
- Their personal usage exceeds 200 times the consumption of a typical user within a sliding five-minute window.
31
31
32
-
The amount of the delay depends on the user's sustained level of consumption. Delays range from a few milliseconds per request up to 30 seconds. Once consumption goes to zero or the resource is no longer overwhelmed, the delays stop within five minutes. If consumption remains high, delays might continue indefinitely to protect the resource.
32
+
The delay depends on the user's sustained level of consumption. Delays range from a few milliseconds per request up to 30 seconds. When consumption drops to zero or the resource isn't overwhelmed, the delays stop within five minutes. If consumption stays high, delays can continue indefinitely to protect the resource.
33
33
34
-
When a user request gets delayed by a significant amount, that user receives an email and a warning banner in the web.
35
-
For the build service account and others without an email address, members of the Project Collection Administrators group get the email.
34
+
When a user request is delayed by a significant amount, the user receives an email and a warning banner in the web.
35
+
For the build service account and others without an email address, members of the Project Collection Administrators group receive the email.
36
36
For more information, see [Usage monitoring](../../organizations/accounts/usage-monitoring.md).
37
37
38
-
When an individual user's requests get blocked, responses with HTTP code 429 (too many requests) are received, with a message similar to the following message:
38
+
When an individual user's requests are blocked, the user receives responses with HTTP code 429 (too many requests) and a message similar to the following:
39
39
40
-
```TF400733: The request has been canceled: Request was blocked due to exceeding usage of resource <resource name> in namespace <namespace ID>.```
40
+
```text
41
+
TF400733: The request has been canceled: Request was blocked due to exceeding usage of resource <resource name> in namespace <namespace ID>.
42
+
```
41
43
42
44
### Azure DevOps throughput units
43
45
44
-
Azure DevOps users consume many shared resources, and consumption depends on the following factors:
46
+
Azure DevOps users consume many shared resources, and the level of consumption depends on factors like:
47
+
* Uploading a large number of files to version control, which puts load on databases and storage accounts.
48
+
* Running complex work item queries, which increases database load based on the number of work items being searched.
49
+
* Running builds, which download files from version control and produce log output.
50
+
* General operations, which consume CPU and memory across different parts of the service.
45
51
46
-
- Uploading a large number of files to version control creates a large amount of load on databases and storage accounts
47
-
- Complex work item tracking queries create database load based on the number of work items they search through
48
-
- Builds drive load by downloading files from version control, producing log output
49
-
- All operations consume CPU and memory on various parts of the service
52
+
To measure this activity, Azure DevOps expresses resource consumption in **Azure DevOps throughput units (TSTUs)**. A TSTU is an abstract unit of load that represents a blend of different resources, including:
53
+
* Database usage—measured primarily through Azure SQL Database DTUs.
54
+
* Compute usage—CPU, memory, and I/O from application tiers and job agents.
55
+
* Storage usage—Azure Storage bandwidth.
50
56
51
-
To accommodate, Azure DevOps resource consumption is expressed in abstract units called Azure DevOps throughput units (TSTUs). TSTUs eventually incorporate a blend of the following items:
52
-
53
-
-[Azure SQL Database DTUs](/azure/azure-sql/database/purchasing-models) as a measure of database consumption
54
-
- Application tier and job agent CPU, memory, and I/O as a measure of compute consumption
55
-
- Azure Storage bandwidth as a measure of storage consumption
56
-
57
-
For now, TSTUs are primarily focused on Azure SQL Database DTUs, since Azure SQL Databases are the shared resources most commonly overwhelmed by excessive consumption. A single TSTU is the average load we expect a typical user of Azure DevOps to generate per five minutes. Typical users also generate spikes in load. These spikes are typically 10 or fewer TSTUs per five minutes. Less frequently, spikes go as high as 100 TSTUs.
57
+
> [!NOTE]
58
+
> TSTUs are intentionally abstract. They aggregate resource consumption across compute, storage, and database layers within a distributed infrastructure. The underlying metrics (CPU, memory, I/O, DTUs) aren't directly exposed or meaningful on their own. TSTUs provide a unified way to represent load, making it easier to manage and monitor usage without exposing the full complexity of individual resource components. You can't calculate usage in TSTUs for an action with a formula, but you can see how many TSTUs an operation consumes on the [usage monitoring](../../organizations/accounts/usage-monitoring.md) page. Some operations, like work item queries, vary in consumption as your organization grows and changes, so you might need to benchmark periodically to stay accurate.
58
59
59
-
The global consumption limit is 200 TSTUs within a sliding five-minute window.
60
+
Currently, TSTUs focus primarily on Azure SQL Database DTUs because databases are the shared resource most likely to be overwhelmed by excessive consumption.
61
+
* One TSTU represents the average load generated by a typical Azure DevOps user over five minutes.
62
+
* Normal user activity can generate spikes of 10 TSTUs or fewer per five minutes.
63
+
* Larger but less frequent spikes can reach up to 100 TSTUs.
64
+
* The global limit is 200 TSTUs within any sliding five-minute window.
60
65
61
-
We recommend that you at least respond to the `Retry-After` header. If you detect a `Retry-After` header in any response, wait until some time passes before you send another request. Doing so helps your client application experience fewer enforced delays. Keep in mind that the response is 200, so you don't need to apply retry logic to the request.
66
+
### Best practices
62
67
63
-
If possible, we further recommend that you monitor `X-RateLimit-Remaining` and `X-RateLimit-Limit` headers. Doing so allows you to approximate how quickly you're approaching the delay threshold. Your client can intelligently react and spread out its requests over time.
68
+
* Honor the Retry-After header: If you receive it in a response, wait the specified time before sending another request. The response still returns HTTP 200, so retry logic isn't required.
69
+
* Monitor X-RateLimit headers: If available, track `X-RateLimit-Remaining` and `X-RateLimit-Limit` to approximate how quickly you're approaching the threshold. This lets your client smooth out request bursts and avoid enforced delays.
64
70
65
71
> [!NOTE]
66
-
> Identities used by tools and applications to integrate with Azure DevOps might occasionally need higher rate and usage limits beyond the allowed consumption limit. You can increase these limits by assigning the [Basic + Test Plans](../../organizations/billing/buy-basic-access-add-users.md#assign-basic-or-basic--test-plans) access level to the desired identities used by your application. Once the need for higher rate limits is fulfilled, you can revert to the previous access level. You get charged for the [Basic + Test Plans](../../organizations/billing/buy-basic-access-add-users.md#assign-basic-or-basic--test-plans) access level only for the duration assigned to the identity.
67
-
>
68
-
> Identities already assigned a Visual Studio Enterprise subscription can't get assigned the [Basic + Test Plans](../../organizations/billing/buy-basic-access-add-users.md#assign-basic-or-basic--test-plans) access level until they get removed.
72
+
> Identities used by tools and applications to integrate with Azure DevOps can occasionally need higher rate and usage limits beyond the allowed consumption limit. Increase these limits by assigning the [Basic + Test Plans](../../organizations/billing/buy-basic-access-add-users.md#assign-basic-or-basic--test-plans) access level to the identities your application uses. After you no longer need higher rate limits, revert to the previous access level. You're charged for the [Basic + Test Plans](../../organizations/billing/buy-basic-access-add-users.md#assign-basic-or-basic--test-plans) access level only for the duration assigned to the identity.
73
+
Identities already assigned a Visual Studio Enterprise subscription can't be assigned the [Basic + Test Plans](../../organizations/billing/buy-basic-access-add-users.md#assign-basic-or-basic--test-plans) access level until you remove the subscription.
69
74
70
75
## Pipelines
71
76
72
-
Rate limiting is similar for Azure Pipelines. Each pipeline gets treated as an individual entity with its own resource consumption tracked. Even if build agents are self-hosted, they generate load in the form of cloning and sending logs.
77
+
Rate limiting works the same way for Azure Pipelines. Each pipeline is an individual entity, and its resource consumption is tracked separately. Even if build agents are self-hosted, they generate load by cloning and sending logs.
73
78
74
-
We apply a 200 TSTU limit for an individual pipeline in a sliding 5-minute window. This limit is the same as the global consumption limit for users.
75
-
If a pipeline gets delayed or blocked by rate limiting, a message appears in the attached logs.
79
+
There's a 200 TSTU limit for each pipeline in a sliding 5-minute window. This limit matches the global consumption limit for users.
80
+
If rate limiting delays or blocks a pipeline, you see a message in the attached logs.
76
81
77
82
## API client experience
78
83
79
-
When requests get delayed or blocked, Azure DevOps returns response headers to help API clients react. While not fully standardized, these headers are [broadly in line with other popular services](https://stackoverflow.com/questions/16022624/examples-of-http-api-rate-limiting-http-response-headers).
84
+
When requests are delayed or blocked, Azure DevOps returns response headers to help API clients react. While not fully standardized, these headers are [broadly in line with other popular services](https://stackoverflow.com/questions/16022624/examples-of-http-api-rate-limiting-http-response-headers).
80
85
81
-
The following table lists the headers available and what they mean.
82
-
Except for `X-RateLimit-Delay`, all of these headers get sent before requests start getting delayed.
83
-
This design gives clients the opportunity to proactively slow down their rate of requests.
86
+
The following table lists the available headers and what they mean.
87
+
Except for `X-RateLimit-Delay`, all these headers are sent before requests start getting delayed.
88
+
This design lets clients proactively slow down their rate of requests.
84
89
85
90
:::row:::
86
91
:::column span="1":::
@@ -93,19 +98,8 @@ This design gives clients the opportunity to proactively slow down their rate of
93
98
---
94
99
:::row:::
95
100
:::column span="1":::
96
-
`Retry-After`
97
-
:::column-end:::
98
-
:::column span="2":::
99
-
The [RFC 6585](https://tools.ietf.org/html/rfc6585#section-4")-specified header sent to tell you how long to wait before you send your next request to fall under the detection threshold. Units: seconds.
100
-
:::column-end:::
101
-
:::row-end:::
102
-
---
103
-
:::row:::
104
-
:::column span="1":::
105
-
`X-RateLimit-Resource`
106
-
:::column-end:::
107
-
:::column span="2":::
108
-
A custom header indicating the service and type of threshold that was reached. Threshold types and service names might vary over time and without warning. We recommend displaying this string to a human, but not relying on it for computation.
101
+
`Retry- A custom header that shows the service and type of threshold reached. Threshold types and service names can vary over time and without warning. Display this string to a human, but don't rely on it for computation.
102
+
109
103
:::column-end:::
110
104
:::row-end:::
111
105
---
@@ -114,7 +108,8 @@ This design gives clients the opportunity to proactively slow down their rate of
114
108
`X-RateLimit-Delay`
115
109
:::column-end:::
116
110
:::column span="2":::
117
-
How long the request was delayed. Units: seconds with up to three decimal places (milliseconds).
111
+
How long the request is delayed. Units: seconds with up to three decimal places (milliseconds).
112
+
118
113
:::column-end:::
119
114
:::row-end:::
120
115
---
@@ -132,7 +127,8 @@ This design gives clients the opportunity to proactively slow down their rate of
132
127
`X-RateLimit-Remaining`
133
128
:::column-end:::
134
129
:::column span="2":::
135
-
Number of TSTUs remaining before being delayed. If requests are already being delayed or blocked, it's 0.
130
+
Number of TSTUs remaining before delays start. If requests are already delayed or blocked, it's 0.
131
+
136
132
:::column-end:::
137
133
:::row-end:::
138
134
---
@@ -141,22 +137,23 @@ This design gives clients the opportunity to proactively slow down their rate of
141
137
`X-RateLimit-Reset`
142
138
:::column-end:::
143
139
:::column span="2":::
144
-
Time at which, if all resource consumption stopped immediately, tracked usage would return to 0 TSTUs. Expressed in Unix epoch time.
140
+
Time when, if all resource consumption stops immediately, tracked usage returns to 0 TSTUs. Expressed in Unix epoch time.
141
+
145
142
:::column-end:::
146
143
:::row-end:::
147
144
---
148
145
149
146
## Work tracking, process, & project limits
150
147
151
-
Azure DevOps imposes limits for the number of projects you can have in an organization and the number of teams you can have within each project. Also be aware of limits for work items, queries, backlogs, boards, dashboards, and more. For more information, see [Work tracking, process, and project limits](../../organizations/settings/work/object-limits.md).
148
+
Azure DevOps limits the number of projects you can have in an organization and the number of teams you can have in each project. There are also limits for work items, queries, backlogs, boards, dashboards, and more. For more information, see [Work tracking, process, and project limits](../../organizations/settings/work/object-limits.md).
152
149
153
150
## Wiki
154
151
155
-
In addition to the usual [repository limits](../../repos/git/limits.md#repository-size), wikis defined for a project are limited to 25 MB per single file.
152
+
In addition to the usual [repository limits](../../repos/git/limits.md#repository-size), a wiki file in a project can be up to 25 MB.
156
153
157
154
## Service connections
158
155
159
-
There are no per-project limits placed on creating service connections. However, there might be limits, which are imposed through Microsoft Entra ID. For additional information, review the following articles:
156
+
There aren't any per-project limits on creating service connections. However, limits might be imposed through Microsoft Entra ID. For more information, see the following articles:
160
157
161
158
-[Microsoft Entra service limits and restrictions](/azure/active-directory/enterprise-users/directory-service-limits-restrictions)
162
159
-[Azure subscription and service limits, quotas, and constraints](/azure/azure-resource-manager/management/azure-subscription-service-limits)
0 commit comments