Skip to content

Commit 14d6ad6

Browse files
authored
Merge pull request #8091 from JKirsch1/refresh-service-hook-troubleshooting-article
Freshness Edit: Azure DevOps - Service hooks
2 parents bc4b223 + f6c3594 commit 14d6ad6

File tree

4 files changed

+75
-69
lines changed

4 files changed

+75
-69
lines changed
68.6 KB
Loading
21.3 KB
Loading
-26.9 KB
Binary file not shown.

docs/service-hooks/troubleshoot.md

Lines changed: 75 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -1,142 +1,148 @@
11
---
22
ms.subservice: azure-devops-service-hooks
3-
ms.topic: conceptual
4-
title: Troubleshoot your service hooks integrations | Azure DevOps Services
5-
description: Troubleshoot problems with the services you've integrated with your organization.
3+
ms.topic: troubleshooting-general
4+
title: Troubleshoot Service Hook Integrations
5+
description: Find out how to access the history of service hook subscriptions in Azure DevOps. Get information about HTTP response failures that affect subscription states.
66
ms.assetid: dcf00653-24c5-4ab6-b9e8-19ec098bbb66
77
ms.custom: engagement-fy23
88
ms.author: chcomley
99
author: chcomley
1010
monikerRange: '<= azure-devops'
11-
ms.date: 12/02/2022
11+
ms.date: 07/03/2025
12+
# customer intent: As a team member, I want to become familiar with service hook failure types and find out how to access the history of subscriptions so that I can troubleshoot problems with service hooks in Azure DevOps.
1213
---
1314

1415
# Troubleshoot service hooks
1516

16-
[!INCLUDE [version-lt-eq-azure-devops](../includes/version-lt-eq-azure-devops.md)]
17+
[!INCLUDE [Azure DevOps Services | Azure DevOps Server 2022 | Azure DevOps Server 2020](../includes/version-gt-eq-2020.md)]
1718

18-
Use this article for general troubleshooting guidance and answers to frequently asked questions (FAQs).
19+
This article offers general troubleshooting guidance for Azure DevOps service hooks. It also provides answers to frequently asked questions (FAQs).
1920

2021
## View activity and debug problems
2122

22-
The **Service Hooks** page in the web access admin shows your recent activity (last 14 days)
23-
for each subscription, and whether a subscription is enabled, disabled, or restricted.
23+
The **Service Hooks** page in the web access admin summarizes activity from the last seven days for each subscription. The page also shows whether each subscription is enabled, disabled, or restricted.
2424

25-
You can access detailed history about a subscription including detailed request/response data, which is useful for debugging a problematic service or subscription.
26-
27-
::: moniker range="<=azure-devops"
25+
For each subscription, you can access a detailed history that includes the complete request and response data for each event. This information can help you debug a problematic service or subscription.
2826

2927
1. To view the activity and status of your subscriptions, go to the **Service Hooks** page.
3028

31-
:::image type="content" source="media/troubleshoot/devops-service-hooks.PNG" alt-text="Screenshot showing view of activity and status of subscriptions.":::
29+
:::image type="content" source="media/troubleshoot/azure-devops-service-hooks.png" alt-text="Screenshot of the Service Hooks page showing the state, status, and other data for subscriptions. Project settings and Service hooks are highlighted." lightbox="media/troubleshoot/azure-devops-service-hooks.png":::
3230

3331
1. To view detailed activity for a subscription, including full request, response, and event payload data, select a subscription in the table, and then select **History**.
3432

35-
:::image type="content" source="media/troubleshoot/detailed-activity.png" alt-text="Screenshot showing detailed view of activity for a subscription.":::
36-
37-
::: moniker-end
33+
:::image type="content" source="media/troubleshoot/detailed-activity.png" alt-text="Screenshot that shows the history of a subscription, with detailed information for a failed event, such as the status, message, and error message." lightbox="media/troubleshoot/detailed-activity.png":::
3834

3935
## Subscription failures and probation (restricted)
4036

37+
If the HTTP response to a notification request indicates an error, the severity of the failure determines how Azure DevOps responds. Certain types of failures can disable subscriptions or put them on probation.
38+
4139
### Failure types
42-
Failures from a Service Hooks notification are grouped into the following categories:
4340

44-
* Terminal Failures
45-
* Transient Failures
46-
* Enduring Failures
41+
Failures from service hook notifications are grouped into the following categories:
42+
43+
* Terminal failures
44+
* Transient failures
45+
* Enduring failures
4746

48-
#### Terminal Failures
47+
The error code from the HTTP response determines how Azure DevOps categorizes the failure.
4948

50-
The only Terminal Failure is HTTP Status Code 410 (Gone). When a subscription sees a Terminal Failure, it's automatically disabled no matter its prior status.
49+
#### Terminal failures
5150

52-
#### Transient Failures
51+
The only HTTP status code that's categorized as a terminal failure is 410 (Gone).
52+
53+
When a terminal failure occurs in a subscription, the subscription is automatically disabled no matter its prior state.
54+
55+
#### Transient failures
56+
57+
HTTP responses with the following status codes are categorized as transient failures:
5358

54-
When a subscription sees a Transient Failure, it attempts to resend the notification up to eight times, with an increasing delay between each attempt. Transient Failures include the following codes:
5559
* 408 (Request Timeout)
5660
* 502 (Bad Gateway)
5761
* 503 (Service Unavailable)
5862
* 504 (Gateway Timeout)
5963

60-
#### Sequence of retries for transient failures
64+
When a transient failure occurs in a subscription, Azure DevOps attempts to resend the notification up to eight times, with an increasing delay between each attempt.
6165

62-
|Retry # |Wait time |
63-
|---------|---------|
64-
|Before retry 1 | wait ~1 second |
65-
|Before retry 2 | wait ~2 seconds (total delay of 3 seconds) |
66-
|Before retry 3 | wait ~4 seconds (total delay of 7 seconds) |
67-
|Before retry 4 | wait ~8 seconds (total delay of 15 seconds) |
68-
|Before retry 5 | wait ~16 seconds (total delay of 31 seconds) |
69-
|Before retry 6 | wait ~32 seconds (total delay of 63 seconds) |
70-
|Before retry 7 | wait ~60 seconds (max backoff time, total delay of 123 seconds) |
71-
|Before retry 8 | wait ~60 seconds (max backoff time, total delay of 183 seconds)|
66+
The following table lists information about retries that are attempted after a transient failure occurs. Included is the approximate _backoff_ time, or the time to wait before attempting to resend a notification. The maximum backoff time is 60 seconds. The table also shows the total delay for each retry.
7267

73-
If the notification exhausts all of its retries and continues to see a Transient Failure for each attempt, the subscription stops trying to send the notification, and treats the notification as if it saw an Enduring Failure.
68+
| Retry number | Backoff time in seconds | Total delay in seconds |
69+
|---------|---------|---------|
70+
| 1 | 1 | 1 |
71+
| 2 | 2 | 3 |
72+
| 3 | 4 | 7 |
73+
| 4 | 8 | 15 |
74+
| 5 | 16 | 31 |
75+
| 6 | 32 | 63 |
76+
| 7 | 60 | 123 |
77+
| 8 | 60 | 183 |
7478

75-
#### Enduring Failures
79+
If all retries for a notification are exhausted and each attempt results in a transient failure, the notification is no longer sent. Instead, it's categorized as an enduring failure.
7680

77-
Enduring Failures include all other HTTP failure codes, for example: 404 (Not Found), 500 (Internal Server Error), and so on.
81+
#### Enduring failures
7882

79-
When a subscription sees an Enduring Failure, it's placed on [probation](#probation).
83+
All other HTTP failure codes, for example, 404 (Not Found) and 500 (Internal Server Error), result in enduring failures.
8084

81-
### Probation
85+
When an enduring failure occurs in a subscription, the subscription is placed on [probation](#probation).
8286

83-
While on probation, a subscription is limited in the number of notifications it can send. If the subscription continues to hit Enduring Failures, then it gets increasingly limited, and eventually disabled. If the subscription receives a successful response while on probation, it gets restored to a fully enabled state.
87+
### Probation
8488

85-
#### Sequence of seven maximum retries while subscription is on probation
89+
When a subscription is on probation, any new events are lost. The system makes a limited number of attempts to resend the failed notification.
8690

87-
When a subscription is in probation, any new events are lost. Once a retry is successful, the subscription is enabled, and events are published again.
91+
The following table lists approximate backoff times and total probation times for retries that are attempted during probation. At most seven retries are attempted, and the maximum backoff time for a probation retry is 15 hours.
8892

89-
|Retry # |Wait time |
90-
|---------|---------|
91-
|Before retry 1 | wait ~20 minutes |
92-
|Before retry 2 | wait ~40 minutes (total probation time of 1 hour) |
93-
|Before retry 3 | wait ~1 hour 20 minutes (total probation time of 2.33 hours) |
94-
|Before retry 4 | wait ~2 hours 40 minutes (total probation time of 5 hours) |
95-
|Before retry 5 | wait ~5 hours 20 minutes (total probation time of 10.33 hours) |
96-
|Before retry 6 | wait ~10 hours 40 minutes (total probation time of 21 hours) |
97-
|Before retry 7 | wait ~15 hours (max backoff time, total probation time of 36 hours) |
93+
| Retry number | Backoff time | Total probation time in hours |
94+
|---------|---------|---------|
95+
| 1 | 20 minutes | 0.33 |
96+
| 2 | 40 minutes | 1 |
97+
| 3 | 1 hour 20 minutes | 2.33 |
98+
| 4 | 2 hours 40 minutes | 5 |
99+
| 5 | 5 hours 20 minutes | 10.33 |
100+
| 6 | 10 hours 40 minutes | 21 |
101+
| 7 | 15 hours | 36 |
98102

99-
After seven retries, the subscription status gets set to _DisabledBySystem_ if notifying the consumer fails.
103+
If the subscription receives a successful response while on probation, it gets restored to a fully enabled state, and events are published again. If all seven retries fail, the subscription state gets set to _DisabledBySystem_.
100104

101105
## FAQs
102106

103-
<!-- BEGINSECTION class="m-qanda" -->
107+
### Q: What is the payload limit of a service hook?
108+
109+
**A:** The payload limit is 2 MB. Larger payloads cause degradation in performance and reliability. As a best practice, service hooks should limit the payload to 2 MB.
110+
111+
### Q: What does the state Enabled (restricted) mean?
112+
113+
**A:** A subscription becomes restricted if too many failures occur. Being in the _Enabled (restricted)_ state is the same as being on probation.
104114

105-
### Q: What is the payload limit of a service-hook?
115+
### Q: What does the state Disabled (due to failures) mean?
106116

107-
**A:** The payload limit is 2 MB. Larger payloads cause degradation in performance and reliability. As a best practice, service-hooks should limit the payload to 2 MB or less.
117+
**A:** A subscription is automatically disabled in the following cases:
108118

109-
### Q: What does the status Enabled (restricted) mean?
119+
* A terminal failure is encountered.
120+
* A series of consecutive failures occurs over a prolonged period.
110121

111-
**A:** A subscription becomes restricted if too many failures occur. Enabled (restricted) is the same as being on probation.
122+
Notifications that result in transient failures are retried several times before being declared enduring failures. Enduring failure notifications are retried a limited number of times during [probation](#probation). If all probation retries fail, the subscription gets disabled.
112123

113-
### Q: What does the status Disabled (due to failures) mean?
124+
The following status codes provide examples of each type of failure:
114125

115-
A: A subscription is automatically disabled after a series of consecutive failures over a prolonged period or a _terminal failure_ is encountered. _Transient failures_ types are retried several times before being declared a failure. _Enduring failure_ types aren't retried. The following are examples of each type of failure.
116126
* Transient: 408 (Request Timeout), 502 (Bad Gateway), 503 (Service Unavailable), 504 (Gateway Timeout)
117127
* Terminal: 410 (Gone)
118128
* Enduring: All failures that aren't transient or terminal
119129

120-
### Q: What does the status Disabled (user left project) mean?
130+
### Q: What does the state Disabled (user left project) mean?
121131

122132
**A:** The user who created the subscription is no longer a member of the team.
123133

124134
### Q: What should I try if a service hook isn't working?
125135

126136
**A:** Check the following items:
127137

128-
- Confirm the subscription is enabled
129-
130-
- Confirm the subscription settings are correct (both event filters and actions)
131-
132-
- Look at the history, especially if there are failures
138+
* Confirm the subscription is enabled.
139+
* Confirm the subscription settings are correct. Check event filters and actions.
140+
* Look at the history, especially if there are failures.
133141

134142
### Q: Can I grant a regular project user the ability to view and manage service hook subscriptions for a project?
135143

136-
**A:** By default, only project administrators have these permissions. To grant them to other users directly, you can use the [command line tool](../organizations/security/manage-tokens-namespaces.md) or the [Security](/rest/api/azure/devops/security/) REST API.
144+
**A:** By default, only project administrators have these permissions. To grant them to other users directly, you can use the [command-line tool](../organizations/security/manage-tokens-namespaces.md) or the [Security](/rest/api/azure/devops/security/) REST API.
137145

138146
### Q: Can I programmatically create subscriptions?
139147

140-
**A:** Yes, use [REST APIs](./create-subscription.md).
141-
142-
<!-- ENDSECTION -->
148+
**A:** Yes, use [REST APIs](create-subscription.md).

0 commit comments

Comments
 (0)