|
1 | 1 | --- |
2 | 2 | ms.subservice: azure-devops-service-hooks |
3 | | -ms.topic: conceptual |
4 | | -title: Troubleshoot your service hooks integrations | Azure DevOps Services |
5 | | -description: Troubleshoot problems with the services you've integrated with your organization. |
| 3 | +ms.topic: troubleshooting-general |
| 4 | +title: Troubleshoot Service Hook Integrations |
| 5 | +description: Find out how to access the history of service hook subscriptions in Azure DevOps. Get information about HTTP response failures that affect subscription states. |
6 | 6 | ms.assetid: dcf00653-24c5-4ab6-b9e8-19ec098bbb66 |
7 | 7 | ms.custom: engagement-fy23 |
8 | 8 | ms.author: chcomley |
9 | 9 | author: chcomley |
10 | 10 | monikerRange: '<= azure-devops' |
11 | | -ms.date: 12/02/2022 |
| 11 | +ms.date: 07/03/2025 |
| 12 | +# customer intent: As a team member, I want to become familiar with service hook failure types and find out how to access the history of subscriptions so that I can troubleshoot problems with service hooks in Azure DevOps. |
12 | 13 | --- |
13 | 14 |
|
14 | 15 | # Troubleshoot service hooks |
15 | 16 |
|
16 | | -[!INCLUDE [version-lt-eq-azure-devops](../includes/version-lt-eq-azure-devops.md)] |
| 17 | +[!INCLUDE [Azure DevOps Services | Azure DevOps Server 2022 | Azure DevOps Server 2020](../includes/version-gt-eq-2020.md)] |
17 | 18 |
|
18 | | -Use this article for general troubleshooting guidance and answers to frequently asked questions (FAQs). |
| 19 | +This article offers general troubleshooting guidance for Azure DevOps service hooks. It also provides answers to frequently asked questions (FAQs). |
19 | 20 |
|
20 | 21 | ## View activity and debug problems |
21 | 22 |
|
22 | | -The **Service Hooks** page in the web access admin shows your recent activity (last 14 days) |
23 | | -for each subscription, and whether a subscription is enabled, disabled, or restricted. |
| 23 | +The **Service Hooks** page in the web access admin summarizes activity from the last seven days for each subscription. The page also shows whether each subscription is enabled, disabled, or restricted. |
24 | 24 |
|
25 | | -You can access detailed history about a subscription including detailed request/response data, which is useful for debugging a problematic service or subscription. |
26 | | - |
27 | | -::: moniker range="<=azure-devops" |
| 25 | +For each subscription, you can access a detailed history that includes the complete request and response data for each event. This information can help you debug a problematic service or subscription. |
28 | 26 |
|
29 | 27 | 1. To view the activity and status of your subscriptions, go to the **Service Hooks** page. |
30 | 28 |
|
31 | | - :::image type="content" source="media/troubleshoot/devops-service-hooks.PNG" alt-text="Screenshot showing view of activity and status of subscriptions."::: |
| 29 | + :::image type="content" source="media/troubleshoot/azure-devops-service-hooks.png" alt-text="Screenshot of the Service Hooks page showing the state, status, and other data for subscriptions. Project settings and Service hooks are highlighted." lightbox="media/troubleshoot/azure-devops-service-hooks.png"::: |
32 | 30 |
|
33 | 31 | 1. To view detailed activity for a subscription, including full request, response, and event payload data, select a subscription in the table, and then select **History**. |
34 | 32 |
|
35 | | - :::image type="content" source="media/troubleshoot/detailed-activity.png" alt-text="Screenshot showing detailed view of activity for a subscription."::: |
36 | | - |
37 | | -::: moniker-end |
| 33 | + :::image type="content" source="media/troubleshoot/detailed-activity.png" alt-text="Screenshot that shows the history of a subscription, with detailed information for a failed event, such as the status, message, and error message." lightbox="media/troubleshoot/detailed-activity.png"::: |
38 | 34 |
|
39 | 35 | ## Subscription failures and probation (restricted) |
40 | 36 |
|
| 37 | +If the HTTP response to a notification request indicates an error, the severity of the failure determines how Azure DevOps responds. Certain types of failures can disable subscriptions or put them on probation. |
| 38 | + |
41 | 39 | ### Failure types |
42 | | -Failures from a Service Hooks notification are grouped into the following categories: |
43 | 40 |
|
44 | | -* Terminal Failures |
45 | | -* Transient Failures |
46 | | -* Enduring Failures |
| 41 | +Failures from service hook notifications are grouped into the following categories: |
| 42 | + |
| 43 | +* Terminal failures |
| 44 | +* Transient failures |
| 45 | +* Enduring failures |
47 | 46 |
|
48 | | -#### Terminal Failures |
| 47 | +The error code from the HTTP response determines how Azure DevOps categorizes the failure. |
49 | 48 |
|
50 | | -The only Terminal Failure is HTTP Status Code 410 (Gone). When a subscription sees a Terminal Failure, it's automatically disabled no matter its prior status. |
| 49 | +#### Terminal failures |
51 | 50 |
|
52 | | -#### Transient Failures |
| 51 | +The only HTTP status code that's categorized as a terminal failure is 410 (Gone). |
| 52 | + |
| 53 | +When a terminal failure occurs in a subscription, the subscription is automatically disabled no matter its prior state. |
| 54 | + |
| 55 | +#### Transient failures |
| 56 | + |
| 57 | +HTTP responses with the following status codes are categorized as transient failures: |
53 | 58 |
|
54 | | -When a subscription sees a Transient Failure, it attempts to resend the notification up to eight times, with an increasing delay between each attempt. Transient Failures include the following codes: |
55 | 59 | * 408 (Request Timeout) |
56 | 60 | * 502 (Bad Gateway) |
57 | 61 | * 503 (Service Unavailable) |
58 | 62 | * 504 (Gateway Timeout) |
59 | 63 |
|
60 | | -#### Sequence of retries for transient failures |
| 64 | +When a transient failure occurs in a subscription, Azure DevOps attempts to resend the notification up to eight times, with an increasing delay between each attempt. |
61 | 65 |
|
62 | | -|Retry # |Wait time | |
63 | | -|---------|---------| |
64 | | -|Before retry 1 | wait ~1 second | |
65 | | -|Before retry 2 | wait ~2 seconds (total delay of 3 seconds) | |
66 | | -|Before retry 3 | wait ~4 seconds (total delay of 7 seconds) | |
67 | | -|Before retry 4 | wait ~8 seconds (total delay of 15 seconds) | |
68 | | -|Before retry 5 | wait ~16 seconds (total delay of 31 seconds) | |
69 | | -|Before retry 6 | wait ~32 seconds (total delay of 63 seconds) | |
70 | | -|Before retry 7 | wait ~60 seconds (max backoff time, total delay of 123 seconds) | |
71 | | -|Before retry 8 | wait ~60 seconds (max backoff time, total delay of 183 seconds)| |
| 66 | +The following table lists information about retries that are attempted after a transient failure occurs. Included is the approximate _backoff_ time, or the time to wait before attempting to resend a notification. The maximum backoff time is 60 seconds. The table also shows the total delay for each retry. |
72 | 67 |
|
73 | | -If the notification exhausts all of its retries and continues to see a Transient Failure for each attempt, the subscription stops trying to send the notification, and treats the notification as if it saw an Enduring Failure. |
| 68 | +| Retry number | Backoff time in seconds | Total delay in seconds | |
| 69 | +|---------|---------|---------| |
| 70 | +| 1 | 1 | 1 | |
| 71 | +| 2 | 2 | 3 | |
| 72 | +| 3 | 4 | 7 | |
| 73 | +| 4 | 8 | 15 | |
| 74 | +| 5 | 16 | 31 | |
| 75 | +| 6 | 32 | 63 | |
| 76 | +| 7 | 60 | 123 | |
| 77 | +| 8 | 60 | 183 | |
74 | 78 |
|
75 | | -#### Enduring Failures |
| 79 | +If all retries for a notification are exhausted and each attempt results in a transient failure, the notification is no longer sent. Instead, it's categorized as an enduring failure. |
76 | 80 |
|
77 | | -Enduring Failures include all other HTTP failure codes, for example: 404 (Not Found), 500 (Internal Server Error), and so on. |
| 81 | +#### Enduring failures |
78 | 82 |
|
79 | | -When a subscription sees an Enduring Failure, it's placed on [probation](#probation). |
| 83 | +All other HTTP failure codes, for example, 404 (Not Found) and 500 (Internal Server Error), result in enduring failures. |
80 | 84 |
|
81 | | -### Probation |
| 85 | +When an enduring failure occurs in a subscription, the subscription is placed on [probation](#probation). |
82 | 86 |
|
83 | | -While on probation, a subscription is limited in the number of notifications it can send. If the subscription continues to hit Enduring Failures, then it gets increasingly limited, and eventually disabled. If the subscription receives a successful response while on probation, it gets restored to a fully enabled state. |
| 87 | +### Probation |
84 | 88 |
|
85 | | -#### Sequence of seven maximum retries while subscription is on probation |
| 89 | +When a subscription is on probation, any new events are lost. The system makes a limited number of attempts to resend the failed notification. |
86 | 90 |
|
87 | | -When a subscription is in probation, any new events are lost. Once a retry is successful, the subscription is enabled, and events are published again. |
| 91 | +The following table lists approximate backoff times and total probation times for retries that are attempted during probation. At most seven retries are attempted, and the maximum backoff time for a probation retry is 15 hours. |
88 | 92 |
|
89 | | -|Retry # |Wait time | |
90 | | -|---------|---------| |
91 | | -|Before retry 1 | wait ~20 minutes | |
92 | | -|Before retry 2 | wait ~40 minutes (total probation time of 1 hour) | |
93 | | -|Before retry 3 | wait ~1 hour 20 minutes (total probation time of 2.33 hours) | |
94 | | -|Before retry 4 | wait ~2 hours 40 minutes (total probation time of 5 hours) | |
95 | | -|Before retry 5 | wait ~5 hours 20 minutes (total probation time of 10.33 hours) | |
96 | | -|Before retry 6 | wait ~10 hours 40 minutes (total probation time of 21 hours) | |
97 | | -|Before retry 7 | wait ~15 hours (max backoff time, total probation time of 36 hours) | |
| 93 | +| Retry number | Backoff time | Total probation time in hours | |
| 94 | +|---------|---------|---------| |
| 95 | +| 1 | 20 minutes | 0.33 | |
| 96 | +| 2 | 40 minutes | 1 | |
| 97 | +| 3 | 1 hour 20 minutes | 2.33 | |
| 98 | +| 4 | 2 hours 40 minutes | 5 | |
| 99 | +| 5 | 5 hours 20 minutes | 10.33 | |
| 100 | +| 6 | 10 hours 40 minutes | 21 | |
| 101 | +| 7 | 15 hours | 36 | |
98 | 102 |
|
99 | | -After seven retries, the subscription status gets set to _DisabledBySystem_ if notifying the consumer fails. |
| 103 | +If the subscription receives a successful response while on probation, it gets restored to a fully enabled state, and events are published again. If all seven retries fail, the subscription state gets set to _DisabledBySystem_. |
100 | 104 |
|
101 | 105 | ## FAQs |
102 | 106 |
|
103 | | -<!-- BEGINSECTION class="m-qanda" --> |
| 107 | +### Q: What is the payload limit of a service hook? |
| 108 | + |
| 109 | +**A:** The payload limit is 2 MB. Larger payloads cause degradation in performance and reliability. As a best practice, service hooks should limit the payload to 2 MB. |
| 110 | + |
| 111 | +### Q: What does the state Enabled (restricted) mean? |
| 112 | + |
| 113 | +**A:** A subscription becomes restricted if too many failures occur. Being in the _Enabled (restricted)_ state is the same as being on probation. |
104 | 114 |
|
105 | | -### Q: What is the payload limit of a service-hook? |
| 115 | +### Q: What does the state Disabled (due to failures) mean? |
106 | 116 |
|
107 | | -**A:** The payload limit is 2 MB. Larger payloads cause degradation in performance and reliability. As a best practice, service-hooks should limit the payload to 2 MB or less. |
| 117 | +**A:** A subscription is automatically disabled in the following cases: |
108 | 118 |
|
109 | | -### Q: What does the status Enabled (restricted) mean? |
| 119 | +* A terminal failure is encountered. |
| 120 | +* A series of consecutive failures occurs over a prolonged period. |
110 | 121 |
|
111 | | -**A:** A subscription becomes restricted if too many failures occur. Enabled (restricted) is the same as being on probation. |
| 122 | +Notifications that result in transient failures are retried several times before being declared enduring failures. Enduring failure notifications are retried a limited number of times during [probation](#probation). If all probation retries fail, the subscription gets disabled. |
112 | 123 |
|
113 | | -### Q: What does the status Disabled (due to failures) mean? |
| 124 | +The following status codes provide examples of each type of failure: |
114 | 125 |
|
115 | | -A: A subscription is automatically disabled after a series of consecutive failures over a prolonged period or a _terminal failure_ is encountered. _Transient failures_ types are retried several times before being declared a failure. _Enduring failure_ types aren't retried. The following are examples of each type of failure. |
116 | 126 | * Transient: 408 (Request Timeout), 502 (Bad Gateway), 503 (Service Unavailable), 504 (Gateway Timeout) |
117 | 127 | * Terminal: 410 (Gone) |
118 | 128 | * Enduring: All failures that aren't transient or terminal |
119 | 129 |
|
120 | | -### Q: What does the status Disabled (user left project) mean? |
| 130 | +### Q: What does the state Disabled (user left project) mean? |
121 | 131 |
|
122 | 132 | **A:** The user who created the subscription is no longer a member of the team. |
123 | 133 |
|
124 | 134 | ### Q: What should I try if a service hook isn't working? |
125 | 135 |
|
126 | 136 | **A:** Check the following items: |
127 | 137 |
|
128 | | -- Confirm the subscription is enabled |
129 | | - |
130 | | -- Confirm the subscription settings are correct (both event filters and actions) |
131 | | - |
132 | | -- Look at the history, especially if there are failures |
| 138 | +* Confirm the subscription is enabled. |
| 139 | +* Confirm the subscription settings are correct. Check event filters and actions. |
| 140 | +* Look at the history, especially if there are failures. |
133 | 141 |
|
134 | 142 | ### Q: Can I grant a regular project user the ability to view and manage service hook subscriptions for a project? |
135 | 143 |
|
136 | | -**A:** By default, only project administrators have these permissions. To grant them to other users directly, you can use the [command line tool](../organizations/security/manage-tokens-namespaces.md) or the [Security](/rest/api/azure/devops/security/) REST API. |
| 144 | +**A:** By default, only project administrators have these permissions. To grant them to other users directly, you can use the [command-line tool](../organizations/security/manage-tokens-namespaces.md) or the [Security](/rest/api/azure/devops/security/) REST API. |
137 | 145 |
|
138 | 146 | ### Q: Can I programmatically create subscriptions? |
139 | 147 |
|
140 | | -**A:** Yes, use [REST APIs](./create-subscription.md). |
141 | | - |
142 | | -<!-- ENDSECTION --> |
| 148 | +**A:** Yes, use [REST APIs](create-subscription.md). |
0 commit comments