|
1 | 1 | ---
|
2 |
| -title: Troubleshooting client response timeouts and errors with API Management |
| 2 | +title: Troubleshooting client response time-outs and errors with API Management |
3 | 3 | description: Troubleshoot intermittent connection errors and related latency issues in API Management
|
4 | 4 | author: dlepow
|
5 | 5 | ms.topic: troubleshooting
|
6 |
| -ms.date: 12/04/2020 |
| 6 | +ms.date: 04/15/2025 |
7 | 7 | ms.author: danlep
|
8 | 8 | ms.service: azure-api-management
|
9 | 9 | ---
|
10 | 10 |
|
11 |
| -# Troubleshooting client response timeouts and errors with API Management |
| 11 | +# Troubleshooting client response time-outs and errors with API Management |
12 | 12 |
|
13 | 13 | [!INCLUDE [api-management-availability-all-tiers](../../includes/api-management-availability-all-tiers.md)]
|
14 | 14 |
|
15 |
| -This article helps you troubleshoot intermittent connection errors and related latency issues in [Azure API Management](./api-management-key-concepts.md). Specifically, this article will provide information and troubleshooting for the exhaustion of source address network translation (SNAT) ports. If you require more help, contact the Azure experts at [Azure Community Support](https://azure.microsoft.com/support/community/) or file a support request with [Azure Support](https://azure.microsoft.com/support/options/). |
| 15 | +This article helps you troubleshoot intermittent connection errors and related latency issues in [Azure API Management](./api-management-key-concepts.md). Specifically, this article provides information and troubleshooting for the exhaustion of source network address translation (SNAT) ports. If you require more help, contact the Azure experts at [Azure Community Support](https://azure.microsoft.com/support/community/) or file a support request with [Azure Support](https://azure.microsoft.com/support/options/). |
16 | 16 |
|
17 | 17 | ## Symptoms
|
18 | 18 |
|
19 |
| -Client applications calling APIs through your API Management (APIM) service may exhibit one or more of the following symptoms: |
| 19 | +Client applications calling APIs through your API Management service may exhibit one or more of the following symptoms: |
20 | 20 |
|
21 | 21 | * Intermittent HTTP 500 errors
|
22 |
| -* Timeout error messages |
| 22 | +* Time-out error messages |
23 | 23 |
|
24 | 24 | These symptoms manifest as instances of `BackendConnectionFailure` in your [Azure Monitor resource logs](/azure/azure-monitor/essentials/resource-logs).
|
25 | 25 |
|
| 26 | +In certain API Management service tiers, you may also see diagnostic information related to SNAT port exhaustion in the Azure portal on the **Diagnose and solve problems** > **SNAT Port Analysis** page for your API Management instance. |
| 27 | + |
26 | 28 | ## Cause
|
27 | 29 |
|
28 |
| -This pattern of symptoms often occurs due to network address translation (SNAT) port limits with your APIM service. |
| 30 | +This pattern of symptoms often occurs due to SNAT port limits with your API Management service. |
29 | 31 |
|
30 |
| -Whenever a client calls one of your APIM APIs, Azure API Management service opens a SNAT port to access your backend API. As discussed in [Outbound connections in Azure](../load-balancer/load-balancer-outbound-connections.md), Azure uses source network address translation (SNAT) and a Load Balancer (not exposed to customers) to communicate with end points outside Azure in the public IP address space, as well as end points internal to Azure that aren't using [Virtual Network service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md). This situation is only applicable to backend APIs exposed on public IPs. |
| 32 | +Whenever a client calls one of your API Management APIs, Azure API Management service opens a SNAT port to access your backend API. As discussed in [Outbound connections in Azure](../load-balancer/load-balancer-outbound-connections.md), Azure uses source network address translation (SNAT) and a load balancer (not exposed to customers) to communicate with endpoints outside Azure in the public IP address space, and to endpoints internal to Azure that aren't using [Virtual Network service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md). This situation is only applicable to backend APIs exposed on public IPs. |
31 | 33 |
|
32 |
| -Each instance of API Management service is initially given a pre-allocated number of SNAT ports. That limit affects opening connections to the same host and port combination. SNAT ports are used up when you have repeated calls to the same address and port combination. Once a SNAT port has been released, the port is available for reuse as needed. The Azure Network load balancer reclaims SNAT ports from closed connections only after waiting four minutes. |
| 34 | +Each instance of the API Management service is initially given a preallocated number of SNAT ports. That limit affects opening connections to the same host and port combination. SNAT ports are used up when you have repeated calls to the same address and port combination. Once a SNAT port has been released, the port is available for reuse as needed. The Azure network load balancer reclaims SNAT ports from closed connections only after waiting four minutes. |
33 | 35 |
|
34 |
| -A rapid succession of client requests to your APIs may exhaust the pre-allocated quota of SNAT ports if these ports are not closed and recycled fast enough, preventing your APIM service from processing client requests in a timely manner. |
| 36 | +A rapid succession of client requests to your APIs may exhaust the preallocated quota of SNAT ports if these ports aren't closed and recycled fast enough, preventing your API Management service from processing client requests in a timely manner. |
35 | 37 |
|
36 | 38 | ## Mitigations and solutions
|
37 | 39 |
|
38 |
| -Addressing the problem of SNAT port exhaustion first requires diagnosing and optimizing the performance of your backend services. |
| 40 | +General strategies for mitigating SNAT port exhaustion are discussed in [Troubleshooting outbound connections failures](../load-balancer/troubleshoot-outbound-connection.md) in the Azure Load Balancer documentation. Of these strategies, the following are applicable to API Management. |
| 41 | + |
| 42 | +### Enable Azure NAT Gateway |
| 43 | + |
| 44 | +For a virtual network-injected instance in the Premium tier of API Management, you can enable [Azure NAT Gateway](/azure/virtual-network/nat-gateway/nat-overview) to provide a larger number of SNAT ports (up to 64K) than are available by default in API Management. If supported in your scenario, this solution is the most effective way to avoid SNAT port exhaustion. |
| 45 | + |
| 46 | +To enable Azure NAT Gateway in the API Management instance's virtual network, set the instance's `natGatewayState` property to `enabled` by using the [API Management Service - Create Or Update](/rest/api/apimanagement/api-management-service/create-or-update#apimanagementcreateservicewithnatgatewayenabled) REST API. |
| 47 | + |
| 48 | +> [!NOTE] |
| 49 | +> * Currently, to set the `natGatewayState` property, the instance can't be in a zonal or zone-redundant configuration. |
| 50 | +> * For an instance injected into a virtual network in internal mode, the NAT gateway works only for outbound traffic to the internet. |
| 51 | +> * Azure NAT Gateway may incur extra costs. |
| 52 | +
|
| 53 | +The default idle time-out set in the NAT gateway is 4 minutes. You can change the idle time-out to a maximum of 120 minutes. For more information, see [Manage NAT Gateway](/azure/nat-gateway/manage-nat-gateway?tabs=manage-nat-portal). |
39 | 54 |
|
40 |
| -General strategies for mitigating SNAT port exhaustion are discussed in [Troubleshooting outbound connections failures](../load-balancer/troubleshoot-outbound-connection.md) from *Azure Load Balancer* documentation. Of these strategies, the following are applicable to API Management. |
| 55 | +If you're unable to use a NAT gateway for outbound connectivity, refer to the other mitigation options described in this section. |
41 | 56 |
|
42 |
| -### Scale your APIM instance |
| 57 | +### Scale your API Management instance |
43 | 58 |
|
44 |
| -Each API Management instance is allocated a number of SNAT ports, based on APIM units. You can allocate additional SNAT ports by scaling your API Management instance with additional units. For more info, see [Scale your API Management service](upgrade-and-scale.md#scale-your-api-management-instance). |
| 59 | +Each API Management instance is allocated a number of SNAT ports, based on API Management units. You can allocate more SNAT ports by scaling your API Management instance with more units. For more information, see [Scale your API Management service](upgrade-and-scale.md#scale-your-api-management-instance). |
45 | 60 |
|
46 | 61 | > [!NOTE]
|
47 | 62 | > SNAT port usage is currently not available as a metric for autoscaling API Management units.
|
48 | 63 |
|
49 | 64 | ### Use multiple IPs for your backend URLs
|
50 | 65 |
|
51 |
| -Each connection from your APIM instance to the same destination IP and destination port of your backend service will use a SNAT port, in order to maintain a distinct traffic flow. Without different SNAT ports for the return traffic from your background service, APIM would have no way to separate one response from another. |
| 66 | +Each connection from your API Management instance to the same destination IP and destination port of your backend service uses a SNAT port, in order to maintain a distinct traffic flow. Without different SNAT ports for the return traffic from your background service, API Management has no way to separate one response from another. |
52 | 67 |
|
53 | 68 | Because SNAT ports can be reused if the destination IP or destination port are different, another way to avoid SNAT port exhaustion is by using multiple IPs for your backend service URLs.
|
54 | 69 |
|
55 | 70 | For more, see [Outbound proxy Azure Load Balancer](../load-balancer/load-balancer-outbound-connections.md).
|
56 | 71 |
|
57 |
| -### Place your APIM and backend service in the same VNet |
| 72 | +### Place your API Management and backend service in the same VNet |
58 | 73 |
|
59 |
| -If your backend API is hosted on an Azure service that supports *service endpoints* such as App Service, you can avoid SNAT port exhaustion issues by placing your APIM instance and backend service in the same virtual network and exposing it through [service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md) or [private endpoints](../private-link/private-endpoint-overview.md). When you use a common VNet and place service endpoints on the integration subnet, outbound traffic from your APIM instance to those services bypasses the internet, thus avoiding SNAT port restrictions. Likewise, if you use a VNet and private endpoints, you will not have any outbound SNAT port issues to that destination. |
| 74 | +If your backend API is hosted on an Azure service that supports *service endpoints* such as App Service, you can avoid SNAT port exhaustion issues by placing your API Management instance and backend service in the same virtual network and exposing it through [service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md) or [private endpoints](../private-link/private-endpoint-overview.md). When you use a common VNet and place service endpoints on the integration subnet, outbound traffic from your API Management instance to those services bypasses the internet, thus avoiding SNAT port restrictions. Likewise, if you use a VNet and private endpoints, you will not have any outbound SNAT port issues to that destination. |
60 | 75 |
|
61 | 76 | For details, see [How to use Azure API Management with virtual networks](api-management-using-with-vnet.md) and [Integrate App Service with an Azure virtual network](../app-service/overview-vnet-integration.md).
|
62 | 77 |
|
63 |
| -### Place your APIM in a virtual network and route outbound calls to Azure Firewall |
| 78 | +### Place your API Management service in a virtual network and route outbound calls to Azure Firewall |
64 | 79 |
|
65 |
| -Similar to placing your APIM and backend services in a virtual network, you can employ Azure Firewall in a VNet with your APIM service, then route outbound APIM calls to Azure Firewall. Between APIM and Azure Firewall (being in the same VNet), no SNAT ports are required. For SNAT connections to your backend services, Azure Firewall has 64,000 available ports, a much higher amount than is allocated to APIM instances. |
| 80 | +Similar to placing your API Management and backend services in a virtual network, you can employ Azure Firewall in a VNet with your API Management service, then route outbound API Management calls to Azure Firewall. Between API Management and Azure Firewall (when placed in the same VNet), no SNAT ports are required. For SNAT connections to your backend services, Azure Firewall has 64,000 available ports, a much higher amount than is allocated to API Management instances. |
66 | 81 |
|
67 |
| -Refer to [Azure Firewall](../firewall/overview.md) documentation for more. |
| 82 | +Refer to [Azure Firewall](../firewall/overview.md) documentation for more information. |
68 | 83 |
|
69 | 84 | ### Consider response caching and other backend performance tuning
|
70 | 85 |
|
71 |
| -Another potential mitigation to consider is improving processing times for your backend APIs. One way to do this is by configuring certain APIs with response caching to reduce latency between client applications calling your API and your APIM backend load. |
| 86 | +Another potential mitigation is to improve processing times for your backend APIs. One way to do this is by configuring certain APIs with response caching to reduce latency between client applications calling your API and your API Management backend load. |
72 | 87 |
|
73 | 88 | For more, see [Add caching to improve performance in Azure API Management](api-management-howto-cache.md).
|
74 | 89 |
|
75 | 90 | ### Consider implementing access restriction policies
|
76 | 91 |
|
77 |
| -If it makes sense for your business scenario, you can implement access restriction policies for your API Management product. For example, the `rate-limit-by-key` policy can be used to prevent API usage spikes on a per key basis by limiting the call rate per a specified time period. |
| 92 | +If it makes sense for your business scenario, you can implement access restriction policies for your API Management product. For example, the [rate-limit-by-key](rate-limit-by-key-policy.md) policy can be used to prevent API usage spikes on a per key basis by limiting the call rate per a specified time period. |
78 | 93 |
|
79 |
| -See [Rate limiting and quota policies](api-management-policies.md#rate-limiting-and-quotas) for more info. |
| 94 | +See [Rate limiting and quota policies](api-management-policies.md#rate-limiting-and-quotas) for more information. |
80 | 95 |
|
81 |
| -## See also |
| 96 | +## Related content |
82 | 97 |
|
83 | 98 | * [Azure Load Balancer: Troubleshooting outbound connections failures](../load-balancer/troubleshoot-outbound-connection.md)
|
84 | 99 | * [Azure App Service: Troubleshooting intermittent outbound connection errors](../app-service/troubleshoot-intermittent-outbound-connection-errors.md)
|
0 commit comments