Skip to content

Commit 0e73e66

Browse files
authored
Fixed errors and corrected grammar and style.
1 parent 384ec7b commit 0e73e66

File tree

1 file changed

+21
-22
lines changed

1 file changed

+21
-22
lines changed

articles/app-service/troubleshoot-intermittent-outbound-connection-errors.md

Lines changed: 21 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ ms.custom: security-recommendations
1313

1414
# Troubleshooting intermittent outbound connection errors in Azure App Service
1515

16-
This article helps you troubleshoot intermittent connection errors and related performance issues in [Azure App Service](https://docs.microsoft.com/azure/app-service/overview). This topic will provide more information on and troubleshooting methodologies for a frequent cause of this problem - exhaustion of source address network translation (SNAT) ports. If you need more help at any point in this article, you can contact the Azure experts on the [MSDN Azure and the Stack Overflow forums](https://azure.microsoft.com/support/forums/). Alternatively, you can also file an Azure support incident. Go to the [Azure Support site](https://azure.microsoft.com/support/options/) and select **Get Support**.
16+
This article helps you troubleshoot intermittent connection errors and related performance issues in [Azure App Service](https://docs.microsoft.com/azure/app-service/overview). This topic will provide more information on, and troubleshooting methodologies for, exhaustion of source address network translation (SNAT) ports. If you require more help at any point in this article, contact the Azure experts at the [MSDN Azure and the Stack Overflow forums](https://azure.microsoft.com/support/forums/). Alternatively, file an Azure support incident. Go to the [Azure Support site](https://azure.microsoft.com/support/options/) and select **Get Support**.
1717

1818
## Symptoms
1919

@@ -26,12 +26,12 @@ Applications and Functions hosted on Azure App service may exhibit one or more o
2626

2727
## Cause
2828

29-
Frequently, one of the major causes of these symptoms is that the application instance is not able to open a new connection to the external endpoint because it has reached one of the following limits:
29+
A major cause of these symptoms is that the application instance is not able to open a new connection to the external endpoint because it has reached one of the following limits:
3030

31-
* TCP Connections: There is a limit on the number of outbound connections that can be made that is associated with the size of the worker used.
32-
* SNAT ports: As discussed in [Outbound connections in Azure](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections), Azure uses source network address translation (SNAT) and a Load Balancer (not exposed to customers) to communicate with end points outside Azure in the public IP address space. Each instance on Azure App service initially gets a pre-allocated number of **128** SNAT ports. That limit affects opening connections to the same host and port combination. If your app creates connections to a mix of address and port combinations, then you will not use up your SNAT ports. The SNAT ports are used up when you have repeated calls to the same address and port combination. Once a port has been released, the port is available for reuse as needed. The Azure Network load balancer reclaims SNAT port from closed connections only after waiting for 4 minutes.
31+
* TCP Connections: There is a limit on the number of outbound connections that can be made. This is associated with the size of the worker used.
32+
* SNAT ports: As discussed in [Outbound connections in Azure](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections), Azure uses source network address translation (SNAT) and a Load Balancer (not exposed to customers) to communicate with end points outside Azure in the public IP address space. Each instance on Azure App service is initially given a pre-allocated number of **128** SNAT ports. That limit affects opening connections to the same host and port combination. If your app creates connections to a mix of address and port combinations, you will not use up your SNAT ports. The SNAT ports are used up when you have repeated calls to the same address and port combination. Once a port has been released, the port is available for reuse as needed. The Azure Network load balancer reclaims SNAT port from closed connections only after waiting for 4 minutes.
3333

34-
When applications or functions rapidly open a new connection, they can quickly exhaust their pre-allocated quota of the 128 ports and are then blocked till a new SNAT port becomes available, either through dynamically allocating additional SNAT ports or through re-use of a reclaimed SNAT port. Applications or functions that are blocked on this inability to create new connections will start experiencing one or more of the issues described in the Symptoms section of this article.
34+
When applications or functions rapidly open a new connection, they can quickly exhaust their pre-allocated quota of the 128 ports. They are then blocked until a new SNAT port becomes available, either through dynamically allocating additional SNAT ports, or through re-use of a reclaimed SNAT port. Applications or functions that are blocked because of this inability to create new connections will begin experiencing one or more of the issues described in the **Symptoms** section of this article.
3535

3636
## Avoiding the problem
3737

@@ -78,7 +78,7 @@ HTTP Connection Pooling
7878

7979
#### PHP
8080

81-
Although PHP does not support connection pooling, you can try using persistent database connections to your backend server.
81+
Although PHP does not support connection pooling, you can try using persistent database connections to your back-end server.
8282

8383
* MySQL server
8484

@@ -113,23 +113,22 @@ For other environments, review provider or driver specific documents for impleme
113113

114114
### Additional guidance specific to App Service:
115115

116-
* A [load test](https://docs.microsoft.com/azure/devops/test/load-test/app-service-web-app-performance-test) should simulate real world data in a steady feeding speed. Testing apps and functions under real world stress is a great way to identify and resolve SNAT port exhaustion issues ahead of time.
117-
* Ensure the backend services can return response quickly. For troubleshooting performance issues with Azure SQL database, review [Troubleshoot Azure SQL Database performance issues with Intelligent Insights](https://docs.microsoft.com/azure/sql-database/sql-database-intelligent-insights-troubleshoot-performance#recommended-troubleshooting-flow).
118-
* Scale out the App Service plan to more instances. For more information on scaling, see [Scale an app in Azure App Service](https://docs.microsoft.com/azure/app-service/manage-scale-up). Each worker instance in an app service plan is allocated some number of SNAT ports. If you spread your usage across more instances, you might be able to get the SNAT port usage per instance below 100 outbound connections per unique remote endpoint, which is a recommended safe limit.
119-
* Consider moving to [App Service Environment (ASE)](https://docs.microsoft.com/azure/app-service/environment/using-an-ase), where you have a single outbound IP address just for
120-
yourself and thus limits for connections and SNAT ports are much higher.
116+
* A [load test](https://docs.microsoft.com/azure/devops/test/load-test/app-service-web-app-performance-test) should simulate real world data in a steady feeding speed. Testing apps and functions under real world stress can identify and resolve SNAT port exhaustion issues ahead of time.
117+
* Ensure that the back-end services can return responses quickly. For troubleshooting performance issues with Azure SQL database, review [Troubleshoot Azure SQL Database performance issues with Intelligent Insights](https://docs.microsoft.com/azure/sql-database/sql-database-intelligent-insights-troubleshoot-performance#recommended-troubleshooting-flow).
118+
* Scale out the App Service plan to more instances. For more information on scaling, see [Scale an app in Azure App Service](https://docs.microsoft.com/azure/app-service/manage-scale-up). Each worker instance in an app service plan is allocated a number of SNAT ports. If you spread your usage across more instances, you might get the SNAT port usage per instance below the recommended limit of 100 outbound connections, per unique remote endpoint.
119+
* Consider moving to [App Service Environment (ASE)](https://docs.microsoft.com/azure/app-service/environment/using-an-ase), where you are alotted a single outbound IP address, and the limits for connections and SNAT ports are much higher.
121120

122-
Avoiding the outbound TCP limits is an easier problem to solve. The limits are set by the size of your worker. You can see the limits here in [Sandbox Cross VM Numerical Limits - TCP Connnections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
121+
Avoiding the outbound TCP limits is easier to solve, as the limits are set by the size of your worker. You can see the limits in [Sandbox Cross VM Numerical Limits - TCP Connnections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
123122

124123
|Limit name|Description|Small (A1)|Medium (A2)|Large (A3)|
125124
|---|---|---|---|---|
126125
|Connections|Number of connections across entire VM|1920|3968|8064|
127126

128-
To avoid outbound TCP limits, you can either increase the size of your workers or scale out horizontally.
127+
To avoid outbound TCP limits, you can either increase the size of your workers, or scale out horizontally.
129128

130129
## Troubleshooting
131130

132-
Knowing the two types of outbound connection limits and what your app does should make it easier to troubleshoot. If you know that your app makes many calls to the same storage account, you might suspect a SNAT limit. If your app creates a great many calls to endpoints all over the internet, you would suspect you are reaching the VM limit.
131+
Knowing the two types of outbound connection limits, and what your app does, should make it easier to troubleshoot. If you know that your app makes many calls to the same storage account, you might suspect a SNAT limit. If your app creates a great many calls to endpoints all over the internet, you would suspect you are reaching the VM limit.
133132

134133
If you do not know the application behavior well enough to determine cause quickly, there are some tools and techniques available in App Service to help with that.
135134

@@ -139,11 +138,10 @@ You can use [App Service Diagnostics](https://docs.microsoft.com/azure/app-servi
139138

140139
1. To access App Service diagnostics, navigate to your App Service web app or App Service Environment in the [Azure portal](https://portal.azure.com/). In the left navigation, select **Diagnose and solve problems**.
141140
2. Select Availability and Performance Category
142-
3. Select SNAT Port Exhaustion tile in the list of available tiles under the category.
143-
practice is to keep it below 128.
144-
If you do need it, you can still open a support ticket and the support engineer will get the metric from backend for you.
141+
3. Select SNAT Port Exhaustion tile in the list of available tiles under the category. The practice is to keep it below 128.
142+
If you do need it, you can still open a support ticket and the support engineer will get the metric from back-end for you.
145143

146-
Note: Since SNAT port usage is not available as a metric, it is not possible to auto-scale based on SNAT port usage, or configure auto scale based on SNAT ports allocation metric.
144+
Note that since SNAT port usage is not available as a metric, it is not possible to either auto-scale based on SNAT port usage, or to configure auto scale based on SNAT ports allocation metric.
147145

148146
### TCP Connections and SNAT Ports
149147

@@ -166,10 +164,11 @@ In the case of a SNAT port exhaustion issue, where WebJobs are unable to connect
166164

167165
You cannot change any Azure settings to release the used SNAT ports sooner, as all SNAT ports will be released as per the below conditions and the behavior is by design.
168166

169-
* If either server or client sends FINACK, [SNAT port will be released](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections#tcp-snat-port-release) after 240 seconds.
170-
* If an RST is seen, SNAT port will be released after 15 seconds.
171-
* If idle timeout has been reached, port is released.
167+
* If either server or client sends FINACK, the [SNAT port will be released](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections#tcp-snat-port-release) after 240 seconds.
168+
* If an RST is seen, the SNAT port will be released after 15 seconds.
169+
* If idle timeout has been reached, the port is released.
172170

173-
Additional information:
171+
## Additional information
172+
174173
* [SNAT with App Service](https://4lowtherabbit.github.io/blogs/2019/10/SNAT/)
175174
* [Troubleshoot slow app performance issues in Azure App Service](https://docs.microsoft.com/azure/app-service/troubleshoot-performance-degradation)

0 commit comments

Comments
 (0)