You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/app-service/troubleshoot-intermittent-outbound-connection-errors.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,15 +31,15 @@ A major cause of these symptoms is that the application instance is not able to
31
31
* TCP Connections: There is a limit on the number of outbound connections that can be made. This is associated with the size of the worker used.
32
32
* SNAT ports: As discussed in [Outbound connections in Azure](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections), Azure uses source network address translation (SNAT) and a Load Balancer (not exposed to customers) to communicate with end points outside Azure in the public IP address space. Each instance on Azure App service is initially given a pre-allocated number of **128** SNAT ports. That limit affects opening connections to the same host and port combination. If your app creates connections to a mix of address and port combinations, you will not use up your SNAT ports. The SNAT ports are used up when you have repeated calls to the same address and port combination. Once a port has been released, the port is available for reuse as needed. The Azure Network load balancer reclaims SNAT port from closed connections only after waiting for 4 minutes.
33
33
34
-
When applications or functions rapidly open a new connection, they can quickly exhaust their pre-allocated quota of the 128 ports. They are then blocked until a new SNAT port becomes available, either through dynamically allocating additional SNAT ports, or through re-use of a reclaimed SNAT port. Applications or functions that are blocked because of this inability to create new connections will begin experiencing one or more of the issues described in the **Symptoms** section of this article.
34
+
When applications or functions rapidly open a new connection, they can quickly exhaust their pre-allocated quota of the 128 ports. They are then blocked until a new SNAT port becomes available, either through dynamically allocating additional SNAT ports, or through reuse of a reclaimed SNAT port. Applications or functions that are blocked because of this inability to create new connections will begin experiencing one or more of the issues described in the **Symptoms** section of this article.
35
35
36
36
## Avoiding the problem
37
37
38
38
Avoiding the SNAT port problem means avoiding the creation of new connections repetitively to the same host and port.
39
39
40
-
General strategies for mitigating SNAT port exhaustion are discussed in the [Problem-solving section](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections#problemsolving) of the **Outbound connections of Azure** documentation. Of these, the following are applicable to apps and functions hosted on Azure App service.
40
+
General strategies for mitigating SNAT port exhaustion are discussed in the [Problem-solving section](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections#problemsolving) of the **Outbound connections of Azure** documentation. Of these strategies, the following are applicable to apps and functions hosted on Azure App service.
41
41
42
-
### Modify the application to use connection pooling:
42
+
### Modify the application to use connection pooling
43
43
44
44
* For pooling HTTP connections, review [Pool HTTP connections with HttpClientFactory](https://docs.microsoft.com/aspnet/core/performance/performance-best-practices#pool-http-connections-with-httpclientfactory).
45
45
* For information on SQL Server connection pooling, review [SQL Server Connection Pooling (ADO.NET)](https://docs.microsoft.com/dotnet/framework/data/adonet/sql-server-connection-pooling).
@@ -97,7 +97,7 @@ Although PHP does not support connection pooling, you can try using persistent d
97
97
*[SQL Server](https://github.com/tediousjs/node-mssql#connection-pools) (NOTE: SQLAlchemy can be used with other databases besides MicrosoftSQL Server)
98
98
*[HTTP Keep-alive](https://requests.readthedocs.io/en/master/user/advanced/#keep-alive)(Keep-Alive is automatic when using sessions [session-objects](https://requests.readthedocs.io/en/master/user/advanced/#keep-alive)).
99
99
100
-
For other environments, review provider or driverspecific documents for implementing connection pooling in your applications.
100
+
For other environments, review provider or driver-specific documents for implementing connection pooling in your applications.
101
101
102
102
### Modify the application to reuse connections
103
103
@@ -109,16 +109,16 @@ For other environments, review provider or driver specific documents for impleme
109
109
110
110
### Use keepalives to reset the outbound idle timeout
111
111
112
-
* For implementing this for Node.js apps, review [My node application is making excessive outbound calls](https://docs.microsoft.com/azure/app-service/app-service-web-nodejs-best-practices-and-troubleshoot-guide#my-node-application-is-making-excessive-outbound-calls).
112
+
* For implementing keepalives for Node.js apps, review [My node application is making excessive outbound calls](https://docs.microsoft.com/azure/app-service/app-service-web-nodejs-best-practices-and-troubleshoot-guide#my-node-application-is-making-excessive-outbound-calls).
113
113
114
114
### Additional guidance specific to App Service:
115
115
116
116
* A [load test](https://docs.microsoft.com/azure/devops/test/load-test/app-service-web-app-performance-test) should simulate real world data in a steady feeding speed. Testing apps and functions under real world stress can identify and resolve SNAT port exhaustion issues ahead of time.
117
117
* Ensure that the back-end services can return responses quickly. For troubleshooting performance issues with Azure SQL database, review [Troubleshoot Azure SQL Database performance issues with Intelligent Insights](https://docs.microsoft.com/azure/sql-database/sql-database-intelligent-insights-troubleshoot-performance#recommended-troubleshooting-flow).
118
118
* Scale out the App Service plan to more instances. For more information on scaling, see [Scale an app in Azure App Service](https://docs.microsoft.com/azure/app-service/manage-scale-up). Each worker instance in an app service plan is allocated a number of SNAT ports. If you spread your usage across more instances, you might get the SNAT port usage per instance below the recommended limit of 100 outbound connections, per unique remote endpoint.
119
-
* Consider moving to [App Service Environment (ASE)](https://docs.microsoft.com/azure/app-service/environment/using-an-ase), where you are alotted a single outbound IP address, and the limits for connections and SNAT ports are much higher.
119
+
* Consider moving to [App Service Environment (ASE)](https://docs.microsoft.com/azure/app-service/environment/using-an-ase), where you are allotted a single outbound IP address, and the limits for connections and SNAT ports are much higher.
120
120
121
-
Avoiding the outbound TCP limits is easier to solve, as the limits are set by the size of your worker. You can see the limits in [Sandbox Cross VM Numerical Limits - TCP Connnections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
121
+
Avoiding the outbound TCP limits is easier to solve, as the limits are set by the size of your worker. You can see the limits in [Sandbox Cross VM Numerical Limits - TCP Connections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
@@ -130,7 +130,7 @@ To avoid outbound TCP limits, you can either increase the size of your workers,
130
130
131
131
Knowing the two types of outbound connection limits, and what your app does, should make it easier to troubleshoot. If you know that your app makes many calls to the same storage account, you might suspect a SNAT limit. If your app creates a great many calls to endpoints all over the internet, you would suspect you are reaching the VM limit.
132
132
133
-
If you do not know the application behavior well enough to determine cause quickly, there are some tools and techniques available in App Service to help with that.
133
+
If you do not know the application behavior enough to determine the cause quickly, there are some tools and techniques available in App Service to help with that determination.
134
134
135
135
### Find SNAT port allocation information
136
136
@@ -141,24 +141,24 @@ You can use [App Service Diagnostics](https://docs.microsoft.com/azure/app-servi
141
141
3. Select SNAT Port Exhaustion tile in the list of available tiles under the category. The practice is to keep it below 128.
142
142
If you do need it, you can still open a support ticket and the support engineer will get the metric from back-end for you.
143
143
144
-
Note that since SNAT port usage is not available as a metric, it is not possible to either auto-scale based on SNAT port usage, or to configure auto scale based on SNAT ports allocation metric.
144
+
Note that since SNAT port usage is not available as a metric, it is not possible to either autoscale based on SNAT port usage, or to configure auto scale based on SNAT ports allocation metric.
145
145
146
146
### TCP Connections and SNAT Ports
147
147
148
148
TCP connections and SNAT ports are not directly related. A TCP connections usage detector is included in the Diagnose and Solve Problems blade of any App Service site. Search for the phrase "TCP connections" to find it.
149
149
150
-
1. The SNAT Ports are only used for external network flows, while the total TCP Connections includes local loopback connections.
151
-
2. A SNAT port can be shared by different flows, if the flows are different in either protocol, IP address or port. The TCP Connections metric counts every TCP connection.
152
-
3. The TCP connections limit happens at the worker instance level. The Azure Network outbound load balancing doesn't use the TCP Connections metric for SNAT port limiting.
153
-
4. The TCP connections limits are described in [Sandbox Cross VM Numerical Limits - TCP Connnections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
150
+
* The SNAT Ports are only used for external network flows, while the total TCP Connections includes local loopback connections.
151
+
* A SNAT port can be shared by different flows, if the flows are different in either protocol, IP address or port. The TCP Connections metric counts every TCP connection.
152
+
* The TCP connections limit happens at the worker instance level. The Azure Network outbound load balancing doesn't use the TCP Connections metric for SNAT port limiting.
153
+
* The TCP connections limits are described in [Sandbox Cross VM Numerical Limits - TCP Connections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
|Connections|Number of connections across entire VM|1920|3968|8064|16,000|
158
158
159
159
### WebJobs and Database connections
160
160
161
-
In the case of a SNAT port exhaustion issue, where WebJobs are unable to connect to the Azure SQL database, there is no metric to show how many connections are opened by each individual web application process. To find the problematic WebJob, move several WebJobs out to another App Service plan to see if the situation improves, or if an issue remains in one of the plans. Repeat the process until you find the problematic WebJob.
161
+
If SNAT ports are exhausted, where WebJobs are unable to connect to the Azure SQL database, there is no metric to show how many connections are opened by each individual web application process. To find the problematic WebJob, move several WebJobs out to another App Service plan to see if the situation improves, or if an issue remains in one of the plans. Repeat the process until you find the problematic WebJob.
0 commit comments