Skip to content

Commit f8431d6

Browse files
authored
Fixed errors
1 parent f417c9c commit f8431d6

File tree

1 file changed

+14
-14
lines changed

1 file changed

+14
-14
lines changed

articles/app-service/troubleshoot-intermittent-outbound-connection-errors.md

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,15 +31,15 @@ A major cause of these symptoms is that the application instance is not able to
3131
* TCP Connections: There is a limit on the number of outbound connections that can be made. This is associated with the size of the worker used.
3232
* SNAT ports: As discussed in [Outbound connections in Azure](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections), Azure uses source network address translation (SNAT) and a Load Balancer (not exposed to customers) to communicate with end points outside Azure in the public IP address space. Each instance on Azure App service is initially given a pre-allocated number of **128** SNAT ports. That limit affects opening connections to the same host and port combination. If your app creates connections to a mix of address and port combinations, you will not use up your SNAT ports. The SNAT ports are used up when you have repeated calls to the same address and port combination. Once a port has been released, the port is available for reuse as needed. The Azure Network load balancer reclaims SNAT port from closed connections only after waiting for 4 minutes.
3333

34-
When applications or functions rapidly open a new connection, they can quickly exhaust their pre-allocated quota of the 128 ports. They are then blocked until a new SNAT port becomes available, either through dynamically allocating additional SNAT ports, or through re-use of a reclaimed SNAT port. Applications or functions that are blocked because of this inability to create new connections will begin experiencing one or more of the issues described in the **Symptoms** section of this article.
34+
When applications or functions rapidly open a new connection, they can quickly exhaust their pre-allocated quota of the 128 ports. They are then blocked until a new SNAT port becomes available, either through dynamically allocating additional SNAT ports, or through reuse of a reclaimed SNAT port. Applications or functions that are blocked because of this inability to create new connections will begin experiencing one or more of the issues described in the **Symptoms** section of this article.
3535

3636
## Avoiding the problem
3737

3838
Avoiding the SNAT port problem means avoiding the creation of new connections repetitively to the same host and port.
3939

40-
General strategies for mitigating SNAT port exhaustion are discussed in the [Problem-solving section](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections#problemsolving) of the **Outbound connections of Azure** documentation. Of these, the following are applicable to apps and functions hosted on Azure App service.
40+
General strategies for mitigating SNAT port exhaustion are discussed in the [Problem-solving section](https://docs.microsoft.com/azure/load-balancer/load-balancer-outbound-connections#problemsolving) of the **Outbound connections of Azure** documentation. Of these strategies, the following are applicable to apps and functions hosted on Azure App service.
4141

42-
### Modify the application to use connection pooling:
42+
### Modify the application to use connection pooling
4343

4444
* For pooling HTTP connections, review [Pool HTTP connections with HttpClientFactory](https://docs.microsoft.com/aspnet/core/performance/performance-best-practices#pool-http-connections-with-httpclientfactory).
4545
* For information on SQL Server connection pooling, review [SQL Server Connection Pooling (ADO.NET)](https://docs.microsoft.com/dotnet/framework/data/adonet/sql-server-connection-pooling).
@@ -97,7 +97,7 @@ Although PHP does not support connection pooling, you can try using persistent d
9797
* [SQL Server](https://github.com/tediousjs/node-mssql#connection-pools) (NOTE: SQLAlchemy can be used with other databases besides MicrosoftSQL Server)
9898
* [HTTP Keep-alive](https://requests.readthedocs.io/en/master/user/advanced/#keep-alive)(Keep-Alive is automatic when using sessions [session-objects](https://requests.readthedocs.io/en/master/user/advanced/#keep-alive)).
9999

100-
For other environments, review provider or driver specific documents for implementing connection pooling in your applications.
100+
For other environments, review provider or driver-specific documents for implementing connection pooling in your applications.
101101

102102
### Modify the application to reuse connections
103103

@@ -109,16 +109,16 @@ For other environments, review provider or driver specific documents for impleme
109109

110110
### Use keepalives to reset the outbound idle timeout
111111

112-
* For implementing this for Node.js apps, review [My node application is making excessive outbound calls](https://docs.microsoft.com/azure/app-service/app-service-web-nodejs-best-practices-and-troubleshoot-guide#my-node-application-is-making-excessive-outbound-calls).
112+
* For implementing keepalives for Node.js apps, review [My node application is making excessive outbound calls](https://docs.microsoft.com/azure/app-service/app-service-web-nodejs-best-practices-and-troubleshoot-guide#my-node-application-is-making-excessive-outbound-calls).
113113

114114
### Additional guidance specific to App Service:
115115

116116
* A [load test](https://docs.microsoft.com/azure/devops/test/load-test/app-service-web-app-performance-test) should simulate real world data in a steady feeding speed. Testing apps and functions under real world stress can identify and resolve SNAT port exhaustion issues ahead of time.
117117
* Ensure that the back-end services can return responses quickly. For troubleshooting performance issues with Azure SQL database, review [Troubleshoot Azure SQL Database performance issues with Intelligent Insights](https://docs.microsoft.com/azure/sql-database/sql-database-intelligent-insights-troubleshoot-performance#recommended-troubleshooting-flow).
118118
* Scale out the App Service plan to more instances. For more information on scaling, see [Scale an app in Azure App Service](https://docs.microsoft.com/azure/app-service/manage-scale-up). Each worker instance in an app service plan is allocated a number of SNAT ports. If you spread your usage across more instances, you might get the SNAT port usage per instance below the recommended limit of 100 outbound connections, per unique remote endpoint.
119-
* Consider moving to [App Service Environment (ASE)](https://docs.microsoft.com/azure/app-service/environment/using-an-ase), where you are alotted a single outbound IP address, and the limits for connections and SNAT ports are much higher.
119+
* Consider moving to [App Service Environment (ASE)](https://docs.microsoft.com/azure/app-service/environment/using-an-ase), where you are allotted a single outbound IP address, and the limits for connections and SNAT ports are much higher.
120120

121-
Avoiding the outbound TCP limits is easier to solve, as the limits are set by the size of your worker. You can see the limits in [Sandbox Cross VM Numerical Limits - TCP Connnections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
121+
Avoiding the outbound TCP limits is easier to solve, as the limits are set by the size of your worker. You can see the limits in [Sandbox Cross VM Numerical Limits - TCP Connections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
122122

123123
|Limit name|Description|Small (A1)|Medium (A2)|Large (A3)|Isolated tier (ASE)|
124124
|---|---|---|---|---|---|
@@ -130,7 +130,7 @@ To avoid outbound TCP limits, you can either increase the size of your workers,
130130

131131
Knowing the two types of outbound connection limits, and what your app does, should make it easier to troubleshoot. If you know that your app makes many calls to the same storage account, you might suspect a SNAT limit. If your app creates a great many calls to endpoints all over the internet, you would suspect you are reaching the VM limit.
132132

133-
If you do not know the application behavior well enough to determine cause quickly, there are some tools and techniques available in App Service to help with that.
133+
If you do not know the application behavior enough to determine the cause quickly, there are some tools and techniques available in App Service to help with that determination.
134134

135135
### Find SNAT port allocation information
136136

@@ -141,24 +141,24 @@ You can use [App Service Diagnostics](https://docs.microsoft.com/azure/app-servi
141141
3. Select SNAT Port Exhaustion tile in the list of available tiles under the category. The practice is to keep it below 128.
142142
If you do need it, you can still open a support ticket and the support engineer will get the metric from back-end for you.
143143

144-
Note that since SNAT port usage is not available as a metric, it is not possible to either auto-scale based on SNAT port usage, or to configure auto scale based on SNAT ports allocation metric.
144+
Note that since SNAT port usage is not available as a metric, it is not possible to either autoscale based on SNAT port usage, or to configure auto scale based on SNAT ports allocation metric.
145145

146146
### TCP Connections and SNAT Ports
147147

148148
TCP connections and SNAT ports are not directly related. A TCP connections usage detector is included in the Diagnose and Solve Problems blade of any App Service site. Search for the phrase "TCP connections" to find it.
149149

150-
1. The SNAT Ports are only used for external network flows, while the total TCP Connections includes local loopback connections.
151-
2. A SNAT port can be shared by different flows, if the flows are different in either protocol, IP address or port. The TCP Connections metric counts every TCP connection.
152-
3. The TCP connections limit happens at the worker instance level. The Azure Network outbound load balancing doesn't use the TCP Connections metric for SNAT port limiting.
153-
4. The TCP connections limits are described in [Sandbox Cross VM Numerical Limits - TCP Connnections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
150+
* The SNAT Ports are only used for external network flows, while the total TCP Connections includes local loopback connections.
151+
* A SNAT port can be shared by different flows, if the flows are different in either protocol, IP address or port. The TCP Connections metric counts every TCP connection.
152+
* The TCP connections limit happens at the worker instance level. The Azure Network outbound load balancing doesn't use the TCP Connections metric for SNAT port limiting.
153+
* The TCP connections limits are described in [Sandbox Cross VM Numerical Limits - TCP Connections](https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#cross-vm-numerical-limits)
154154

155155
|Limit name|Description|Small (A1)|Medium (A2)|Large (A3)|Isolated tier (ASE)|
156156
|---|---|---|---|---|---|
157157
|Connections|Number of connections across entire VM|1920|3968|8064|16,000|
158158

159159
### WebJobs and Database connections
160160

161-
In the case of a SNAT port exhaustion issue, where WebJobs are unable to connect to the Azure SQL database, there is no metric to show how many connections are opened by each individual web application process. To find the problematic WebJob, move several WebJobs out to another App Service plan to see if the situation improves, or if an issue remains in one of the plans. Repeat the process until you find the problematic WebJob.
161+
If SNAT ports are exhausted, where WebJobs are unable to connect to the Azure SQL database, there is no metric to show how many connections are opened by each individual web application process. To find the problematic WebJob, move several WebJobs out to another App Service plan to see if the situation improves, or if an issue remains in one of the plans. Repeat the process until you find the problematic WebJob.
162162

163163
### Using SNAT ports sooner
164164

0 commit comments

Comments
 (0)