Merge branch 'ashmckenzie/increase-somaxconn-default' into 'master'

Increase net.core.somaxconn default to 2048

See merge request https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/7454

Merged-by: Balasankar 'Balu' C <[email protected]>
Approved-by: Stan Hu <[email protected]>
Approved-by: Balasankar 'Balu' C <[email protected]>
Co-authored-by: Ash McKenzie <[email protected]>

Author: balasankarc

doc/troubleshooting.md

@@ -351,7 +351,7 @@ You may have to repeat this process for other lines. For example, reconfigure fa
 ```plaintext
 kernel.shmall = 4194304
 kernel.sem = 250 32000 32 262
-net.core.somaxconn = 1024
+net.core.somaxconn = 2048
 kernel.shmmax = 17179869184
 ```
 
@@ -1020,3 +1020,24 @@ To resolve this issue, you have three options:
 - If you cannot allowlist by domain, add the [CloudFront IP address ranges](https://d7uri8nf7uskq.cloudfront.net/tools/list-cloudfront-ips) to your firewall settings. You must
   keep this list synced with your firewall settings because they can change.
 - Manually download the package file and upload it to your server.
+
+## Do I need to increase `net.core.somaxconn` ?
+
+The following may assist in identifying if the value of `net.core.somaxconn`
+is set too low:
+
+```shell
+$ netstat -ant | grep -c SYN_RECV
+4
+```
+
+The return value from `netstat -ant | grep -c SYN_RECV` is the number of connections
+waiting to be established. If the value is greater than `net.core.somaxconn`:
+
+```shell
+$ sysctl net.core.somaxconn
+net.core.somaxconn = 1024
+```
+
+You may experience timeouts or HTTP 502 errors and is recommended to increase this
+value by updating the `puma['somaxconn']` variable in your `gitlab.rb`.

files/gitlab-config-template/gitlab.rb.template

@@ -1228,7 +1228,7 @@ external_url 'GENERATED_EXTERNAL_URL'
 # puma['listen'] = '127.0.0.1'
 # puma['port'] = 8080
 # puma['socket'] = '/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
-# puma['somaxconn'] = 1024
+# puma['somaxconn'] = 2048
 
 ### SSL settings
 # puma['ssl_listen'] = nil

files/gitlab-cookbooks/gitlab/attributes/default.rb

@@ -678,7 +678,7 @@
 default['gitlab']['puma']['prometheus_scrape_tls_server_name'] = nil
 default['gitlab']['puma']['prometheus_scrape_tls_skip_verification'] = false
 
-default['gitlab']['puma']['somaxconn'] = 1024
+default['gitlab']['puma']['somaxconn'] = 2048
 # Path to the puma server Process ID file
 # defaults to /opt/gitlab/var/puma/puma.pid. The install-dir path is set at build time
 default['gitlab']['puma']['pidfile'] = "#{node['package']['install-dir']}/var/puma/puma.pid"

spec/chef/cookbooks/gitlab/recipes/puma_spec.rb

@@ -87,7 +87,7 @@
     end
 
     it 'creates sysctl files' do
-      expect(chef_run).to create_gitlab_sysctl('net.core.somaxconn').with_value(1024)
+      expect(chef_run).to create_gitlab_sysctl('net.core.somaxconn').with_value(2048)
     end
   end