Skip to content
This repository was archived by the owner on Jun 22, 2024. It is now read-only.

Commit 2eca4bb

Browse files
VietND96VietND96
committed
feat(chart): Configure fixed-sized thread pool for the Distributor in autoscaling
Configure fixed-sized thread pool for the Distributor to create new sessions based on sum of maxReplicaCount of all enabled Nodes in autoscaling Signed-off-by: Viet Nguyen Duc <[email protected]>
1 parent 97941f8 commit 2eca4bb

File tree

13 files changed

+86
-27
lines changed

13 files changed

+86
-27
lines changed

‎.github/workflows/helm-chart-test.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,10 @@ on:
1515
description: 'Test parameter for different request timeout'
1616
required: false
1717
default: '370'
18+
max-replicas-count:
19+
description: 'Test parameter for autoscaling to set maxReplicaCount'
20+
required: false
21+
default: '30'
1822
log-level:
1923
description: 'Test parameter for different log level'
2024
required: false
@@ -116,8 +120,11 @@ jobs:
116120
if: (matrix.test-strategy == 'job' || matrix.test-strategy == 'deployment') && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
117121
run: |
118122
echo "AUTOSCALING_POLL_INTERVAL=${AUTOSCALING_POLL_INTERVAL}" >> $GITHUB_ENV
123+
echo "SET_MAX_REPLICAS=${SET_MAX_REPLICAS}" >> $GITHUB_ENV
124+
echo "LOG_LEVEL=${LOG_LEVEL}" >> $GITHUB_ENV
119125
env:
120126
AUTOSCALING_POLL_INTERVAL: ${{ github.event.inputs.request-timeout || '370' }}
127+
SET_MAX_REPLICAS: ${{ github.event.inputs.max-replicas-count || '30' }}
121128
LOG_LEVEL: ${{ github.event.inputs.log-level || 'FINE' }}
122129
- name: Test Selenium Grid on Kubernetes ${{ matrix.k8s-version }} with Autoscaling ${{ matrix.test-strategy }}
123130
uses: nick-invision/retry@master

‎charts/selenium-grid/README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ This chart enables the creation of a Selenium Grid Server in Kubernetes.
1414
* [Settings common for both `job` and `deployment` scalingType](#settings-common-for-both-job-and-deployment-scalingtype)
1515
* [Settings when scalingType with `deployment`](#settings-when-scalingtype-with-deployment-)
1616
* [Settings when scalingType with `job`](#settings-when-scalingtype-with-job)
17+
* [Settings fixed-sized thread pool for the Distributor to create new sessions](#settings-fixed-sized-thread-pool-for-the-distributor-to-create-new-sessions)
1718
* [Updating Selenium-Grid release](#updating-selenium-grid-release)
1819
* [Uninstalling Selenium Grid release](#uninstalling-selenium-grid-release)
1920
* [Ingress Configuration](#ingress-configuration)
@@ -195,6 +196,16 @@ autoscaling:
195196

196197
Settings that KEDA [ScaledJob spec](https://keda.sh/docs/latest/concepts/scaling-jobs/#scaledjob-spec) supports can be set via `autoscaling.scaledJobOptions`.
197198

199+
### Settings fixed-sized thread pool for the Distributor to create new sessions
200+
201+
When enabling autoscaling, the Distributor might be under a high workload with parallelism tests, which are many requests incoming and nodes scaling up simultaneously. (Refer to: [SeleniumHQ/selenium#13723](https://github.com/SeleniumHQ/selenium/issues/13723)).
202+
203+
By default, the Distributor uses a fixed-sized thread pool with default value is `no. of available processors * 3`.
204+
205+
In autoscaling, by default, it will calculate based on `no. of node types * maxReplicaCount`. For example: `autoscaling.scaledOptions.maxReplicaCount=50`, 3 node types (`Chrome, Firefox, Edge` enabled), the value is `50 * 3 + 1 = 151` is set to environment variable `SE_NEW_SESSION_THREAD_POOL_SIZE` to adjust the Distributor config `--newsession-threadpool-size`
206+
207+
You can override the default calculation by another value via `components.distributor.newSessionThreadPoolSize` (in full distributed mode) or `hub.newSessionThreadPoolSize` (in basic mode).
208+
198209
## Updating Selenium-Grid release
199210

200211
Once you have a new chart version, you can update your selenium-grid running:

‎charts/selenium-grid/templates/_helpers.tpl‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,27 @@ Is tracing enabled
127127
{{- or .Values.tracing.enabled .Values.tracing.enabledWithExistingEndpoint | ternary "true" "" -}}
128128
{{- end -}}
129129

130+
{{/*
131+
Configure fixed-sized thread pool for the Distributor to create new sessions
132+
based on sum of maxReplicaCount of all enabled Nodes in autoscaling
133+
*/}}
134+
{{- define "seleniumGrid.autoscaling.distributor.threadPoolSize" -}}
135+
{{- $threadPoolSize := 1 -}}
136+
{{- if .Values.chromeNode.enabled -}}
137+
{{- $maxReplicaCount := default .Values.autoscaling.scaledOptions.maxReplicaCount (.Values.chromeNode.scaledOptions).maxReplicaCount -}}
138+
{{- $threadPoolSize = add $threadPoolSize $maxReplicaCount -}}
139+
{{- end -}}
140+
{{- if $.Values.firefoxNode.enabled -}}
141+
{{- $maxReplicaCount := default .Values.autoscaling.scaledOptions.maxReplicaCount (.Values.firefoxNode.scaledOptions).maxReplicaCount -}}
142+
{{- $threadPoolSize = add $threadPoolSize $maxReplicaCount -}}
143+
{{- end -}}
144+
{{- if $.Values.edgeNode.enabled -}}
145+
{{- $maxReplicaCount := default .Values.autoscaling.scaledOptions.maxReplicaCount (.Values.edgeNode.scaledOptions).maxReplicaCount -}}
146+
{{- $threadPoolSize = add $threadPoolSize $maxReplicaCount -}}
147+
{{- end -}}
148+
{{- $threadPoolSize -}}
149+
{{- end -}}
150+
130151
{{/*
131152
Common autoscaling spec template
132153
*/}}

‎charts/selenium-grid/templates/distributor-deployment.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,12 @@ spec:
5252
value: '{{ template "seleniumGrid.sessionQueue.fullname" . }}.{{ .Release.Namespace }}'
5353
- name: SE_SESSION_QUEUE_PORT
5454
value: {{ .Values.components.sessionQueue.port | quote }}
55-
{{- with .Values.components.distributor.newSessionThreadPoolSize }}
55+
{{- if .Values.components.distributor.newSessionThreadPoolSize }}
5656
- name: SE_NEW_SESSION_THREAD_POOL_SIZE
57-
value: {{ . | quote }}
57+
value: {{ .Values.components.distributor.newSessionThreadPoolSize | quote }}
58+
{{- else if (eq (include "seleniumGrid.useKEDA" $) "true") }}
59+
- name: SE_NEW_SESSION_THREAD_POOL_SIZE
60+
value: '{{ template "seleniumGrid.autoscaling.distributor.threadPoolSize" $ }}'
5861
{{- end }}
5962
{{- with .Values.components.extraEnvironmentVariables }}
6063
{{- tpl (toYaml .) $ | nindent 12 }}

‎charts/selenium-grid/templates/hub-deployment.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,9 +109,12 @@ spec:
109109
- name: SE_DISABLE_UI
110110
value: {{ .Values.hub.disableUI | quote }}
111111
{{- end }}
112-
{{- with .Values.hub.newSessionThreadPoolSize }}
112+
{{- if .Values.hub.newSessionThreadPoolSize }}
113113
- name: SE_NEW_SESSION_THREAD_POOL_SIZE
114-
value: {{ . | quote }}
114+
value: {{ .Values.hub.newSessionThreadPoolSize | quote }}
115+
{{- else if (eq (include "seleniumGrid.useKEDA" $) "true") }}
116+
- name: SE_NEW_SESSION_THREAD_POOL_SIZE
117+
value: '{{ template "seleniumGrid.autoscaling.distributor.threadPoolSize" $ }}'
115118
{{- end }}
116119
{{- with .Values.hub.extraEnvironmentVariables }}
117120
{{- tpl (toYaml .) $ | nindent 12 }}

‎charts/selenium-grid/values.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -722,7 +722,7 @@ chromeNode:
722722
url: '{{ template "seleniumGrid.graphqlURL" . }}'
723723
browserName: 'chrome'
724724
sessionBrowserName: 'chrome'
725-
platformName: 'Linux'
725+
platformName: 'linux'
726726
# browserVersion: '91.0' # Optional. Only required when supporting multiple versions of browser in your Selenium Grid.
727727
unsafeSsl: '{{ template "seleniumGrid.graphqlURL.unsafeSsl" . }}' # Optional
728728

@@ -883,7 +883,7 @@ firefoxNode:
883883
url: '{{ template "seleniumGrid.graphqlURL" . }}'
884884
browserName: 'firefox'
885885
sessionBrowserName: 'firefox'
886-
platformName: 'Linux'
886+
platformName: 'linux'
887887
unsafeSsl: '{{ template "seleniumGrid.graphqlURL.unsafeSsl" . }}' # Optional
888888

889889
# It is used to add initContainers in the same pod of the browser node.
@@ -1042,7 +1042,7 @@ edgeNode:
10421042
url: '{{ template "seleniumGrid.graphqlURL" . }}'
10431043
browserName: 'MicrosoftEdge'
10441044
sessionBrowserName: 'msedge'
1045-
platformName: 'Linux'
1045+
platformName: 'linux'
10461046
unsafeSsl: '{{ template "seleniumGrid.graphqlURL.unsafeSsl" . }}' # Optional
10471047

10481048
# It is used to add initContainers in the same pod of the browser node.

‎tests/charts/ci/DeploymentAutoscaling-values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ autoscaling:
33
scaledOptions:
44
minReplicaCount: 0
55
maxReplicaCount: 3
6-
pollingInterval: 20
6+
pollingInterval: 10
77
scaledObjectOptions:
88
cooldownPeriod: 30
99
terminationGracePeriodSeconds: 360

‎tests/charts/ci/JobAutoscaling-values.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ autoscaling:
77
strategy: default
88
scaledOptions:
99
minReplicaCount: 0
10-
maxReplicaCount: 30
11-
pollingInterval: 20
10+
maxReplicaCount: 7
11+
pollingInterval: 10
1212
# Configuration for chrome nodes
1313
chromeNode:
1414
nameOverride: my-chrome-name

‎tests/charts/ci/base-resources-values.yaml‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,11 @@ components:
1010
distributor:
1111
resources:
1212
requests:
13-
cpu: 100m
14-
memory: 256Mi
13+
cpu: "1"
14+
memory: 1Gi
1515
limits:
16-
cpu: 200m
17-
memory: 2500Mi
16+
cpu: "2"
17+
memory: 2Gi
1818
eventBus:
1919
resources:
2020
requests:
@@ -43,11 +43,11 @@ components:
4343
hub:
4444
resources:
4545
requests:
46-
cpu: 100m
47-
memory: 256Mi
46+
cpu: "1"
47+
memory: 1Gi
4848
limits:
49-
cpu: 500m
50-
memory: 2500Mi
49+
cpu: "2"
50+
memory: 2Gi
5151

5252
chromeNode:
5353
resources:

‎tests/charts/make/chart_test.sh‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,15 @@ if [ "${CHART_ENABLE_INGRESS_HOSTNAME}" = "true" ]; then
118118
if [[ ! $(cat /etc/hosts) == *"${HOSTNAME_ADDRESS}"* ]]; then
119119
sudo -- sh -c -e "echo \"$(hostname -i) ${HOSTNAME_ADDRESS}\" >> /etc/hosts"
120120
fi
121+
if [[ ! $(cat /etc/hosts) == *"alertmanager.${HOSTNAME_ADDRESS}"* ]]; then
122+
sudo -- sh -c -e "echo \"$(hostname -i) alertmanager.${HOSTNAME_ADDRESS}\" >> /etc/hosts"
123+
fi
124+
if [[ ! $(cat /etc/hosts) == *"grafana.${HOSTNAME_ADDRESS}"* ]]; then
125+
sudo -- sh -c -e "echo \"$(hostname -i) grafana.${HOSTNAME_ADDRESS}\" >> /etc/hosts"
126+
fi
127+
if [[ ! $(cat /etc/hosts) == *"pts.${HOSTNAME_ADDRESS}"* ]]; then
128+
sudo -- sh -c -e "echo \"$(hostname -i) pts.${HOSTNAME_ADDRESS}\" >> /etc/hosts"
129+
fi
121130
ping -c 2 ${HOSTNAME_ADDRESS}
122131
HELM_COMMAND_SET_IMAGES="${HELM_COMMAND_SET_IMAGES} \
123132
--set ingress.hostname=${HOSTNAME_ADDRESS} \
@@ -127,15 +136,6 @@ else
127136
HELM_COMMAND_SET_IMAGES="${HELM_COMMAND_SET_IMAGES} \
128137
--set global.K8S_PUBLIC_IP=${SELENIUM_GRID_HOST} \
129138
"
130-
if [[ ! $(cat /etc/hosts) == *"alertmanager.selenium-grid.prod"* ]]; then
131-
sudo -- sh -c -e "echo \"$(hostname -i) alertmanager.selenium-grid.prod\" >> /etc/hosts"
132-
fi
133-
if [[ ! $(cat /etc/hosts) == *"grafana.selenium-grid.prod"* ]]; then
134-
sudo -- sh -c -e "echo \"$(hostname -i) grafana.selenium-grid.prod\" >> /etc/hosts"
135-
fi
136-
if [[ ! $(cat /etc/hosts) == *"pts.selenium-grid.prod"* ]]; then
137-
sudo -- sh -c -e "echo \"$(hostname -i) pts.selenium-grid.prod\" >> /etc/hosts"
138-
fi
139139
fi
140140

141141
if [ "${CHART_ENABLE_BASIC_AUTH}" = "true" ]; then

0 commit comments

Comments
 (0)