Skip to content

Commit b768f28

Browse files
sandesvitorsandesvitor
committed
Merged master
2 parents 9d67d2d + 181f448 commit b768f28

File tree

5 files changed

+230
-9
lines changed

5 files changed

+230
-9
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
* Prefix template helpers with chart name and format helper comments as template comments. (#272) (by @cognifloyd)
66
* New feature: Add `extra_volumes` to all python-based st2 deployments. This can facilitate changing log levels by loading logging conf file(s) from a custom ConfigMap. (#276) (by @cognifloyd)
77
* Initialize basic unittest infrastructure using `helm-unittest`. Added tests for labels, custom annotations, SecurityContext, pullSecrets, pullPolicy, Resources, nodeSelector, tolerations, affinity, dnsPolicy, dnsConfig, ServiceAccount attach, postStartScript, and both sensor-modes. (#284, #288)
8+
* Allow partitioning sensors using the hash_range strategy instead of one sensor per pod. (#218) (by @cognifloyd)
89
* New feature to include possibility for external services in st2api, st2stream and st2auth, setting default value for this services as `ClusterIP` and `hostname: ""`. (by @sandesvitor)
910

1011
## v0.80.0

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,11 @@ which in our case is `Redis`.
123123
st2sensorcontainer manages StackStorm sensors: It starts, stops and restarts them as subprocesses.
124124
By default, deployment is configured with `1` replica containing all the sensors.
125125

126+
You can increase the number of `st2sensorcontainer` pods by increasing the number of deployments.
127+
The replicas count is still only `1` per deployment, but the sensors are distributed between the deployments using
128+
[Sensor Hash Range Partitioning](https://docs.stackstorm.com/reference/sensor_partitioning.html#hash).
129+
The hash ranges are calculated automatically based on the number of deployments.
130+
126131
st2sensorcontainer also supports a more Docker-friendly single-sensor-per-container mode as a way of
127132
[Sensor Partitioning](https://docs.stackstorm.com/latest/reference/sensor_partitioning.html). This
128133
distributes the computing load between many pods and relies on K8s failover/reschedule mechanisms,

‎templates/deployments.yaml‎

Lines changed: 57 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1164,13 +1164,29 @@ spec:
11641164
tolerations: {{- toYaml . | nindent 8 }}
11651165
{{- end }}
11661166

1167+
{{- $sensor_deployments_count := max 1 ($.Values.st2sensorcontainer.deployments|int) | int }}
11671168
{{- $one_sensor_per_pod := not ($.Values.st2.packs.sensors | empty) }}
1168-
{{- range ($one_sensor_per_pod | ternary ($.Values.st2.packs.sensors) (until 1)) }}
1169-
{{- $sensor := omit $.Values.st2sensorcontainer "name" "ref" "postStartScript" }}
1169+
{{- $some_sensors_per_pod := gt $sensor_deployments_count 1 }}
1170+
{{- if and $one_sensor_per_pod $some_sensors_per_pod }}
1171+
{{- fail "The sensor values are ambiguous. To use one-sensor-per-pod, use `st2.packs.sensors`. To use multiple-sensors-per-pod, use `st2sensorcontainer.deployments > 1`. Do not use both!" }}
1172+
{{- end }}
1173+
{{- range ($one_sensor_per_pod | ternary ($.Values.st2.packs.sensors) (until $sensor_deployments_count)) }}
1174+
{{- $sensor := omit $.Values.st2sensorcontainer "name" "ref" "deployments" "postStartScript" }}
11701175
{{- if $one_sensor_per_pod }}
1176+
{{/* . is an entry in st2.packs.sensors */}}
11711177
{{- range $key, $val := . }}
11721178
{{- $_ := set $sensor $key $val }}
11731179
{{- end }}
1180+
{{- else if $some_sensors_per_pod }}
1181+
{{/* . is an index number of deployments (starting from 0) */}}
1182+
{{- $max_hash := 4294967296 }}
1183+
{{- $sensor_range_size := div (add1 $max_hash) $sensor_deployments_count }}
1184+
{{- $sensor_range_start := mul . $sensor_range_size }}
1185+
{{- $sensor_range_end := add $sensor_range_start $sensor_range_size -1 }}
1186+
{{/* deal with integer math rounding errors by moving the last range_end to the max_hash */}}
1187+
{{- $sensor_range_end := eq (add1 .) $sensor_deployments_count | ternary $max_hash $sensor_range_end }}
1188+
{{- $_ := set $sensor "hash_ranges" (print $sensor_range_start ".." $sensor_range_end) }}
1189+
{{- $_ := set $sensor "name" (. | toString) }}
11741190
{{- end }}
11751191
{{- $name := print "st2sensorcontainer" (include "stackstorm-ha.hyphenPrefix" $sensor.name) }}
11761192
---
@@ -1196,8 +1212,8 @@ spec:
11961212
# https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer
11971213
# It is possible to run st2sensorcontainer(s) in one of these modes:
11981214
# (1) run all sensors in one pod (1 deployment with 1 pod, the default); or
1199-
# (2) run one sensor per pod using st2.packs.sensors.
1200-
# It is possible to run st2sensorcontainer in HA mode by running one process on each compute instance.
1215+
# (2) run multiple sensors per pod (2+ deployments with 1 pod each) using hash range partitions; or
1216+
# (3) run one sensor per pod using st2.packs.sensors.
12011217
# Each sensor node needs to be provided with proper partition information to share work with other sensor nodes
12021218
# so that the same sensor does not run on different nodes. See: https://docs.stackstorm.com/reference/sensor_partitioning.html
12031219
replicas: 1
@@ -1219,6 +1235,9 @@ spec:
12191235
{{- end }}
12201236
{{- if $one_sensor_per_pod }}
12211237
stackstorm/sensor-mode: one-sensor-per-pod
1238+
{{- else if $some_sensors_per_pod }}
1239+
stackstorm/sensor-mode: multiple-sensors-per-pod
1240+
stackstorm/sensor-hash-ranges: {{ $sensor.hash_ranges | quote }}
12221241
{{- else }}
12231242
stackstorm/sensor-mode: all-sensors-in-one-pod
12241243
{{- end }}
@@ -1234,6 +1253,23 @@ spec:
12341253
{{- include "stackstorm-ha.packs-pullSecrets" $ | nindent 6 }}
12351254
{{- end }}
12361255
initContainers:
1256+
{{- if $some_sensors_per_pod }}
1257+
- name: {{ $name }}-init-config
1258+
image: busybox:1.28
1259+
volumeMounts:
1260+
- name: st2-sensor-config-vol
1261+
mountPath: /tmp/st2
1262+
command:
1263+
- 'sh'
1264+
- '-ec'
1265+
- |
1266+
cat <<EOT > /tmp/st2/st2.sensorcontainer.conf
1267+
[sensorcontainer]
1268+
sensor_node_name = {{ $name }}
1269+
partition_provider = name:hash, hash_ranges:{{ $sensor.hash_ranges }}
1270+
single_sensor_mode = False
1271+
EOT
1272+
{{- end }}
12371273
{{- include "stackstorm-ha.init-containers-wait-for-db" $ | nindent 6 }}
12381274
{{- include "stackstorm-ha.init-containers-wait-for-mq" $ | nindent 6 }}
12391275
{{- if and $.Values.st2.packs.images (not $.Values.st2.packs.volumes.enabled) }}
@@ -1255,14 +1291,19 @@ spec:
12551291
livenessProbe:
12561292
{{- toYaml . | nindent 10 }}
12571293
{{- end }}
1258-
{{- if $one_sensor_per_pod }}
1294+
{{- if or $one_sensor_per_pod $some_sensors_per_pod }}{{/* ie: when there is more than one pod of sensors */}}
12591295
command:
12601296
- /opt/stackstorm/st2/bin/st2sensorcontainer
12611297
- --config-file=/etc/st2/st2.conf
12621298
- --config-file=/etc/st2/st2.docker.conf
12631299
- --config-file=/etc/st2/st2.user.conf
1300+
{{- if $one_sensor_per_pod }}{{/* only in st2.packs.sensors[] */}}
12641301
- --single-sensor-mode
12651302
- --sensor-ref={{ required "You must define `ref` for everything in st2.packs.sensors. This assigns each sensor to a pod." $sensor.ref }}
1303+
{{- else if $some_sensors_per_pod }}
1304+
# injected by {{ $name }}-init-config
1305+
- --config-file=/etc/st2/st2.sensorcontainer.conf
1306+
{{- end }}
12661307
{{- end }}
12671308
{{- if $sensor.env }}
12681309
env: {{- include "stackstorm-ha.customEnv" . | nindent 8 }}
@@ -1277,6 +1318,12 @@ spec:
12771318
volumeMounts:
12781319
{{- include "stackstorm-ha.st2-config-volume-mounts" $ | nindent 8 }}
12791320
{{- include "stackstorm-ha.packs-volume-mounts" $ | nindent 8 }}
1321+
{{- if $some_sensors_per_pod }}
1322+
- name: st2-sensor-config-vol
1323+
mountPath: /etc/st2/st2.sensorcontainer.conf
1324+
subPath: st2.sensorcontainer.conf
1325+
readOnly: true
1326+
{{- end }}
12801327
{{- if ne "disable" (default "" $.Values.st2.datastore_crypto_key) }}
12811328
- name: st2-encryption-key-vol
12821329
mountPath: /etc/st2/keys
@@ -1311,6 +1358,11 @@ spec:
13111358
{{- end }}
13121359
{{- include "stackstorm-ha.st2-config-volume" $ | nindent 8 }}
13131360
{{- include "stackstorm-ha.packs-volumes" $ | nindent 8 }}
1361+
{{- if $some_sensors_per_pod }}
1362+
- name: st2-sensor-config-vol
1363+
emptyDir: # This is for a tiny file
1364+
medium: Memory
1365+
{{- end }}
13141366
{{- range $sensor.extra_volumes }}
13151367
- name: {{ required "Each volume must have a 'name' in $sensor.extra_volumes" .name }}
13161368
{{- tpl (required "Each volume must have a 'volume' definition in $sensor.extra_volumes" .volume | toYaml) $ | nindent 10 }}

‎tests/unit/st2sensors_test.yaml‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -247,3 +247,154 @@ tests:
247247
asserts:
248248
- failedTemplate:
249249
errorMessage: "You must define `ref` for everything in st2.packs.sensors. This assigns each sensor to a pod."
250+
251+
- it: stackstorm/sensor-mode = multiple-sensors-per-pod
252+
template: deployments.yaml
253+
release:
254+
name: multi-sensor-release
255+
set:
256+
st2:
257+
packs: { sensors: [] }
258+
image:
259+
tag: globaldefault
260+
st2sensorcontainer:
261+
image:
262+
tag: sensordefault
263+
deployments: 3
264+
securityContext: *override_security_context
265+
securityContext: *global_security_context
266+
asserts:
267+
- hasDocuments:
268+
count: 15 # all but st2chatops
269+
270+
- isKind:
271+
of: Deployment
272+
documentIndex: *first_sensor_doc
273+
- isKind:
274+
of: Deployment
275+
documentIndex: *second_sensor_doc
276+
- isKind:
277+
of: Deployment
278+
documentIndex: *third_sensor_doc
279+
280+
- equal:
281+
path: metadata.name
282+
value: multi-sensor-release-st2sensorcontainer-0
283+
documentIndex: *first_sensor_doc
284+
- equal:
285+
path: metadata.name
286+
value: multi-sensor-release-st2sensorcontainer-1
287+
documentIndex: *second_sensor_doc
288+
- equal:
289+
path: metadata.name
290+
value: multi-sensor-release-st2sensorcontainer-2
291+
documentIndex: *third_sensor_doc
292+
293+
- equal:
294+
path: metadata.labels.app
295+
value: st2sensorcontainer-0
296+
documentIndex: *first_sensor_doc
297+
- equal:
298+
path: metadata.labels.app
299+
value: st2sensorcontainer-1
300+
documentIndex: *second_sensor_doc
301+
- equal:
302+
path: metadata.labels.app
303+
value: st2sensorcontainer-2
304+
documentIndex: *third_sensor_doc
305+
306+
- equal:
307+
path: spec.template.metadata.labels.app
308+
value: st2sensorcontainer-0
309+
documentIndex: *first_sensor_doc
310+
- equal:
311+
path: spec.template.metadata.labels.app
312+
value: st2sensorcontainer-1
313+
documentIndex: *second_sensor_doc
314+
- equal:
315+
path: spec.template.metadata.labels.app
316+
value: st2sensorcontainer-2
317+
documentIndex: *third_sensor_doc
318+
319+
- equal:
320+
path: spec.selector.matchLabels.app
321+
value: st2sensorcontainer-0
322+
documentIndex: *first_sensor_doc
323+
- equal:
324+
path: spec.selector.matchLabels.app
325+
value: st2sensorcontainer-1
326+
documentIndex: *second_sensor_doc
327+
- equal:
328+
path: spec.selector.matchLabels.app
329+
value: st2sensorcontainer-2
330+
documentIndex: *third_sensor_doc
331+
332+
- equal: *oneReplica
333+
documentIndex: *first_sensor_doc
334+
- equal: *oneReplica
335+
documentIndex: *second_sensor_doc
336+
- equal: *oneReplica
337+
documentIndex: *third_sensor_doc
338+
339+
- equal: &multiSensorAnnotation
340+
path: spec.template.metadata.annotations.stackstorm/sensor-mode
341+
value: multiple-sensors-per-pod
342+
documentIndex: *first_sensor_doc
343+
- equal: *multiSensorAnnotation
344+
documentIndex: *second_sensor_doc
345+
- equal: *multiSensorAnnotation
346+
documentIndex: *third_sensor_doc
347+
348+
- notContains: &notSingleSensorMode
349+
path: spec.template.spec.containers[0].command
350+
content: '--single-sensor-mode'
351+
documentIndex: *first_sensor_doc
352+
- notContains: *notSingleSensorMode
353+
documentIndex: *second_sensor_doc
354+
- notContains: *notSingleSensorMode
355+
documentIndex: *third_sensor_doc
356+
357+
- contains: &sensorConf
358+
path: spec.template.spec.containers[0].command
359+
content: '--config-file=/etc/st2/st2.sensorcontainer.conf'
360+
documentIndex: *first_sensor_doc
361+
- contains: *sensorConf
362+
documentIndex: *second_sensor_doc
363+
- contains: *sensorConf
364+
documentIndex: *third_sensor_doc
365+
366+
# st2sensorcontainer values apply to all sensor containers replacing globals
367+
368+
- matchRegex: &multiSensorTag
369+
path: spec.template.spec.containers[0].image
370+
pattern: '.*:sensordefault$'
371+
documentIndex: *first_sensor_doc
372+
- matchRegex: *multiSensorTag
373+
documentIndex: *second_sensor_doc
374+
- matchRegex: *multiSensorTag
375+
documentIndex: *third_sensor_doc
376+
377+
- equal: &multiSensorSecurityContext
378+
path: spec.template.spec.containers[0].securityContext
379+
value: *override_security_context
380+
documentIndex: *first_sensor_doc
381+
- equal: *multiSensorSecurityContext
382+
documentIndex: *second_sensor_doc
383+
- equal: *multiSensorSecurityContext
384+
documentIndex: *third_sensor_doc
385+
386+
- it: ambiguous stackstorm/sensor-mode fails
387+
template: deployments.yaml
388+
release:
389+
name: missing-sensor-ref-release
390+
set:
391+
st2:
392+
packs:
393+
sensors:
394+
- name: foo
395+
ref: some_pack.foo_sensor
396+
st2sensorcontainer:
397+
deployments: 2
398+
asserts:
399+
- failedTemplate:
400+
errorMessage: "The sensor values are ambiguous. To use one-sensor-per-pod, use `st2.packs.sensors`. To use multiple-sensors-per-pod, use `st2sensorcontainer.deployments > 1`. Do not use both!"

‎values.yaml‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -152,10 +152,11 @@ st2:
152152
# https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer
153153
# It is possible to run st2sensorcontainer(s) in one of these modes:
154154
# (1) run all sensors in one pod (1 deployment with 1 pod, the default); or
155-
# (2) run one sensor per pod using st2.packs.sensors (here).
155+
# (2) run multiple sensors per pod (2+ deployments with 1 pod each) using hash range partitions; or
156+
# (3) run one sensor per pod using st2.packs.sensors (here).
156157
# Each sensor node needs to be provided with proper partition information to share work with other sensor
157158
# nodes so that the same sensor does not run on different nodes.
158-
# When this is empty (the default), the chart adds one pod to run all sensors.
159+
# By default, this is empty and st2sensorcontainer.deployments (below) is 1 so that there is one pod to run all sensors.
159160
sensors: []
160161
# This is a list of sensor pods (stackstorm/sensor-mode=one-sensor-per-pod).
161162
# Each entry should have `name` (the pod name) and `ref` (which sensor to run in the pod).
@@ -679,10 +680,21 @@ st2actionrunner:
679680
# https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer
680681
# It is possible to run st2sensorcontainer(s) in one of these modes:
681682
# (1) run all sensors in one pod (1 deployment with 1 pod, the default); or
682-
# (2) run one sensor per pod using st2.packs.sensors (see above).
683-
# To use the default mode (all sensors in one pod), st2.packs.sensors must be empty.
683+
# (2) run multiple sensor per pod (2+ deployments with 1 pod each) using hash range partitions; or
684+
# (3) run one sensor per pod using st2.packs.sensors (see above).
685+
# To use the deployments (modes 1 and 2 in this list), st2.packs.sensors must be empty.
684686
# For one-sensor-per-pod, define defaults here and add config for each sensor to st2.packs.sensors (above).
685687
st2sensorcontainer:
688+
# "deployments" is analogous to "replicas" for other services. Each deployment has 1 replica.
689+
# The deployments count cannot be greater than 1 unless st2.packs.sensors is empty.
690+
# If deployments is 1 (the default), then StackStorm puts all sensors in one pod.
691+
# If deployments is 2+, StackStorm is configured to use sensor hash ranges to ensure that
692+
# each sensor runs on only one pod. Each hash range is assigned to a separate
693+
# deployment to ease maintenance of the sensor deployments and pods.
694+
# See: https://docs.stackstorm.com/reference/sensor_partitioning.html#hash
695+
deployments: 1
696+
697+
# values below this point are also used as defaults in st2.packs.sensors (if used)
686698
resources:
687699
requests:
688700
memory: "100Mi"

0 commit comments

Comments
 (0)