Skip to content

feat: add livenessProbe and readinessProbe to all services #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kodiakhq merged 5 commits into from
Aug 8, 2025
Merged

feat: add livenessProbe and readinessProbe to all services #88

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
840a450
feat: add livenessProbe and readinessProbe to all services
knudtty Jul 28, 2025
db03ca5
test: add livenessProbe and readinessProbe tests
knudtty Jul 28, 2025
1e8e0b2
fix: tcpSocket probe rather than cmd
knudtty Jul 28, 2025
8889877
add configurable probe values
knudtty Aug 8, 2025
e778a9d
Merge branch 'main' into aaron/health-probes
knudtty Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/pretty-ants-arrive.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"helm-charts": patch
---

feat: add livenessProbe and readinessProbe for services
16 changes: 16 additions & 0 deletions charts/hdx-oss-v2/templates/clickhouse-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,22 @@ spec:
env:
- name: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
value: "1"
livenessProbe:
httpGet:
path: /ping
port: {{ .Values.clickhouse.port }}
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /ping
port: {{ .Values.clickhouse.port }}
initialDelaySeconds: 1
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: config
mountPath: /etc/clickhouse-server/config.xml
Expand Down
16 changes: 16 additions & 0 deletions charts/hdx-oss-v2/templates/hyperdx-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,22 @@ spec:
resources:
{{- toYaml .Values.hyperdx.resources | nindent 12 }}
{{- end }}
livenessProbe:
httpGet:
path: /health
port: {{ .Values.hyperdx.apiPort }}
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: {{ .Values.hyperdx.apiPort }}
initialDelaySeconds: 1
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
envFrom:
- configMapRef:
name: {{ include "hdx-oss.fullname" . }}-app-config
Expand Down
20 changes: 20 additions & 0 deletions charts/hdx-oss-v2/templates/mongodb-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,26 @@ spec:
image: "{{ .Values.mongodb.image }}"
ports:
- containerPort: {{ .Values.mongodb.port }}
livenessProbe:
exec:
command:
- mongosh
- --eval
- "db.adminCommand('ping')"
Copy link
Contributor

@teeohhem teeohhem Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious where this came from? I hesitate to run elevated commands as part of a liveness or readiness probe.

Maybe we do just a simple tcpport listening check?

tcpSocket:
    port: {{ .Values.mongodb.port }}

Copy link
Collaborator Author

@knudtty knudtty Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can tell online that's the healthcheck that is recommended for mongo, but there really isn't much info to find on this. I get the apprehension on admin privileges though, I'll adopt the tcpSocket approach

initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
exec:
command:
- mongosh
- --eval
- "db.adminCommand('ping')"
initialDelaySeconds: 1
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: mongodb-data
mountPath: /data/db
Expand Down
16 changes: 16 additions & 0 deletions charts/hdx-oss-v2/templates/otel-collector-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,22 @@ spec:
resources:
{{- toYaml .Values.otel.resources | nindent 12 }}
{{- end }}
livenessProbe:
httpGet:
path: /
port: {{ .Values.otel.port }}
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: {{ .Values.otel.port }}
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
env:
- name: CLICKHOUSE_ENDPOINT
value: "{{ .Values.otel.clickhouseEndpoint | default (printf "tcp://%s-clickhouse:%v?dial_timeout=10s" (include "hdx-oss.fullname" .) .Values.clickhouse.nativePort) }}"
Expand Down
53 changes: 52 additions & 1 deletion charts/hdx-oss-v2/tests/clickhouse-deployment_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,4 +145,55 @@ tests:
path: spec.template.spec.volumes[3].emptyDir
- documentIndex: 0
isNull:
path: spec.template.spec.volumes[3].persistentVolumeClaim
path: spec.template.spec.volumes[3].persistentVolumeClaim

- it: should include livenessProbe with default values
set:
clickhouse:
enabled: true
port: 8123
asserts:
- documentIndex: 0
isSubset:
path: spec.template.spec.containers[0].livenessProbe
content:
httpGet:
path: /ping
port: 8123
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3

- it: should include readinessProbe with default values
set:
clickhouse:
enabled: true
port: 8123
asserts:
- documentIndex: 0
isSubset:
path: spec.template.spec.containers[0].readinessProbe
content:
httpGet:
path: /ping
port: 8123
initialDelaySeconds: 1
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3

- it: should use custom port in probes when provided
set:
clickhouse:
enabled: true
port: 8124
asserts:
- documentIndex: 0
equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.port
value: 8124
- documentIndex: 0
equal:
path: spec.template.spec.containers[0].readinessProbe.httpGet.port
value: 8124
40 changes: 39 additions & 1 deletion charts/hdx-oss-v2/tests/hyperdx-deployment_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,4 +128,42 @@ tests:
content: -c
- matchRegex:
path: spec.template.spec.initContainers[0].command[2]
pattern: "until nc -z .+-mongodb [0-9]+; do echo waiting for mongodb; sleep 2; done;"
pattern: "until nc -z .+-mongodb [0-9]+; do echo waiting for mongodb; sleep 2; done;"

- it: should include livenessProbe with default values
asserts:
- isSubset:
path: spec.template.spec.containers[0].livenessProbe
content:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3

- it: should include readinessProbe with default values
asserts:
- isSubset:
path: spec.template.spec.containers[0].readinessProbe
content:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 1
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3

- it: should use custom apiPort in probes when provided
set:
hyperdx:
apiPort: 9000
asserts:
- equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.port
value: 9000
- equal:
path: spec.template.spec.containers[0].readinessProbe.httpGet.port
value: 9000
40 changes: 39 additions & 1 deletion charts/hdx-oss-v2/tests/mongodb-deployment_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,42 @@ tests:
path: spec.template.spec.volumes[0].emptyDir
- documentIndex: 0
isNull:
path: spec.template.spec.volumes[0].persistentVolumeClaim
path: spec.template.spec.volumes[0].persistentVolumeClaim

- it: should include livenessProbe with exec command
set:
mongodb:
enabled: true
asserts:
- documentIndex: 0
isSubset:
path: spec.template.spec.containers[0].livenessProbe
content:
exec:
command:
- mongosh
- --eval
- "db.adminCommand('ping')"
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3

- it: should include readinessProbe with exec command
set:
mongodb:
enabled: true
asserts:
- documentIndex: 0
isSubset:
path: spec.template.spec.containers[0].readinessProbe
content:
exec:
command:
- mongosh
- --eval
- "db.adminCommand('ping')"
initialDelaySeconds: 1
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
51 changes: 51 additions & 0 deletions charts/hdx-oss-v2/tests/otel-collector_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -686,3 +686,54 @@ tests:
checksum/config: "abcdef1234567890"
deployment.kubernetes.io/revision: "1"
sidecar.istio.io/inject: "false"

- it: should include livenessProbe with default values
set:
otel:
enabled: true
port: 13133
asserts:
- documentIndex: 0
isSubset:
path: spec.template.spec.containers[0].livenessProbe
content:
httpGet:
path: /
port: 13133
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3

- it: should include readinessProbe with default values
set:
otel:
enabled: true
port: 13133
asserts:
- documentIndex: 0
isSubset:
path: spec.template.spec.containers[0].readinessProbe
content:
httpGet:
path: /
port: 13133
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3

- it: should use custom port in probes when provided
set:
otel:
enabled: true
port: 13134
asserts:
- documentIndex: 0
equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.port
value: 13134
- documentIndex: 0
equal:
path: spec.template.spec.containers[0].readinessProbe.httpGet.port
value: 13134