Issue separating dbconnector into it's own service

[stephenmoloney]

In a recent release, 6.4.0, it because possible to separate the dbconnector into it's own deployment.

I think this is a great idea. But unfortunately, despite making multiple attempts at the deployment with different cpu and memory settings - including way over capacity, (including the node options for the heap size), I keep getting back this error;

│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; shutting down workers","timestamp":"2025-03-31T16:57:40.700Z"}                                                                    │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.701Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.701Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.701Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.701Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.701Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.701Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.701Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.703Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.703Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.703Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.703Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.703Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.703Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.703Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 134","timestamp":"2025-03-31T16:57:40.703Z"}                                                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 120","timestamp":"2025-03-31T16:57:40.703Z"}                                                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 112","timestamp":"2025-03-31T16:57:40.703Z"}                                                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 100","timestamp":"2025-03-31T16:57:40.703Z"}                                                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 148","timestamp":"2025-03-31T16:57:40.703Z"}                                                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 114","timestamp":"2025-03-31T16:57:40.703Z"}                                                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 128","timestamp":"2025-03-31T16:57:40.703Z"}                                                                                          │
│ retool-dbconnector {"level":"info","message":"SIGTERM signal received; stopping server","timestamp":"2025-03-31T16:57:40.709Z"}                                                                          │
│ retool-dbconnector {"level":"info","message":"Stopping database connector server on port","port":"3002","timestamp":"2025-03-31T16:57:40.710Z"}                                                          │
│ retool-dbconnector {"level":"info","message":"Server closed on pid 99","timestamp":"2025-03-31T16:57:40.711Z"}
│ retool-dbconnector {"level":"info","message":"Server closed on pid 99","timestamp":"2025-03-31T16:57:40.711Z"}                                                                                           │
│ retool-dbconnector {"code":0,"kernelOutput":"Command failed: dmesg -T | grep -F -i -e \"$(dmesg -T | grep -F -i 'killed process' | tail -n 1 | grep -oP '(?<=\\[)[^]]+' | head -n 1 | { read input; [ -z │
│ retool-dbconnector {"level":"info","message":"[Master] All workers are dead; shutting down dbconnector","timestamp":"2025-03-31T16:57:41.014Z"}  

[
  {
    "code": 0,
    "kernelOutput": "Command failed: dmesg -T | grep -F -i -e \"$(dmesg -T | grep -F -i 'killed process' | tail -n 1 | grep -oP '(?<=\\[)[^]]+' | head -n 1 | { read input; [ -z \"$input\" ] && echo \"[oom string was not found]\" || echo \"$input\"; })\" -e \"$(dmesg -T | grep -F -i 'invoked oom-killer' | tail -n 1 | grep -oP '(?<=\\[)[^]]+' | head -n 1 | { read input; [ -z \"$input\" ] && echo \"[oom string was not found]\" || echo \"$input\"; })\"\ndmesg: read kernel buffer failed: Operation not permitted\ndmesg: read kernel buffer failed: Operation not permitted\ndmesg: read kernel buffer failed: Operation not permitted\n",
    "level": "info",
    "message": "[Master] Worker died",
    "pid": 100,
    "signal": null,
    "source": "master",
    "timestamp": "2025-03-31T15:35:07.152Z"
  },
  {
    "level": "info",
    "message": "[Master] All workers are dead; shutting down dbconnector",
    "timestamp": "2025-03-31T15:35:07.152Z"
  }
]

dmesg: read kernel buffer failed: Operation not permitted\ seems like a consquequence of something else that is already failing.

The YAML for the dbconnector is as follows;

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: retool
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: retool-dbconnector
    helm.sh/chart: retool-6.4.4
    retoolService: retool-dbconnector
    telemetry.retool.com/service-name: dbconnector
  name: retool-dbconnector
  namespace: retool
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      retoolService: retool-dbconnector
  template:
    metadata:
      annotations: null
      labels:
        app.kubernetes.io/instance: retool
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: retool-dbconnector
        helm.sh/chart: retool-6.4.4
        retoolService: retool-dbconnector
        telemetry.retool.com/service-name: dbconnector
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app.kubernetes.io/name
                      operator: In
                      values:
                        - retool
                topologyKey: kubernetes.io/hostname
              weight: 100
      containers:
        - args:
            - bash
            - -c
            - chmod -R +x ./docker_scripts; sync; ./docker_scripts/wait-for-it.sh -t 0
              "$(POSTGRES_HOST):$(POSTGRES_PORT)"; ./docker_scripts/start_api.sh
          env:
            - name: SERVICE_TYPE
              value: DB_CONNECTOR,DB_SSH_CONNECTOR,JAVA_DBCONNECTOR
            - name: DEPLOYMENT_TEMPLATE_TYPE
              value: k8s-helm
            - name: DEPLOYMENT_TEMPLATE_VERSION
              value: 6.4.4
            - name: NODE_ENV
              value: production
            - name: NODE_OPTIONS
              value: --max_old_space_size=1024
            - name: DBCONNECTOR_HTTP_AGENT_MAX_SOCKETS
              value: "1000"
            - name: DBCONNECTOR_POSTGRES_POOL_MAX_SIZE
              value: "100"
            - name: DISABLE_DATABASE_MIGRATIONS
              value: "true"
            - name: WORKFLOW_BACKEND_HOST
              value: http://retool-workflow-backend
            - name: CLIENT_ID
              value: null
            - name: COOKIE_INSECURE
              value: "false"
            - name: NODE_OPTIONS
              value: --max-old-space-size=12288
            - name: ENCRYPTION_KEY
              valueFrom:
                secretKeyRef:
                  key: encryption-key
                  name: retool
            - name: JWT_SECRET
              valueFrom:
                secretKeyRef:
                  key: jwt-secret
                  name: retool
            - name: LICENSE_KEY
              valueFrom:
                secretKeyRef:
                  key: license-key
                  name: retool
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: postgresql-password
                  name: retool
          image: tryretool/backend:3.148.5-stable
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /api/checkHealth
              port: 3002
            initialDelaySeconds: 30
            timeoutSeconds: 10
          name: retool-dbconnector
          readinessProbe:
            httpGet:
              path: /api/checkHealth
              port: 3002
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 5
            timeoutSeconds: 10
          resources:
            limits:
              memory: 16384Mi
            requests:
              cpu: 8192m
              memory: 16384Mi
          volumeMounts: null
      nodeSelector:
        kubernetes.io/arch: amd64
      serviceAccountName: retool
      terminationGracePeriodSeconds: 30
      tolerations: []

Any ideas?

[jjlgao]

Hi Stephen,

We use dmesg to print out messages in the event of an OOM, to be able to debug if it's coming from the underlying OS that the containers run on. It's useful for us, although the command may only have permission on certain OSes.

With that being said, it's happening after the pods are asked to shut down. It looks like something is happening after startup that causes something external to trigger a shutdown on the pod, before it's able to take traffic.

Would you be able to provide any logs prior to the shutdown message, and if you use a container orchestration tool, some kind of list of events on the dbconnector containers that are shutting down?

[stephenmoloney]

Hi @jjlgao,

I'm not too keen to send over logs on the public github.

Are you working for retool the company? Would it be possible to arrange a video call through a retool support ticket - is there some way you could raise one internally for this?

I am working with a prominent client in the financial services sphere.

[jjlgao]

Hi @stephenmoloney , apologies for seeing this late. I am part of Retool, a member of the team that maintains our helm charts. If you need support assistance, please reach out through the proper support channels, I unfortunately don't have the ability to raise one internally: https://docs.retool.com/support#support

[jharveymann]

I remember seeing something similar a while ago debugging something else, but couldn't repro again using my usual local deployments. Spent some time digging in, and think it's actually an issue with the db-connector's health check.

This checks the container's memory, but behind the scenes appears to rely on the host running the container to be using cgroups v1. On deployments with cgroups v2, it leads to an uncaught error when hitting the /api/checkHealth endpoint which is probably what's leading to the runtime sending a SIGTERM to the processes. Tested this against local Docker deployments running the same Retool version on Windows and Mac, which in my case use v1 and v2 respectively:

You can exec in and run a similar check in any of the containers to confirm in your case, and it's something we'll need to better handle in the code.

Another issue to note for Helm specifically is that unlike the backend deployment, the db-connector probe configs aren't conditionally applied via {{ .Values.[probe].enabled }} or something similar. Looks like they're hard-coded to always be set without an option in the values file. So you can manually remove the checks by editing the deployment to try to unblock for now before we can fix the cgroups issue.

[marcinkubica]

@jjlgao
I have the same issue. With liveness probe removed this is what's being reported by healthcheck

looks what's reported by @jharveymann

* Connected to 10.104.9.10 (10.104.9.10) port 3002
> GET /api/checkHealth HTTP/1.1
> Host: 10.104.9.10:3002
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/html; charset=utf-8
< Content-Length: 28
< ETag: W/"1c-Kb/8pveL6uLfkcmoo594wJOVeig"
< Date: Tue, 02 Dec 2025 18:16:22 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
< 
{ [28 bytes data]
100    28  100    28    0     0   3209      0 --:--:-- --:--:-- --:--:--  3500
* Connection #0 to host 10.104.9.10 left intact
Error checking system memory%                                                                                

dmesg is not helpful here and "SIGTERM signal received; is just livenessprobe kicking

The remark about probes is also valid - charts seem to share probe settings, they should be delegated to own config. With this done user can override probes to work around the problem in the short them (ie just by changing to tcp 3002 for instance).

Finally, enabling dbconnector doesn't make rest of retool services using it exclusively for db connections. I see following services still on list of direct postgres db connections:

retool-workflow-worker
retool-jobs-runner
retool
retool-workflow-backend