Should i run and how should i run the prestop hook in a k8s environment

[david-gang]

Question

At

cloud-sql-proxy/examples/k8s-health-check/proxy_with_http_health_check.yaml

Line 119 in 9f36f09

preStop:

we have an example of

        lifecycle:
          preStop:
            httpGet:
              path: /quitquitquit
              port: 9092
              scheme: HTTP

but there exist a problem. In order that this hook should be accessible to kubelet it should listen to 0.0.0.0

if i use this example i get a Warning FailedPreStopHook event.

The reason is simple while it is possible to override

- name: CSQL_PROXY_HTTP_ADDRESS
  value: 0.0.0.0

the address of the admin server is hard coded as localhost and inaccessible to kubelet

cloud-sql-proxy/cmd/root.go

Line 1136 in 9f36f09

net.JoinHostPort("localhost", cmd.conf.AdminPort),

so either it is not important to call the prestop hook or we need another way calling it but i have no idea how.

Code

Additional Details

No response

[kgala2]

Hi @david-gang ,

Using exit-zero-on-sigterm might help exit gracefully in k8s environment

cloud-sql-proxy/docs/cmd/cloud-sql-proxy.md

Line 243 in 9f36f09

--exit-zero-on-sigterm Exit with 0 exit code when Sigterm received (default is 143)

Please let me know if this helps.

Thanks

[david-gang]

Hi,

Thanks for your prompt response.
I already use this flag through the environment variable

- name: CSQL_PROXY_EXIT_ZERO_ON_SIGTERM
    value: "true"

There exist two distinct issues here:

Graceful termination

I am getting these logs

{"timestamp":"2025-11-04T07:52:49Z","severity":"DEBUG","message":"Sending SIGTERM signal for proxy to shutdown."}
{"timestamp":"2025-11-04T07:52:49Z","severity":"INFO","message":"SIGTERM signal received. Shutting down..."}
{"timestamp":"2025-11-04T07:52:49Z","severity":"DEBUG","message":"proxy server error: accept tcp 127.0.0.1:5000: use of closed network connection"}

This may be connected to #2372

Calling prestop hook from kubelet

Additionally i am getting in the events of kubernetes

31s Warning FailedPreStopHook pod/my-cool-pod PreStopHook failed

When looking at the logs of the nodes i get the following error:

PreStop hook failed" err="Get "http://localhost:9091/quitquitquit\": dial tcp [::1]:9091: connect: connection refused"

so this issue is not connected to graceful termination but rather that the prestop hook is not reachable

The root cause of this problem is that the admin server just listens to localhost and therefore isn't reachable for kubelet:

	if needsAdminServer {
		go startHTTPServer(
			ctx,
			cmd.logger,
			net.JoinHostPort("localhost", cmd.conf.AdminPort),
			m,
			shutdownCh,
		)
	}

The problem with the healthchecks was that it is possible to pass the address of the server which receives the health checks through the CSQL_PROXY_HTTP_ADDRESS variable but this is not possible with the admin server

Let's now determine: Is it important to call the prestop hook before receiving the sigterm?

Code analysis of quitquit vs graceful shutdown

quitquit pushes the signal of errQuitQuitQuit to the shutdownCh

func quitquitquit(quitOnce *sync.Once, shutdownCh chan<- error) http.HandlerFunc {
	return func(rw http.ResponseWriter, req *http.Request) {
		if req.Method != http.MethodPost && req.Method != http.MethodGet {
			rw.WriteHeader(400)
			return
		}
		quitOnce.Do(func() {
			select {
			case shutdownCh <- errQuitQuitQuit:
			default:
				// The write attempt to shutdownCh failed and
				// the proxy is already exiting.
			}
		})
	}
}

In case of receiving sigterm i also push signals to the shutDownCh:

case syscall.SIGTERM:
			cmd.logger.Debugf("Sending SIGTERM signal for proxy to shutdown.")
			if cmd.conf.ExitZeroOnSigterm {
				shutdownCh <- errSigTermZero
			} else {
				shutdownCh <- errSigTerm
			}
		}

Then in the main function:

go func() {
		err := p.Serve(ctx, notifyStarted)
		cmd.logger.Debugf("proxy server error: %v", err)
		shutdownCh <- err
	}()

	err = <-shutdownCh
	switch {
	case errors.Is(err, errSigInt):
		cmd.logger.Infof("SIGINT signal received. Shutting down...")
		time.Sleep(cmd.conf.WaitBeforeClose)
	case errors.Is(err, errSigTerm):
		cmd.logger.Infof("SIGTERM signal received. Shutting down...")
		time.Sleep(cmd.conf.WaitBeforeClose)
	case errors.Is(err, errSigTermZero):
		cmd.logger.Infof("SIGTERM signal received. Shutting down...")
		time.Sleep(cmd.conf.WaitBeforeClose)
	case errors.Is(err, errQuitQuitQuit):
		cmd.logger.Infof("/quitquitquit received request. Shutting down...")
		time.Sleep(cmd.conf.WaitBeforeClose)
	default:
		cmd.logger.Errorf("The proxy has encountered a terminal error: %v", err)
	}
	return err

so except the type of the error we get the logic is the same. we just sleep for cmd.conf.WaitBeforeClose

so by my understanding the calling of the quitquit handler through the prestop hook does not add here anything. Please correct me if i am wrong .

so if this analysis is correct then the example of https://github.com/GoogleCloudPlatform/cloud-sql-proxy/blob/main/examples/k8s-health-check/proxy_with_http_health_check.yaml needs to be changed and the prestop hook should be deleted

        # Configure kubernetes to call the /quitquitquit endpoint on the
        # admin server before sending SIGTERM to the proxy before stopping
        # the pod. This will give the proxy more time to gracefully exit.
        lifecycle:
          preStop:
            httpGet:
              path: /quitquitquit
              port: 9092
              scheme: HTTP

for two reasons

1. it is unreachable from kubelet
2. the graceful termination of the container achieves the same

If we don't need the quitquit handler we also don't need the admin server by my understanding and we can remove

        # Enable the admin api server (which only listens for local connections)
        # and enable the /quitquitquit endpoint. This allows other pods
        # to shut down the proxy gracefully when they are ready to exit.
        - name: CSQL_PROXY_QUITQUITQUIT
          value: "true"
        - name: CSQL_PROXY_ADMIN_PORT
          value: "9092"

simplifying the example even more

[enocom]

Not a maintainer anymore here, so drive-by comment:

Have you considered have your main container call the /quitquitquit endpoint, and then configure a preStop hook against your main container?

[david-gang]

Not a maintainer anymore here, so drive-by comment:

Have you considered have your main container call the /quitquitquit endpoint, and then configure a preStop hook against your main container?

I considered it but i don't like the solution to much because:

• I need a sort of http client like curl in the main container and i don't have it in the main containers i deploy . i try to have minimal containers.
• The sql proxy container will be less self contained and it makes it harder to manage this from the devops side
• I am still not convinced of the need to run the /quitquitquit endpoint and how it differs from just cloud sql proxy handling the sigterm as i explained in the previous comment.

[enocom]

The /quitquitquit endpoint really only exists to support clean shutdown from K8s jobs. Before we had that endpoint, it was challenging to stop the Proxy without having the job marked as a failure.

So unless you're running a job, you probably don't need /quitquitquit and can use the flag suggested by @kgala2 above.

[david-gang]

So the example of https://github.com/GoogleCloudPlatform/cloud-sql-proxy/blob/main/examples/k8s-health-check/proxy_with_http_health_check.yaml needs to be changed. I could open a pr if desired.

In the meantime the following parts of the example should be removed:

        # Enable the admin api server (which only listens for local connections)
        # and enable the /quitquitquit endpoint. This allows other pods
        # to shut down the proxy gracefully when they are ready to exit.
        - name: CSQL_PROXY_QUITQUITQUIT
          value: "true"
        - name: CSQL_PROXY_ADMIN_PORT
          value: "9092"

        # Configure kubernetes to call the /quitquitquit endpoint on the
        # admin server before sending SIGTERM to the proxy before stopping
        # the pod. This will give the proxy more time to gracefully exit.
        lifecycle:
          preStop:
            httpGet:
              path: /quitquitquit
              port: 9092
              scheme: HTTP