You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Kubernetes plugin enforces a maximum number of concurrent
connections to the Kubernetes API server to avoid crowding it. Once the
limit is reached (the default is 32), connection requests are queued and
wait until an existing connection is terminated. Pending connection
requests can time out after the configured interval (the default is 5s)
if not serviced. Such timeouts manifest as the following error:
io.fabric8.kubernetes.client.KubernetesClientException: not ready
after 5000 MILLISECONDS
For each `sh` step executed in a pod that's *not* the JNLP container, a
Kubernetes connection is allocated. In our pipeline, the `build` job
runs everything in the cosa container. We have sections where many
`sh` steps are executed in parallel (the `cosa buildextend-${it}`
closures), and if multiple jobs running in parallel coincide on this
section, the maximum connection limit can be reached. And because
building disk images can easily take more than 5s, we can timeout as
described above.
Mitigate this by increasing the amount of concurrent connections
permissible to 64.
It could be interesting to have a derived cosa image containing the
Jenkins agent, which would fully fix this issue. But using a derived
cosa in our pipeline has deep consequences. (An alternative of course is
baking it directly in cosa, though that's not appropriate either.)
A "recipe" model of building FCOS would be another way to mitigate this,
because it would considerably reduce the number of `sh` steps used.
For more details, see:
https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-masters/considerations-for-kubernetes-clients-connection-when-using-kubernetes-plugin
0 commit comments