Skip to content

Commit fc35327

Browse files
committed
jenkins/config: bump kubernetes-plugin maxRequestsPerHost
The Kubernetes plugin enforces a maximum number of concurrent connections to the Kubernetes API server to avoid crowding it. Once the limit is reached (the default is 32), connection requests are queued and wait until an existing connection is terminated. Pending connection requests can time out after the configured interval (the default is 5s) if not serviced. Such timeouts manifest as the following error: io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS For each `sh` step executed in a pod that's *not* the JNLP container, a Kubernetes connection is allocated. In our pipeline, the `build` job runs everything in the cosa container. We have sections where many `sh` steps are executed in parallel (the `cosa buildextend-${it}` closures), and if multiple jobs running in parallel coincide on this section, the maximum connection limit can be reached. And because building disk images can easily take more than 5s, we can timeout as described above. Mitigate this by increasing the amount of concurrent connections permissible to 64. It could be interesting to have a derived cosa image containing the Jenkins agent, which would fully fix this issue. But using a derived cosa in our pipeline has deep consequences. (An alternative of course is baking it directly in cosa, though that's not appropriate either.) A "recipe" model of building FCOS would be another way to mitigate this, because it would considerably reduce the number of `sh` steps used. For more details, see: https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-masters/considerations-for-kubernetes-clients-connection-when-using-kubernetes-plugin
1 parent 689616a commit fc35327

File tree

1 file changed

+7
-0
lines changed

1 file changed

+7
-0
lines changed
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# mitigation for
2+
# https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-masters/considerations-for-kubernetes-clients-connection-when-using-kubernetes-plugin
3+
jenkins:
4+
clouds:
5+
- kubernetes:
6+
maxRequestsPerHost: 64
7+
name: "openshift"

0 commit comments

Comments
 (0)