jobs/build: increase timeout when WAIT_FOR_RELEASE_JOB set

Our default 240 minute timeout won't be enough when
`WAIT_FOR_RELEASE_JOB` is set since the release job itself needs to wait
for all the build-arch jobs to finish before proceeding. Set the timeout
to a higher value accordingly.

A downside with this is that it increases the timeout for all the other
bits of the job too. Notably cloud uploads are susceptible to hangs if
the infra is flaky (as it recently happened with Alibaba). To counter
this, add a timeout to cloud uploads. I chose 45 mins after looking at
past times in the available build history; the longest cloud uploads
ever took was 23 mins, so 45 mins should be a safe value. We can add
more scopes elsewhere in the future if we feel it's warranted.

Author: jlebon

jobs/build-arch.Jenkinsfile

@@ -96,6 +96,9 @@ assert params.VERSION != ""
 def newBuildID = params.VERSION
 def basearch = params.ARCH
 
+// matches between build/build-arch job
+def timeout_mins = 240
+
 // release lock: we want to block the release job until we're done.
 // ideally we'd lock this from the main pipeline and have lock ownership
 // transferred to us when we're triggered. in practice, it's very unlikely the
@@ -104,7 +107,7 @@ lock(resource: "release-${params.VERSION}-${basearch}") {
 // build lock: we don't want multiple concurrent builds for the same stream and
 // arch (though this should work fine in theory)
 lock(resource: "build-${params.STREAM}-${basearch}") {
-    timeout(time: 240, unit: 'MINUTES') {
+    timeout(time: timeout_mins, unit: 'MINUTES') {
     cosaPod(cpu: "${ncpus}",
             memory: "${cosa_memory_request_mb}Mi",
             image: cosa_controller_img,

jobs/build.Jenkinsfile

@@ -109,8 +109,20 @@ currentBuild.description = "${build_description} Waiting"
 // declare these early so we can use them in `finally` block
 def newBuildID, basearch
 
+// matches between build/build-arch job
+def timeout_mins = 240
+
+if (params.WAIT_FOR_RELEASE_JOB) {
+    // Waiting for the release job effectively means waiting for all the build-
+    // arch jobs we trigger to finish. While we do overlap in execution (by
+    // a lot when EARLY_ARCH_JOBS is set), let's just simplify and add its
+    // timeout value to ours to account for this. Add 30 minutes more for the
+    // release job itself.
+    timeout_mins += timeout_mins + 30
+}
+
 lock(resource: "build-${params.STREAM}") {
-    timeout(time: 240, unit: 'MINUTES') {
+    timeout(time: timeout_mins, unit: 'MINUTES') {
     cosaPod(cpu: "${ncpus}",
             memory: "${cosa_memory_request_mb}Mi",
             image: cosa_img,

libcloud.groovy

@@ -304,6 +304,9 @@ def upload_to_clouds(pipecfg, basearch, buildID, stream) {
     }
 
     // Run the resulting set of uploaders in parallel
-    parallel uploaders
+    // It shouldn't take more than 45 minutes.
+    timeout(time: 45, unit: 'MINUTES') {
+        parallel uploaders
+    }
 }
 return this