timeout during build

[fcami]

During a build:

***************************
2020-08-05 09:01:36,178    DEBUG  changed: [builder]
2020-08-05 09:01:36,184    DEBUG  
2020-08-05 09:01:36,185    DEBUG  TASK [builder/build : create directory for sources] ****************************
2020-08-05 09:01:36,751    DEBUG  changed: [builder]
2020-08-05 09:01:36,753    DEBUG  
2020-08-05 09:01:36,753    DEBUG  TASK [builder/build : pack the sources to tar.gz] ******************************
2020-08-05 09:01:45,058    DEBUG  changed: [builder]
2020-08-05 09:01:45,064    DEBUG  
2020-08-05 09:01:45,064    DEBUG  TASK [builder/build : create srpm] *********************************************
2020-08-05 09:17:59,448 CRITICAL  Terminating execution, runtime exceeded 1800s

I don't see how this step could take 17 minutes but unfortunately without more logs this is next to impossible to debug.

[f-trivino]

@fcami thanks for reporting, we are going to look into this by sending PR in the parallel infra.

[f-trivino]

@fcami how frequent is this issue happening? is it an isolated build? could you provide more details, steps on how to reproduce it?

[fcami]

I only saw that particular timeout "during build" once.
But we are seeing runners that seemingly pause or stop execution.
The controller recycles them when the job timeout is exceeded.
To my "ops" eyes it looks more like an infrastructure issue than anything else, as if a part of the VM would stop scheduling CPU time and not the whole VM (just because I don't see any SSH timeout) but that's really an unproven hunch.
And... the obvious conclusion of that hunch would be this is a nested virt bug.

Reproducing this is impossible, but we get that from time to time on any PR-CI instance.