When a test times-out, it sometimes jumps from runner to runner multiple times

[flo-renaud]

An example is freeipa/freeipa#4701 which was created May 18, 9:46 GMT+2 but was taken by multiple runners and is still running now on a different runner (Taken by small-runner-15 on 2020-05-19 13:14 UTC).

This is annoying as more than 1 day later the test is not finished. I would rather see the result as "Time-out".

[flo-renaud]

Now displaying as "Taken by small-runner-20 on 2020-05-19 14:17 UTC".

[flo-renaud]

Prev log on small-runner-15: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/f931a778-999d-11ea-aa63-fa163e8fbded/runner.log.gz

[f-trivino]

issue is fixed, I close this ticket.

[netoarmando]

I've been investigating this, but it's a moving target. Sometimes I think it's Github responding not the lastest status, I saw it a couple of times, although hard to reproduce.
This is the most strange example ("locked" in the description is from WIP changes:

The picture shows that there are cases where "pending for rerun" was added 1-5 min after a runner picked the job.

[bhavikbhavsar]

HI @netoarmando. What if we put a message broker in between runner and github instead of polling mechanism?

[netoarmando]

HI @netoarmando. What if we put a message broker in between runner and github instead of polling mechanism?

I'm drafting something around these lines, that can be an optional feature for PR-CI runners and use the prci-automation host for that, similar to Vagrant catalog.

In the meanwhile, I've opened #403 to improve the data we collect to troubleshoot this issue.