CI often fails with "Could not resolve host: github.com"

[evgenyrp]

If it's expected that it can fail we should add retries to all our steps:

[vcs 2024-04-30T01:09:53.281Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.com

Here is an example task: https://firefox-ci-tc.services.mozilla.com/tasks/JdDA-zYDQnG4166Zbsqq6w

[bhearsum]

I'm asking around other projects to see if they're seeing this as well.

[bhearsum]

Haven't seen reports of this elsewhere.

@eu9ene - have you seen this on GPU workers only? Or also on the CPU workers?

[evgenyrp]

Haven't seen reports of this elsewhere.

@eu9ene - have you seen this on GPU workers only? Or also on the CPU workers?

I'm not sure but I feel like I've been seeing this in random places.

[gregtatum]

Here's an instance on a GPU worker: https://firefox-ci-tc.services.mozilla.com/tasks/XRYrb2BxTOyJa209jrLQ7A/runs/0/logs/public/logs/live.log

[bhearsum]

Thanks; so it seems very unlikely to be related to specific worker images.

@aerickson - I don't suppose you have any idea what's going on here?

[aerickson]

@bhearsum I'm not sure what's going on. Translations GPU workers on GCP should use GCP's DNS servers (provided by DHCP) and the Snakepit worker use our internal infoblox servers (configured in dnsmasq). It seems like a network blip or perhaps the DNS server could be overloaded for a second?

I haven't heard about any Github outages around DNS. I've never really heard of DNS outages (it's a pretty resilient service/protocol).

If we find a concentrated event or location let me know and I'll dig in some more.

[bhearsum]

We do see these fairly often - I would say maybe on 5-10% of the tasks run. I'll try to collect some data to help us analyze this better.

[bhearsum]

Here's failures by worker group:

defaultdict(<class 'int'>,
            {'us-central1': 7,
             'us-central1-a': 9,
             'us-central1-b': 7,
             'us-central1-c': 7,
             'us-central1-f': 9,
             'us-west1': 5,
             'us-west1-a': 5,
             'us-west1-b': 11})

And here's timestamps when we hit the failures:

['2024-01-19T21:20:34.616Z',
 '2024-02-02T20:02:49.997Z',
 '2024-02-22T16:07:57.737Z',
 '2024-02-26T23:13:59.353Z',
 '2024-02-28T00:36:35.015Z',
 '2024-02-28T17:48:06.906Z',
 '2024-02-29T15:03:54.666Z',
 '2024-03-06T14:32:56.066Z',
 '2024-03-06T18:10:27.186Z',
 '2024-03-21T15:03:54.430Z',
 '2024-03-21T20:29:42.584Z',
 '2024-03-26T13:29:20.434Z',
 '2024-03-30T22:11:50.010Z',
 '2024-04-01T14:10:30.644Z',
 '2024-04-01T15:18:22.619Z',
 '2024-04-09T10:47:13.166Z',
 '2024-04-18T12:41:01.952Z',
 '2024-04-22T19:58:27.752Z',
 '2024-04-23T19:46:39.366Z',
 '2024-04-24T17:57:31.559Z',
 '2024-04-25T07:33:55.024Z',
 '2024-04-29T17:55:15.937Z',
 '2024-04-30T00:38:45.847Z',
 '2024-04-30T01:08:46.769Z',
 '2024-04-30T14:06:18.417Z',
 '2024-04-30T18:56:31.177Z',
 '2024-05-01T23:21:11.540Z',
 '2024-05-02T00:22:43.542Z',
 '2024-05-06T14:40:00.719Z',
 '2024-05-06T16:41:19.051Z',
 '2024-05-07T00:01:30.432Z',
 '2024-05-07T21:23:47.145Z',
 '2024-05-07T21:36:54.706Z',
 '2024-05-09T19:01:17.356Z',
 '2024-05-09T22:10:58.358Z',
 '2024-05-09T22:11:47.850Z',
 '2024-05-10T18:42:00.163Z',
 '2024-05-10T22:14:23.709Z',
 '2024-05-13T23:01:23.656Z',
 '2024-05-13T23:24:46.266Z',
 '2024-05-14T14:49:27.875Z',
 '2024-05-14T15:08:43.843Z',
 '2024-05-14T23:18:39.852Z',
 '2024-05-14T23:43:02.154Z',
 '2024-05-15T01:01:44.678Z',
 '2024-05-15T14:30:49.732Z',
 '2024-05-15T23:20:08.322Z',
 '2024-05-16T21:56:58.285Z',
 '2024-05-17T11:38:30.465Z',
 '2024-05-17T14:02:32.250Z',
 '2024-05-17T16:18:16.690Z',
 '2024-05-17T17:51:07.523Z',
 '2024-05-17T20:37:44.360Z',
 '2024-05-17T20:49:01.813Z',
 '2024-05-17T20:49:18.471Z',
 '2024-05-17T21:36:37.947Z',
 '2024-05-17T22:05:41.152Z',
 '2024-05-17T22:06:42.275Z',
 '2024-05-17T23:11:32.583Z',
 '2024-05-20T15:29:29.312Z']

And by worker image:

defaultdict(<class 'int'>, {'gpu': 58, 'cpu': 2})

Clearly the most notable part here here is that we're seeing more issues on GPU images. And within that, we seem to have gotten more beginning in late April/early May. We added a test pool with a new image on April 22nd (that I was running a lot to test things) in https://phabricator.services.mozilla.com/D208202. That image went to production on May 8th in https://phabricator.services.mozilla.com/D209840.

mozilla-platform-ops/monopacker#140 was the PR related to this image, but I don't know how deterministic the other parts are? Eg: could we have picked up a change to a system package that is now causing problems?

@aerickson - do you have any thoughts? If we still have the old image, maybe we could poke around and compare the new to old one? (I'd be happy to do this if you want.)

[evgenyrp]

Ok, it happens every single pipeline run for me now and does not restart. Marking as blocker.