pipelinesghubeus4.actions.githubusercontent.com failing intermitently

[moltar]

I am observing a really strange behaviour, where pipelinesghubeus4.actions.githubusercontent.com host cannot be reached randomly.

I've entered the pod and I am running curl manually, and it works about every second time.

I have read that a potential issue could be related to ipv6, so I have disabled ipv6 entirely at the network card level on the server.

I am still seeing the same.

Bare metal ifconfig:

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:22:65:52:95  txqueuelen 0  (Ethernet)
        RX packets 18565  bytes 11318888 (11.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20710  bytes 77042323 (77.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 95.X.X.X  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 90:X:b8  txqueuelen 1000  (Ethernet)
        RX packets 107959  bytes 146596865 (146.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25367  bytes 11431714 (11.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xf7200000-f7220000

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 256311  bytes 194943126 (194.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 256311  bytes 194943126 (194.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Runner pod ifconfig:

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:b2:13:9f:2a  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.143.136  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 2a:02:76:ba:e1:1b  txqueuelen 1000  (Ethernet)
        RX packets 5366  bytes 31140635 (31.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3747  bytes 278063 (278.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Traceroute from the bare metal:

traceroute to 20.232.252.48 (20.232.252.48), 30 hops max, 60 byte packets
 1  static.81.100.21.65.clients.your-server.de (65.21.100.81)  0.562 ms  0.513 ms  0.487 ms
 2  core32.hel1.hetzner.com (213.239.224.145)  0.385 ms core31.hel1.hetzner.com (213.239.224.141)  0.510 ms core32.hel1.hetzner.com (213.239.224.145)  0.554 ms
 3  core8.fra.hetzner.com (213.239.224.149)  20.179 ms core9.fra.hetzner.com (213.239.224.166)  20.081 ms core9.fra.hetzner.com (213.239.224.170)  20.224 ms
 4  core1.fra.hetzner.com (213.239.224.173)  20.286 ms core1.fra.hetzner.com (213.239.245.125)  20.429 ms core4.fra.hetzner.com (213.239.245.85)  20.463 ms
 5  ae72-0.fra-96cbe-1b.ntwk.msn.net (104.44.37.193)  22.874 ms  22.921 ms  22.904 ms
 6  ae22-0.icr02.fra23.ntwk.msn.net (104.44.230.20)  27.461 ms * *
 7  * * *
 8  be-12-0.ibr01.par21.ntwk.msn.net (104.44.17.172)  123.521 ms  123.501 ms *
 9  be-26-0.ibr01.mrs20.ntwk.msn.net (104.44.29.51)  123.413 ms  128.516 ms be-9-0.ibr02.gva20.ntwk.msn.net (104.44.30.123)  119.738 ms
10  be-34-0.ibr01.bio70.ntwk.msn.net (104.44.29.217)  125.079 ms be-6-0.ibr02.par21.ntwk.msn.net (104.44.18.6)  119.781 ms be-34-0.ibr01.bio70.ntwk.msn.net (104.44.29.217)  123.435 ms
11  be-3-0.ibr03.orf70.ntwk.msn.net (104.44.29.232)  124.365 ms be-1-0.ibr02.par30.ntwk.msn.net (104.44.7.215)  125.244 ms be-3-0.ibr03.orf70.ntwk.msn.net (104.44.29.232)  124.319 ms
12  be-3-0.ibr02.bio70.ntwk.msn.net (104.44.29.155)  120.956 ms  120.441 ms  119.788 ms
13  be-4-0.ibr04.orf70.ntwk.msn.net (104.44.29.234)  120.192 ms ae142-0.icr02.bl20.ntwk.msn.net (104.44.21.224)  140.713 ms *
14  be-4-0.ibr03.bl20.ntwk.msn.net (104.44.16.178)  125.111 ms be-5-0.ibr04.bl20.ntwk.msn.net (104.44.30.97)  120.993 ms *
15  * * ae164-0.icr03.bl20.ntwk.msn.net (104.44.32.43)  121.253 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Traceroute from a runner pod:

traceroute to 20.232.252.48 (20.232.252.48), 30 hops max, 60 byte packets
 1  95.X.X.X.kubernetes.default.svc.cluster.local (95.X.X.X)  0.541 ms  0.496 ms  0.472 ms
 2  * * *
 3  core31.hel1.hetzner.com (213.239.224.141)  1.193 ms core32.hel1.hetzner.com (213.239.224.145)  1.173 ms  1.145 ms
 4  core9.fra.hetzner.com (213.239.224.166)  20.228 ms core8.fra.hetzner.com (213.239.224.153)  20.197 ms core9.fra.hetzner.com (213.239.224.166)  20.174 ms
 5  core4.fra.hetzner.com (213.239.245.85)  20.436 ms core4.fra.hetzner.com (213.239.224.177)  20.464 ms  20.412 ms
 6  ae72-0.fra-96cbe-1b.ntwk.msn.net (104.44.37.193)  23.097 ms hetzner.fra-96cbe-1a.ntwk.msn.net (104.44.197.103)  26.283 ms  26.233 ms
 7  * ae22-0.icr01.fra23.ntwk.msn.net (104.44.230.16)  39.745 ms  39.696 ms
 8  * * *
 9  be-12-0.ibr01.par21.ntwk.msn.net (104.44.17.172)  124.194 ms  125.294 ms *
10  be-14-0.ibr02.lon24.ntwk.msn.net (104.44.30.110)  116.082 ms  114.930 ms be-26-0.ibr01.mrs20.ntwk.msn.net (104.44.29.51)  123.812 ms
11  be-1-0.ibr02.lon22.ntwk.msn.net (104.44.16.57)  116.379 ms be-1-0.ibr02.mrs20.ntwk.msn.net (104.44.7.28)  123.650 ms  123.754 ms
12  be-17-0.ibr02.bio70.ntwk.msn.net (104.44.29.219)  123.383 ms be-7-0.ibr02.nyc30.ntwk.msn.net (104.44.18.154)  114.974 ms  114.774 ms
13  * * be-4-0.ibr04.orf70.ntwk.msn.net (104.44.29.234)  123.571 ms
14  be-5-0.ibr04.bl20.ntwk.msn.net (104.44.30.97)  124.288 ms ae160-0.icr01.bl20.ntwk.msn.net (104.44.22.210)  114.442 ms  114.396 ms
15  ae166-0.icr04.bl20.ntwk.msn.net (104.44.32.47)  122.929 ms  122.935 ms *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

[savitha-qs]

We'd been running fine with ARC runners on GKE for the past couple of months, but we were hit by this problem today. Would appreciate someone taking a look at this, thanks!