Runitor unable to ping healthchecks but curl works fine

[ep1cman]

I am having a strange issue as of this morning. When I try to ping a self hosted healthchecks instance with runitor it fails:

root@Neon:/tmp/backup# runitor --api-url https://healthcheck.mydomain.com/ping  -uuid 3e5b2cea-7e7c-4eaa-be46-184f94d319e0 echo hello_world
2021/09/18 14:49:37 PingStart: max tries reached after try 3. last error: Post "https://healthcheck.mydomain.com/ping/3e5b2cea-7e7c-4eaa-be46-184f94d319e0/start": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
hello_world
2021/09/18 14:49:58 PingSuccess: max tries reached after try 3. last error: Post "https://healthcheck.mydomain.com/ping/3e5b2cea-7e7c-4eaa-be46-184f94d319e0": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

If I try to do the same thing manually with curl it works just fine:

root@Neon:/tmp/backup# curl https://healthcheck.mydomain.com/ping/3e5b2cea-7e7c-4eaa-be46-184f94d319e0/start
OK
root@Neon:/tmp/backup# curl https://healthcheck.mydomain.com/ping/3e5b2cea-7e7c-4eaa-be46-184f94d319e0/
OK

Do you have any suggestions on how I could debug/resolve this? It has me totally stumped

[ep1cman]

Ah figured it out, its the timeout, bumping it up from 5 even to 6 fixed it. I have no idea why the API is being so slow, 5s should be enough to make 100s of requests locally. I will close this now, if I find a reason I will report back.

[bdd]

Assuming you're observing similar request-response latency when using curl, maybe this little helper I lug around with my dotfiles may help you.

https://github.com/bdd/.dotfiles/blob/main/dot.local/bin/curltime

Example output:

~% curltime -o /dev/null https://hc-ping.com/<uuid>
resolve     0.020542
tcp rts     0.180493
tls rts     0.428618 (fail=0)
req sent    0.428873 (111 bytes)
redir       0.000000 (0 redirs) 
first byte  0.588803 (HTTP 200)
         Σ  0.589035 (1 connect(s), rx: 2 bytes / tx: 0 bytes)

Above output tells us, it took 20ms to resolve the name, on the 180th millisecond (160ms after resolving) TCP 3-way handshake was complete and curl was ready to send; 428 ms after the command TLS handshake was done; the very same time request was sent; and finally 588ms after initial command, client read the first byte of the response from the socket.

Hopefully this little curl wrapper will help you debug the where exactly the issue lies.