Internal Server Error: /cleanup #53
Description
Bug Description
During the certificate request flow, we received an error on the client (httprequest-lego-k8s):
2024-04-10T13:02:41.188Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log 2024/04/10 13:02:40 [WARN] [internal-redacted-url.canonical.com] acme: cleaning up failed: httpreq: unexpected status code: [status code: 500] body: <!doctype html>
2024-04-10T13:02:41.190Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log <html lang="en">
2024-04-10T13:02:41.193Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log <head>
2024-04-10T13:02:41.195Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log <title>Server Error (500)</title>
2024-04-10T13:02:41.197Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log </head>
2024-04-10T13:02:41.200Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log <body>
2024-04-10T13:02:41.203Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log <h1>Server Error (500)</h1><p></p>
2024-04-10T13:02:41.206Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log </body>
2024-04-10T13:02:41.208Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log </html>
2024-04-10T13:02:41.212Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log 2024/04/10 13:02:40 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/REDACTED
2024-04-10T13:02:41.214Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log 2024/04/10 13:02:41 Could not obtain certificates:
2024-04-10T13:02:41.217Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log error: one or more domains had a problem:
2024-04-10T13:02:41.220Z [container-agent] 2024-04-10 13:02:41 ERROR juju-log [internal-redacted-url.canonical.com] [internal-redacted-url.canonical.com] acme: error presenting token: httpreq: unable to communicate with the API server: error: Post "https://internal-redacted-lego-url.canonical.com/present": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
To Reproduce
- kubectl delete pod traefik-ingress-0 # Deleting the tls_certificates pod retriggers the flow.
Environment
Canonical Prodstack 6:
- httprequest-lego-provider: prod-lego-certs-k8s@is-bastion-ps6 (latest/edge 22)
- httprequest-lego-k8s: prod-is-cos@is-bastion-ps6 (latest/edge 68)
Relevant log output
https://pastebin.canonical.com/p/ss2zHv8xnX/Additional context
Per DNS repository log, it looks like the DNS entry already existed previously and was not properly cleaned up:
https://git.launchpad.net/canonical-is-dns-configs/log/canonical.com.domain?h=lego
Metadata
Metadata
Assignees
Labels
No labels