(workaround) codejail-service and all k8s pods are not seen by GoCD after coming up

[timmc-edx]

codejail-service deploys get stuck because a pod fails to come up within the readiness probe retries window. Pods that are deleted also fail to come up.

Sometimes pods do come up; we're not sure under which circumstances. It seems like k8s eventually replaces the broken pods

A/C:

• Give GoCD longer timeouts to allow an overloaded ArgoCD time to say "yes, the sync has finished". Monitor the timeouts to ensure the new value chosen is appropriately handling the failures AND minimizing a lengthy timeout.
• Possibly reduce priority level on deployment alerts (since they're less reliable)
• File a ticket for 1) reducing ArgoCD staleness and 2) then tightening up our deploy standards again (undoing the above -- search for code references to this ticket)

Notes:

• This problem wasn't occurring in the first month that codejail-service was deployed. But as of June 23 if we kill a pod, the replacement has successful startup checks within 6 seconds yet apparently fails to respond to readiness and liveness checks after 1-2 minutes.
• We tried increasing retry counts but it didn't help: https://github.com/edx/edx-internal/pull/12996
• We've also been experiencing some Datadog metrics and APM cutouts that have impeded diagnosing this. Unclear if related. (Infrastructure issue?)
• EKS upgrades seem to have fixed some issues with pod readiness/liveness probes.
• Our latest information (as of early July 2025) is that ArgoCD seems to just be overloaded and is providing stale information at times (to both GoCD and in the UI).

[timmc-edx]

I tried increasing the initial delay to 2 minutes, and that didn't help. I don't think that an even longer delay will help; something is just wrong with the pods or infra.

[robrap]

We're guessing/hoping that there is an infrastructure issue that might get resolved on the other ticket that would also resolve this. Blocked on:

• Datadog metrics cutouts in codejail-service #1077

[robrap]

• Codejail-service currently is getting data from DD, so considering this unblocked (for now).
• We need to confirm if there is still even an active issue, or if the DD reporting issue being resolved was a part of resolving other issues (because we don't yet know what the underlying issue was).

[jristau1984]

Confirmed this is still an active issue, and that increasing retries did not solve it. This ticket needs to be updated to reflect current state and what is still needed.

[timmc-edx]

Issue is updated to be a broader "codejail deploys are broken" ticket.

[timmc-edx]

Saad has bumped the EKS version in edge and thinks it has helped. (Prod and stage were bumped previously, and only edge was still showing this symptom.)

Now we're in a state where things look slow from GoCD's perspective (and the ArgoCD UI) but Saad can see the cluster coming up quickly when he looks via Freelens. He suspects it's actually ArgoCD being slow (overloaded?) and showing stale information.

EDIT: For reference, the dates of EKS Ubuntu upgrades from 1.31 (ami-00fd47edae472f9d6) to 1.32 (ami-093e9e9ebc6b33081):

• stage: 2025-06-25
• prod: 2025-06-27
• edge: 2025-07-08