Skip to content

Conversation

@boomanaiden154
Copy link
Contributor

@boomanaiden154 boomanaiden154 commented Jan 18, 2025

We are running into reliability problems with the kubernetes executor mode, hypothesized currently to be due to a complex interaction with konnectivity-agent pods dying and subsequently killing in-process execs through the k8s control plane. The plan is to switch back to the in-container executor mode for now while we sort out the issue upstream with the konnectivity developers.

This is related to #362.

@boomanaiden154
Copy link
Contributor Author

ci-ubuntu-22.04-agent currently does not exist yet. I need to put up another PR against the monorepo's container build job to generate it. I don't want to bundle the runner in with the normal container as it bloats the size by 40-50%. This can land once that has landed.

We are running into reliability problems with the kubernetes executor mode,
hypothesized currently to be due to a complex interaction with
konnectivity-agent pods dying and subsequently killing in-process execs through
the k8s control plane. The plan is to switch back to the in-container executor
mode for now while we sort out the issue upstream with the konnectivity
developers.
@boomanaiden154 boomanaiden154 force-pushed the linux-no-kubernetes-executor-mode branch from bc27ff2 to fec632c Compare January 21, 2025 01:51
@boomanaiden154
Copy link
Contributor Author

https://github.com/llvm/llvm-project/pkgs/container/ci-ubuntu-22.04-agent exists now, so we should be good to go to land this.

Once this lands I'll terraform apply and land llvm/llvm-project#123483 at the same time, and it should be a relatively smooth transition.

Copy link
Contributor

@Keenuts Keenuts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but as the other PR: please link the description to an llvm-zorg issue describing the reliability issues symptoms and the switch, including konnectivity issue link.

@boomanaiden154 boomanaiden154 merged commit 154b6a1 into llvm:main Jan 21, 2025
2 checks passed
@boomanaiden154 boomanaiden154 deleted the linux-no-kubernetes-executor-mode branch January 21, 2025 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants