Please improve CRD upgrade approach to avoid deletion/recreation

[tomhaynes]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

v0.12,v0.11,v0.9

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Follow the upgrade instructions for any of the mentioned releases.

Describe the bug

Requiring CRD removal / reinstallation causes a very large overhead / headache for engineering teams using these runners. This is removing the entire CI/CD framework, so the only re-course is to manually resolve this by hand. With a large, distributed estate this takes a lot of time and effort.

Describe the expected behavior

Please follow best practice with regards to CRD management, specifically versioning https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/

Best practice summary:

1. Never delete and reinstall a CRD in a production cluster. This leads to data loss.
2. For non-breaking changes (like adding new optional fields), you can often update the schema of an existing version directly. Always test these changes.
3. For any breaking changes, introduce a new version to the spec.versions list.
4. Implement a Conversion Webhook to handle the translation of objects between different API versions on the fly.

Additional Context

N/A

Controller Logs

N/A

Runner Pod Logs

N/A

[DPatrickBoyd]

I second this. I have to upgrade my cluster at interesting hours to keep outages and interruptions down

[nikola-jokic]

Hey @tomhaynes,

I agree with you, the upgrade procedure is rather painful. We will invest time into making it better, but we cannot rely on webhooks as one of our goals is not to use them. However, there are approaches we can take to handle it, and we should do it.

I moved this to a discussion since it is more of a request than it is a bug. Again, it is worth saying that you are completely right and the upgrade should be handled more appropriately.