Delayed runtime deletion in case of provisioning failure

[ebensom]

Description

Enhance the provisioning operation so that the Runtime CR deletion is delayed with a configurable duration, and performed in the background (i.e. the provisioning operation step just marks the instance for delayed Runtime CR deletion).

Reasons

Since January we face an increased level of provisioning failures where the Kyma SRE team and the Gardener team cannot investigate the root cause properly because the runtime shoot infrastrastructure gets deleted immediately after the operation fails. This is a major flaw impacting, long-term reliability. A few examples from the near past where proper RCA couldn't be done:

• https://github.tools.sap/kyma/backlog/issues/8658
• https://github.tools.sap/kyma/backlog/issues/8659
• https://github.tools.sap/kubernetes-live/issues-live/issues/8953

Attachments

[piotrmiskiewicz]

The idea of delayed Runtime CR deletion makes the process complicated. How long it should stay, 1-2hours? If longer, who will pay for that?

As I remember the root cause of a feature of deletion runtime CR if it is not ready in a given time: it could happen that provisioning takes longer than a timeout and we don't wan't to see Failed instance if cluster is Ready.

Maybe we have to copy the Shoot/Runtime CRs (make a dump) to provide it for investigations?

[ebensom]

We already have dump of Shoot and Runtime CRs. for proper RCA, the whole shoot infra should be retained for an extended period.

How long it should stay, 1-2hours?

It should be configurable, we will configure and tune the duration as necessary.

If longer, who will pay for that?

Kyma operations cost center, as assigned on all hyperscaler accounts.

[ebensom]

As I remember the root cause of a feature of deletion runtime CR if it is not ready in a given time: it could happen that provisioning takes longer than a timeout and we don't wan't to see Failed instance if cluster is Ready.

That is not the root cause. There is an underlying cause WHY the Runtime CR couldn't become ready. There are a lot more WHYs to answer, and until these are not answered, you cannot say "root" cause.

1. Why the Runtime CR couldn't become ready? >> Because Shoot couldn't become Succeeded in sufficient amount of time
2. Why the Gardener Shoot couldn't become ready? >> If shoot infrastructure is deleted, all evidences, including gardener monitoring and logs data are deleted and nobody can tell. If nobody can tell, nobody can have proper enhancements to either the cloud provider infrastructure or Gardener to make the shoot creation more reliable.

[piotrmiskiewicz]

Idea of an implementation:

1. In case of timeout or error in the Runtime CR KEB adds a special Label(s) to the runtime cr, for example: "to-delete-at: 2026-02-31 11:59"
2. A job watches runtime CRs and if there is an item for deletion - deletes it

[MarekMichali]

We should think about adding more security layers, as this would be a job that deletes instances on prod. IMHO one label is not enough. We should at least also check the database to verify if the instance was deprovisioned.

[PK85]

Tbh that time where we will not delete could be again exceeded, and we will dont know why Runtime CR was not in Ready state. I think we should stop deleting that (on KEB side), and then deleting a runtime CR will be on SRE or Frogs duty when checking why was already done.

Update:
We can label such RubtimeCr. That will be easier to monitor such ones, and you will know which to delete after investigation.

[PK85]

@ebensom @tobiscr I will organise a meeting with you two. And we need to decide how it should be resolved. I don't know to have that logic in KEB, it is not our resposibility.

[PK85]

1. After a meeting Mar 6th 2026 with @akgalwas @tobiscr @ebensom @piotrmiskiewicz and myself we decided that KEB will stop deleting RuntimeCr in this scenario and will apply a label on that RuntimeCR saying that provisioning is permanently failed from user perspective.

2. @tobiscr will create a ticket on KIM side, to make sure such a RuntimeCR is always NOT billable. They will provide a way to see and send alerts about such situations. Later on create a ticket for gardener to investigate. Than ask SRE to delete such RuntimeCr.

3. When KIM task is resolved and deployed on all landscapes including restricted markets, then KEB part would be released, ref: 1)

[tobiscr]

Issue on KIM side: kyma-project/kyma-infrastructure-manager#1342