Add blog for stale controller mitigation#54792

Open

michaelasp wants to merge 1 commit intokubernetes:mainfrom

michaelasp:staleBlog

Contributor

michaelasp commented Mar 6, 2026 •

edited

Loading

Description

Issue

Ref: kubernetes/enhancements#5647

This blog goes over the new stale controller mitigations we have added in 1.36 and explains the feature gates and changes done.

k8s-ci-robot added the area/blog label

Contributor

k8s-ci-robot commented Mar 6, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign natalisucks for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

content/en/blog/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from Gauravpadam and lmktfy

March 6, 2026 17:18

k8s-ci-robot added language/en cncf-cla: yes size/XS labels

michaelasp force-pushed the staleBlog branch from 445a2e6 to ecdd781 Compare

March 6, 2026 17:20

michaelasp mentioned this pull request

Stale Controller Mitigation kubernetes/enhancements#5647

Open

17 tasks

michaelasp force-pushed the staleBlog branch from ecdd781 to 625dcc8 Compare

March 6, 2026 17:26

k8s-ci-robot added size/S and removed size/XS labels

netlify bot commented Mar 6, 2026

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`445a2e6`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-io-main-staging/deploys/69ab0c45f6219a00087b125e
😎 Deploy Preview	https://deploy-preview-54792--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify bot commented Mar 6, 2026 •

edited

Loading

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`372be5d`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-io-main-staging/deploys/69d44785bd216c000702abcc
😎 Deploy Preview	https://deploy-preview-54792--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

michaelasp changed the title ~~Add blog for stale controller mitigation~~ WIP: Add blog for stale controller mitigation

k8s-ci-robot added the do-not-merge/work-in-progress label

Contributor

chadmcrowell commented Mar 9, 2026

Hi @michaelasp 👋 v1.36 Communications team here,

@lukasmetzner as author of #54693, I'd like you to be a writing buddy for @michaelasp on this PR.

Please:

Review this PR, paying attention to the guidelines and review hints
Update your own PR based on any best practices you identify that should be applied
Remember to be compassionate with your fellow article author

chadmcrowell mentioned this pull request

feature-blog(ccm): new metric route_sync_total #54693

Open

michaelasp force-pushed the staleBlog branch from 625dcc8 to 4deecbf Compare

March 26, 2026 21:46

k8s-ci-robot added size/M and removed size/S labels

michaelasp force-pushed the staleBlog branch from 4deecbf to efc7637 Compare

March 26, 2026 21:47

michaelasp changed the title ~~WIP: Add blog for stale controller mitigation~~ Add blog for stale controller mitigation

k8s-ci-robot removed the do-not-merge/work-in-progress label

michaelasp commented

View reviewed changes

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

serathius reviewed

View reviewed changes

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Show resolved Hide resolved

Contributor

lukasmetzner commented Mar 31, 2026

Hey @michaelasp, I am your assigned writing buddy. The blog article looks good to me apart from the open thread above, where I agree on adding a basic example.

sbueringer mentioned this pull request

✨ Read your own write client design kubernetes-sigs/controller-runtime#3473

Open

michaelasp force-pushed the staleBlog branch from efc7637 to fddd23f Compare

March 31, 2026 18:00

k8s-ci-robot removed the size/M label

k8s-ci-robot added the size/L label

alvaroaleman reviewed

View reviewed changes

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated

+              An informer author can use the `ConsistencyStore` to track the latest resource version of the objects that the informer cares about.
+              It provides 3 main functions:
+. `consistencyStore.WroteAt(owningObj, UID, groupResource, resourceVersion)`: This function is called by the controller when it writes to the API server for an object.

Member

alvaroaleman Apr 1, 2026

I find this pseudo code confusing, because the parameter names are sometimes a description like owningObject and sometypes the type like UID, I think maybe just using go code to have both a parameter name and a type would be better

Contributor Author

michaelasp Apr 1, 2026

I just dropped in the interface we provide here, I think that should work.

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

michaelasp force-pushed the staleBlog branch from fddd23f to 169694c Compare

April 1, 2026 17:27

michaelasp requested a review from alvaroaleman

April 1, 2026 17:27

Contributor

serathius commented Apr 1, 2026

/cc @jpbetz @liggitt

k8s-ci-robot requested review from jpbetz and liggitt

April 1, 2026 18:30

lmktfy reviewed

View reviewed changes

Member

lmktfy left a comment

Thanks.

avoid using "we"
for official post-repease Comms I do recommend aligning to the style guide
I think the feature gate names might be wrong in one place

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated

+              ### client-go improvements
+              In client-go, we have added the `AtomicFIFO` feature on top of the existing RealFIFO queue implementation. Doing so allows for
+              the queue to atomically handle operations that batch events such as the List portion of the ListWatch operation. This ensures

Member

lmktfy Apr 5, 2026

Suggested change

      
            the queue to atomically handle operations that batch events such as the List portion of the ListWatch operation. This ensures
          
            the queue to atomically handle operations that batch events such as the list portion of the `ListWatch()` operation. This ensures

I recommend the parents to imply it's a function you can call. Have I got that right? The actual verb is watch.

Contributor Author

michaelasp Apr 6, 2026

It's based on the list->watch pattern that informers do that has recently been updated to streaming lists https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists

It doesn't really matter which you use. I updated this section a bit to make this more clear.

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated


		### Metrics

		We have added the following alpha metrics to kube-controller-manager:

Member

lmktfy Apr 5, 2026

Suggested change

We have added the following alpha metrics to kube-controller-manager:

Nit: could hyperlink to the relevant entry in the metrics reference docs.

Contributor Author

michaelasp Apr 6, 2026

Those are auto generated right? How should I best list this? I don't see a specific way to link any better than https://kubernetes.io/docs/reference/instrumentation/metrics/#list-of-alpha-kubernetes-metrics

Added that link for now since it matches the alpha metrics line.

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

Contributor Author

michaelasp commented Apr 6, 2026

for official post-repease Comms I do recommend aligning to the style guide

Let me look through that along with your review and update this, thanks for the pass!

michaelasp commented

View reviewed changes

Contributor Author

michaelasp left a comment

Thanks for the review, this should address most of your comments. Looking through the style guide to see if anything else was missed.

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated

+              ### client-go improvements
+              In client-go, we have added the `AtomicFIFO` feature on top of the existing RealFIFO queue implementation. Doing so allows for
+              the queue to atomically handle operations that batch events such as the List portion of the ListWatch operation. This ensures

Contributor Author

michaelasp Apr 6, 2026

It's based on the list->watch pattern that informers do that has recently been updated to streaming lists https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists

It doesn't really matter which you use. I updated this section a bit to make this more clear.

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated


		### Metrics

		We have added the following alpha metrics to kube-controller-manager:

Contributor Author

michaelasp Apr 6, 2026

Those are auto generated right? How should I best list this? I don't see a specific way to link any better than https://kubernetes.io/docs/reference/instrumentation/metrics/#list-of-alpha-kubernetes-metrics

Added that link for now since it matches the alpha metrics line.

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved

content/en/blog/_posts/2026/2026-03-26-staleness-mitigation-for-controllers/index.md Outdated Show resolved Hide resolved


          Add blog for stale controller mitigation

372be5d

michaelasp force-pushed the staleBlog branch from 169694c to 372be5d Compare

April 6, 2026 23:53

michaelasp mentioned this pull request

KEP-5647: Stale informer docs #52901

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

lmktfy lmktfy left review comments

Gauravpadam Awaiting requested review from Gauravpadam

alvaroaleman Awaiting requested review from alvaroaleman

jpbetz Awaiting requested review from jpbetz

liggitt Awaiting requested review from liggitt

+2 more reviewers

serathius serathius left review comments

lukasmetzner lukasmetzner left review comments

Labels

area/blog cncf-cla: yes language/en size/L