[Fleet] Implement agent upgrade rollback by jillguyonnet · Pull Request #247398 · elastic/kibana

jillguyonnet · 2025-12-23T16:18:38Z

Summary

Closes https://github.com/elastic/ingest-dev/issues/6212

This PR adds Fleet support for Elastic Agent upgrade rollback (main issue: https://github.com/elastic/ingest-dev/issues/3971).

Key points:

Feature is behind enableAgentRollback feature flag
Feature is only available for Enterprise license
New single agent upgrade rollback API endpoint: POST /api/fleet/agents/{agentId}/rollback
New bulk agent upgrade rollback API endpoint: POST /api/fleet/agents/bulk_rollback
Upgrade rollback generates an UPGRADE type agent action with rollback: true and the rollback version set by the agent
For multiple agents, one action per rollback version is generated
Single agent and bulk upgrade rollback available from Fleet UI

Screenshots

Single agent confirmation modal:

Multiple agents confirmation modal:

Action result example:

If action was correctly created and rollback failed, it shows in the agent's upgrade details:

Testing

Properly testing agent rollback is not straightforward right now as the feature is new and requires a "rollbackable" upgrade. See elastic/elastic-agent#11143 for steps.

Alternatively, the agent document can be directly edited with fake upgrade rollback information in order to test that Fleet behaves correctly (the action will fail in this case).

Scenarios to check:

Single agent
- If no upgrade rollback, UI element should be disabled and API request should fail fast
- Same if agent has expired rollback
- If agent has a valid rollback, an UPGRADE type action should be created with rollback: true and the correct rollback version
Multiple agents
- Should work with list of agent ids (manual selection in the UI)
- Should work with agent kuery (select all in the UI)
- If agents have different rollback versions, there should be one action per version
- Agents not found and agents with no (valid) rollback should generate corresponding action errors

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

Identify risks

This is a new experimental feature behind the enableAgentRollback feature flag and leaves existing flows largely untouched.

Release note

Adds capability for rolling back a recent upgrade of a Fleet-managed Elastic Agent upgrade using Fleet UI or API.

kibanamachine · 2025-12-24T14:15:16Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10238

[✅] x-pack/platform/test/fleet_api_integration/config.agent.ts: 25/25 tests passed.

see run history

elasticmachine · 2025-12-24T15:08:01Z

Pinging @elastic/fleet (Team:Fleet)

x-pack/platform/plugins/shared/fleet/server/types/rest_spec/agent.ts

+        schema.arrayOf(
+          schema.object({
+            valid_until: schema.string(),
+            version: schema.string(),
+          })
+        )


x-pack/platform/plugins/shared/fleet/server/types/rest_spec/agent.ts

+
+export const PostBulkAgentRollbackRequestSchema = {
+  body: schema.object({
+    agents: schema.oneOf([schema.arrayOf(schema.string()), schema.string()]),


x-pack/platform/plugins/shared/fleet/server/types/rest_spec/agent.ts

+};
+
+export const PostBulkAgentRollbackResponseSchema = schema.object({
+  actionIds: schema.arrayOf(schema.string()),


nchaulet · 2026-01-02T14:45:21Z

@elasticmachine merge upstream

nchaulet · 2026-01-02T18:31:37Z

x-pack/platform/plugins/shared/fleet/server/routes/agent/rollback_handlers.ts

+  if (agent.unenrollment_started_at || agent.unenrolled_at) {
+    throw new AgentRollbackError('cannot roll back an unenrolling or unenrolled agent');
+  }
+
+  if (isAgentUpgrading(agent)) {
+    throw new AgentRollbackError('cannot roll back an upgrading agent');
+  }


wdyt of moving this to the application service sendRollbackAgentAction this way it will be safer to use by other plugins

jillguyonnet · 2026-01-05T08:16:09Z

@elasticmachine merge upstream

florent-leborgne

LGTM for API docs and a few copy nits

florent-leborgne · 2026-01-05T09:00:29Z

...ed/fleet/public/applications/fleet/sections/agents/components/agent_rollback_modal/index.tsx

+        {isSingleAgent ? (
+          <FormattedMessage
+            id="xpack.fleet.rollbackAgents.singleAgentDescription"
+            defaultMessage="This action will roll back the agent running on ''{hostName}'' to version {version}. This action can not be undone. Are you sure you wish to continue?"


Suggested change

defaultMessage="This action will roll back the agent running on ''{hostName}'' to version {version}. This action can not be undone. Are you sure you wish to continue?"

defaultMessage="You are about to roll back the agent running on ''{hostName}'' to version {version}. This action cannot be undone."

We usually don't need the "are you sure" part if the confirm/cancel actions are clearly presented

Thanks! FYI in case this should be reviewed at some stage, there seems to be 27 existing occurrences of "are you sure" in Fleet codebase.

florent-leborgne · 2026-01-05T09:01:30Z

...ed/fleet/public/applications/fleet/sections/agents/components/agent_rollback_modal/index.tsx

+        ) : (
+          <FormattedMessage
+            id="xpack.fleet.rollbackAgents.multipleAgentsDescription"
+            defaultMessage="This action will roll back agents to their previous version. This action can not be undone. Are you sure you wish to continue?"


Suggested change

defaultMessage="This action will roll back agents to their previous version. This action can not be undone. Are you sure you wish to continue?"

defaultMessage="You are about to roll back agents to their previous version. This action cannot be undone."

florent-leborgne · 2026-01-05T09:04:34Z

...shared/fleet/public/applications/fleet/sections/agents/hooks/use_single_agent_menu_items.tsx

+                  name: (
+                    <FormattedMessage
+                      id="xpack.fleet.agentList.rollbackOneButton"
+                      defaultMessage="Rollback"


Suggested change

defaultMessage="Rollback"

defaultMessage="Roll back"

Other buttons next to this one seem to be verbs as well

nchaulet

🚀

elasticmachine · 2026-01-06T15:18:30Z

💚 Build Succeeded

Buildkite Build
Commit: b196d13

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	1384	1385	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1565	1570	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	2.1MB	2.1MB	+3.4KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`fleet`	111	112	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	180.1KB	180.9KB	+877.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1698	1704	+6

History

💛 Build #378182 was flaky 58fc4cb
💔 Build #378172 failed be5cba2
💔 Build #378167 failed 46b6596
💛 Build #377850 was flaky 29ff274
💔 Build #377806 failed 4d27cc1
💛 Build #377534 was flaky 4738c68

cc @jillguyonnet

pmuellr

I'm not seeing any new usage of TaskManager APIs, and given this is using a framework (ActionRunner) around TM, I'd assume this is ok. LMK if otherwise.

One concern I have is not using our provided AbortController in the task runners, so check to see if the task has been cancelled during it's run.

jillguyonnet · 2026-01-06T16:34:09Z

Thanks for your review @pmuellr

One concern I have is not using our provided AbortController in the task runners, so check to see if the task has been cancelled during it's run.

I had a look and this looks like a implementation gap we can address. I filed a tech debt issue for it: #247971

## Summary Closes elastic/ingest-dev#6212 This PR adds Fleet support for Elastic Agent upgrade rollback (main issue: elastic/ingest-dev#3971). Key points: * Feature is behind `enableAgentRollback` feature flag * Feature is only available for Enterprise license * New single agent upgrade rollback API endpoint: `POST /api/fleet/agents/{agentId}/rollback` * New bulk agent upgrade rollback API endpoint: `POST /api/fleet/agents/bulk_rollback` * Upgrade rollback generates an `UPGRADE` type agent action with `rollback: true` and the rollback version set by the agent * For multiple agents, one action per rollback version is generated * Single agent and bulk upgrade rollback available from Fleet UI ### Screenshots Single agent confirmation modal: <img width="1919" height="773" alt="Screenshot 2025-12-23 at 16 06 09" src="https://github.com/user-attachments/assets/6b3948b6-b530-4c18-b2b8-8763234ae75d" /> Multiple agents confirmation modal: <img width="1919" height="773" alt="Screenshot 2025-12-23 at 16 02 29" src="https://github.com/user-attachments/assets/1b7e36b2-7c67-4b17-aae5-75a86fb7e015" /> Action result example: <img width="1919" height="881" alt="Screenshot 2025-12-23 at 16 01 24" src="https://github.com/user-attachments/assets/f84eb8bb-fc55-4fb7-ac22-48042da87718" /> If action was correctly created and rollback failed, it shows in the agent's upgrade details: <img width="1919" height="881" alt="Screenshot 2025-12-23 at 15 40 41" src="https://github.com/user-attachments/assets/3aa51fcb-12ac-4c59-920f-6d685605fe09" /> ### Testing Properly testing agent rollback is not straightforward right now as the feature is new and requires a "rollbackable" upgrade. See elastic/elastic-agent#11143 for steps. Alternatively, the agent document can be directly edited with fake upgrade rollback information in order to test that Fleet behaves correctly (the action will fail in this case). Scenarios to check: * Single agent * If no upgrade rollback, UI element should be disabled and API request should fail fast * Same if agent has expired rollback * If agent has a valid rollback, an `UPGRADE` type action should be created with `rollback: true` and the correct rollback version * Multiple agents * Should work with list of agent ids (manual selection in the UI) * Should work with agent kuery (select all in the UI) * If agents have different rollback versions, there should be one action per version * Agents not found and agents with no (valid) rollback should generate corresponding action errors ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks This is a new experimental feature behind the `enableAgentRollback` feature flag and leaves existing flows largely untouched. ## Release note Adds capability for rolling back a recent upgrade of a Fleet-managed Elastic Agent upgrade using Fleet UI or API. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

jillguyonnet · 2026-01-15T15:35:39Z

💚 All backports created successfully

Status	Branch	Result
✅	9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary Closes elastic/ingest-dev#6212 This PR adds Fleet support for Elastic Agent upgrade rollback (main issue: elastic/ingest-dev#3971). Key points: * Feature is behind `enableAgentRollback` feature flag * Feature is only available for Enterprise license * New single agent upgrade rollback API endpoint: `POST /api/fleet/agents/{agentId}/rollback` * New bulk agent upgrade rollback API endpoint: `POST /api/fleet/agents/bulk_rollback` * Upgrade rollback generates an `UPGRADE` type agent action with `rollback: true` and the rollback version set by the agent * For multiple agents, one action per rollback version is generated * Single agent and bulk upgrade rollback available from Fleet UI ### Screenshots Single agent confirmation modal: <img width="1919" height="773" alt="Screenshot 2025-12-23 at 16 06 09" src="https://github.com/user-attachments/assets/6b3948b6-b530-4c18-b2b8-8763234ae75d" /> Multiple agents confirmation modal: <img width="1919" height="773" alt="Screenshot 2025-12-23 at 16 02 29" src="https://github.com/user-attachments/assets/1b7e36b2-7c67-4b17-aae5-75a86fb7e015" /> Action result example: <img width="1919" height="881" alt="Screenshot 2025-12-23 at 16 01 24" src="https://github.com/user-attachments/assets/f84eb8bb-fc55-4fb7-ac22-48042da87718" /> If action was correctly created and rollback failed, it shows in the agent's upgrade details: <img width="1919" height="881" alt="Screenshot 2025-12-23 at 15 40 41" src="https://github.com/user-attachments/assets/3aa51fcb-12ac-4c59-920f-6d685605fe09" /> ### Testing Properly testing agent rollback is not straightforward right now as the feature is new and requires a "rollbackable" upgrade. See elastic/elastic-agent#11143 for steps. Alternatively, the agent document can be directly edited with fake upgrade rollback information in order to test that Fleet behaves correctly (the action will fail in this case). Scenarios to check: * Single agent * If no upgrade rollback, UI element should be disabled and API request should fail fast * Same if agent has expired rollback * If agent has a valid rollback, an `UPGRADE` type action should be created with `rollback: true` and the correct rollback version * Multiple agents * Should work with list of agent ids (manual selection in the UI) * Should work with agent kuery (select all in the UI) * If agents have different rollback versions, there should be one action per version * Agents not found and agents with no (valid) rollback should generate corresponding action errors ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks This is a new experimental feature behind the `enableAgentRollback` feature flag and leaves existing flows largely untouched. ## Release note Adds capability for rolling back a recent upgrade of a Fleet-managed Elastic Agent upgrade using Fleet UI or API. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit f4ab534)

# Backport This will backport the following commits from `main` to `9.3`: - [[Fleet] Implement agent upgrade rollback (#247398)](#247398)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

## Summary Make `enableAgentRollback` feature flag `true` by default. Implementation PR: #247398 Backport to 9.3: #249250 ### Identify risks Agent rollback is mostly insulated from other existing flows: low probability risk of affecting agent actions menus in Fleet UI. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## Summary Make `enableAgentRollback` feature flag `true` by default. Implementation PR: elastic#247398 Backport to 9.3: elastic#249250 ### Identify risks Agent rollback is mostly insulated from other existing flows: low probability risk of affecting agent actions menus in Fleet UI. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit eebeab1) # Conflicts: # x-pack/platform/plugins/shared/fleet/common/experimental_features.ts

## Summary Make `enableAgentRollback` feature flag `true` by default. Implementation PR: elastic#247398 Backport to 9.3: elastic#249250 ### Identify risks Agent rollback is mostly insulated from other existing flows: low probability risk of affecting agent actions menus in Fleet UI. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

[Fleet] Implement agent upgrade rollback

64e8492

jillguyonnet self-assigned this Dec 23, 2025

jillguyonnet added Team:Fleet Team label for Observability Data Collection Fleet team release_note:feature Makes this part of the condensed release notes backport:skip This PR does not require backporting labels Dec 23, 2025

kibanamachine and others added 4 commits December 23, 2025 16:35

Changes from yarn openapi:bundle

7d9e66b

Changes from make api-docs

d0ccd25

Fix check types

e3ef41c

Add task type

d8c95a3

jillguyonnet marked this pull request as ready for review December 24, 2025 15:07

jillguyonnet requested review from a team as code owners December 24, 2025 15:07

github-advanced-security bot found potential problems Dec 24, 2025

View reviewed changes

nchaulet self-requested a review January 2, 2026 14:45

Merge branch 'main' into fleet/agent-upgrade-rollback

4738c68

nchaulet reviewed Jan 2, 2026

View reviewed changes

Merge branch 'main' into fleet/agent-upgrade-rollback

75e83b8

florent-leborgne approved these changes Jan 5, 2026

View reviewed changes

jillguyonnet added 2 commits January 5, 2026 10:46

Feedback

4d27cc1

Fix unit tests

29ff274

nchaulet approved these changes Jan 5, 2026

View reviewed changes

jillguyonnet and others added 4 commits January 6, 2026 10:49

Add maxSize to new schemas

46b6596

Changes from yarn openapi:bundle

be5cba2

Changes from make api-docs

58fc4cb

Merge branch 'main' into fleet/agent-upgrade-rollback

b196d13

pmuellr approved these changes Jan 6, 2026

View reviewed changes

jillguyonnet mentioned this pull request Jan 6, 2026

[Fleet] The ActionRunner framework should use AbortController #247971

Open

jillguyonnet merged commit f4ab534 into elastic:main Jan 6, 2026
13 checks passed

jillguyonnet deleted the fleet/agent-upgrade-rollback branch January 6, 2026 16:34

kibanamachine added the v9.4.0 label Jan 6, 2026

jillguyonnet removed the backport:skip This PR does not require backporting label Jan 15, 2026

kibanamachine added the backport:skip This PR does not require backporting label Jan 15, 2026

jillguyonnet added backport:version Backport to applied version labels v9.3.0 and removed backport:skip This PR does not require backporting v9.4.0 labels Jan 15, 2026

kibanamachine added backport:skip This PR does not require backporting and removed backport:version Backport to applied version labels labels Jan 15, 2026

jillguyonnet mentioned this pull request Jan 15, 2026

[9.3] [Fleet] Implement agent upgrade rollback (#247398) #249250

Merged

This was referenced Jan 16, 2026

Enable enableAgentRollback feature flag #249411

Closed

Enable enableAgentRollback feature flag #249416

Merged

jillguyonnet mentioned this pull request Jan 19, 2026

[Docs] Kibana release notes for 9.3.0 #248511

Merged

20 tasks

	defaultMessage="This action will roll back the agent running on ''{hostName}'' to version {version}. This action can not be undone. Are you sure you wish to continue?"
	defaultMessage="You are about to roll back the agent running on ''{hostName}'' to version {version}. This action cannot be undone."

Comments

Conversation

jillguyonnet commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Screenshots

Testing

Checklist

Identify risks

Release note

Uh oh!

kibanamachine commented Dec 24, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10238

Uh oh!

elasticmachine commented Dec 24, 2025

Uh oh!

Check warning

Check warning

Check warning

nchaulet commented Jan 2, 2026

Uh oh!

nchaulet Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

jillguyonnet commented Jan 5, 2026

Uh oh!

florent-leborgne left a comment

Choose a reason for hiding this comment

Uh oh!

florent-leborgne Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

jillguyonnet Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

florent-leborgne Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florent-leborgne Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

nchaulet left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 6, 2026

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

History

Uh oh!

pmuellr left a comment

Choose a reason for hiding this comment

Uh oh!

jillguyonnet commented Jan 6, 2026

Uh oh!

Uh oh!

jillguyonnet commented Jan 15, 2026

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jillguyonnet commented Dec 23, 2025 •

edited

Loading

florent-leborgne Jan 5, 2026 •

edited

Loading