[Flaky Test]: TestFleetManagedUpgradeRollbackOnRestarts – agent never came online with version 9.3.0-SNAPSHOT #10917
Description
Failing test case
TestFleetManagedUpgradeRollbackOnRestarts
Error message
agent never came online with version 9.3.0-SNAPSHOT
Build
https://buildkite.com/elastic/elastic-agent/builds/29626
OS
Linux, Windows
Stacktrace and notes
=== Failed
=== FAIL: testing/integration/ess TestFleetManagedUpgradeRollbackOnRestarts/downgrade_from_current_version_to_previous_minor (274.05s)
fetcher.go:102: Using existing artifact elastic-agent-9.3.0-SNAPSHOT-windows-x86_64.zip
fixture.go:311: Extracting artifact elastic-agent-9.3.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor2469862036
fixture.go:329: Completed extraction of artifact elastic-agent-9.3.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor2469862036
fixture.go:1018: Components were not modified from the fetched artifact
fixture.go:861: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor2469862036\elastic-agent-9.3.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
fetcher_artifact.go:252: Downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.0-windows-x86_64.zip
fetcher_artifact.go:331: Downloading artifact progress 59.97%
fetcher_artifact.go:331: Downloading artifact progress 100.00%
fetcher_artifact.go:298: Completed downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.0-windows-x86_64.zip
fetcher_artifact.go:252: Downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.0-windows-x86_64.zip.sha512
fetcher_artifact.go:298: Completed downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.0-windows-x86_64.zip.sha512
fetcher_artifact.go:252: Downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.0-windows-x86_64.zip.asc
fetcher_artifact.go:298: Completed downloading artifact from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.2.0-windows-x86_64.zip.asc
fixture.go:311: Extracting artifact elastic-agent-9.2.0-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor1732461727
fixture.go:329: Completed extraction of artifact elastic-agent-9.2.0-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor1732461727
fixture.go:1018: Components were not modified from the fetched artifact
fixture.go:861: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor1732461727\elastic-agent-9.2.0-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
upgrade_rollback_test.go:423: Testing Elastic Agent upgrade from 9.3.0-SNAPSHOT to 9.2.0...
fixture.go:861: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor2469862036\elastic-agent-9.3.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
fixture.go:861: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor1732461727\elastic-agent-9.2.0-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
upgrade_fleet_test.go:409: Creating Agent policy...
upgrade_fleet_test.go:416: Creating Agent enrollment API key...
upgrade_fleet_test.go:425: Getting default Fleet Server URL...
upgrade_fleet_test.go:432: Installing Elastic Agent (unprivileged: false)...
fixture_install.go:200: [test TestFleetManagedUpgradeRollbackOnRestarts/downgrade_from_current_version_to_previous_minor] Inside fixture install function
fixture_install.go:228: [test TestFleetManagedUpgradeRollbackOnRestarts/downgrade_from_current_version_to_previous_minor] Inside fixture installNoPkgManager function
fixture_install.go:247: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor2469862036\elastic-agent-9.3.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive --url https://bf2df2e5cecfe986c2ab06c50b5cf402.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token TFMzc01wb0JPNS03WHo5NThkRFI6cFBubnBzX1QyRnFKUGwtdTlBYndfZw==]
upgrade_fleet_test.go:447: install start agent output:
Installing in non-interactive mode.
[====] Service Started [4s] Elastic Agent successfully installed, starting enrollment.
[====] Waiting For Enroll... [4s] {"log.level":"info","@timestamp":"2025-10-30T02:23:08.023Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/enroll.EnrollWithBackoff","file.name":"enroll/enroll.go","file.line":84},"message":"Starting enrollment to URL: https://bf2df2e5cecfe986c2ab06c50b5cf402.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-10-30T02:23:09.483Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":387},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-10-30T02:23:09.485Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":205},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=== ] Done [6s]
Elastic Agent has been successfully installed.
upgrade_fleet_test.go:457: Setting custom watcher config
upgrade_fleet_test.go:461: Waiting for Agent to be correct version and healthy...
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:569: waiting for healthy agent and proper version: agent state is not healthy: got 0
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:569: waiting for healthy agent and proper version: agent state is not healthy: got 0
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrade_fleet_test.go:471: Agent ID: "09046525-9c9c-4815-bebf-709c30a546f9"
upgrade_fleet_test.go:473: Waiting for enrolled Agent status to be online...
upgrade_fleet_test.go:486: Upgrading from version "9.3.0-SNAPSHOT-d3d5bad995f577baa7231b54fc2fd7550fdc857c" to version "9.2.0-e7bf319beb9675435029da09a2c11760dc4100bd"...
upgrade_fleet_test.go:494: Waiting from upgrade details to show up in Fleet
upgrade_fleet_test.go:510: Waiting for upgrade watcher to start...
upgrade_fleet_test.go:515: Upgrade watcher started
upgrade_rollback_test.go:593: Stopping agent via service to simulate crashing
upgrade_rollback_test.go:598: Got an allowed error on Windows: failed to stop service (Elastic Agent): The service has not been started.
upgrade_rollback_test.go:613: Stopped agent via service to simulate crashing
upgrade_rollback_test.go:616: Starting agent via service to simulate crashing
upgrade_rollback_test.go:628: Started agent via service to simulate crashing
upgrade_rollback_test.go:593: Stopping agent via service to simulate crashing
upgrade_rollback_test.go:613: Stopped agent via service to simulate crashing
upgrade_rollback_test.go:616: Starting agent via service to simulate crashing
upgrade_rollback_test.go:628: Started agent via service to simulate crashing
upgrade_rollback_test.go:593: Stopping agent via service to simulate crashing
upgrade_rollback_test.go:613: Stopped agent via service to simulate crashing
upgrade_rollback_test.go:616: Starting agent via service to simulate crashing
upgrade_rollback_test.go:628: Started agent via service to simulate crashing
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 1m49.8867228s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 1m39.8854393s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 1m29.8835935s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 1m19.8836802s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 1m9.8843159s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 59.8826024s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 49.8816451s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 39.8821874s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 29.8862102s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 19.8856057s
fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:562: version mismatch, ignoring it, time until timeout: 9.8859856s
upgrader.go:569: waiting for healthy agent and proper version: agent status failed: context deadline exceeded
upgrade_rollback_test.go:459:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-gcp-1761789337983067494/elastic/elastic-agent/testing/integration/ess/upgrade_rollback_test.go:459
C:/buildkite-agent/builds/bk-agent-prod-gcp-1761789337983067494/elastic/elastic-agent/testing/integration/ess/upgrade_rollback_test.go:337
Error: Received unexpected error:
failed waiting for healthy agent and version (context deadline exceeded): agent status failed: context deadline exceeded
Test: TestFleetManagedUpgradeRollbackOnRestarts/downgrade_from_current_version_to_previous_minor
Messages: agent never came online with version 9.3.0-SNAPSHOT
fixture_install.go:311: [test TestFleetManagedUpgradeRollbackOnRestarts/downgrade_from_current_version_to_previous_minor] Inside fixture cleanup function
fixture_install.go:327: collecting diagnostics; test failed
fixture_install.go:789: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1761789337983067494\elastic\elastic-agent\build\diagnostics\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor-2025-10-30T02-27-07Z-diagnostics.zip]
fixture_install.go:740: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall --force]
fixture.go:1143: Dumping running processes in C:\buildkite-agent\builds\bk-agent-prod-gcp-1761789337983067494\elastic\elastic-agent\build\diagnostics\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor-2025-10-30T02-27-07Z-ProcessDump-cleanup.json
fixture.go:1434: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor1732461727" preserved for investigation/debugging
fixture.go:1434: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestFleetManagedUpgradeRollbackOnRestarts-downgrade_from_current_version_to_previous_minor2469862036" preserved for investigation/debugging
--- FAIL: TestFleetManagedUpgradeRollbackOnRestarts/downgrade_from_current_version_to_previous_minor (274.05s)