ci.jenkins.io EC2 agents sometimes hang and fail to connect with SSH key not found

[MarkEWaite]

Jenkins and plugins versions report

Environment

Jenkins: 2.528.3
OS: Linux - 6.8.0-1044-aws
Java: 21.0.9 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
analysis-model-api:13.18.0-933.v90e9a_fdff4e2
ansicolor:1.0.6
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-170.v023de017ccd7
artifact-manager-s3:962.v15f7000205fa_
asm-api:9.9.1-189.vb_5ef2964da_91
authentication-tokens:1.144.v5ff4a_5ec5c33
aws-credentials:254.v978a_5e206a_d7
aws-global-configuration:152.ve13a_2a_c319a_9
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-ec2:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-s3:2.33.4-62.vc1a_8df64b_4c9
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1268.v044a_87612da_8
build-discarder:158.vce570d01ce4c
buildtriggerbadge:325.vdcd8a_826e87a_
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
catppuccin-theme:3.v903b_635ffa_b_4
checks-api:373.vfe7645102093
chocolate-theme:7.v6a_8b_000467c6
cloud-stats:377.vd8a_6c953e98e
cloudbees-folder:6.1073.va_7888eb_dd514
command-launcher:123.v37cfdc92ef67
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-210.v7480a_da_70b_9e
config-file-provider:1006.vc7366c201f57
configuration-as-code:2006.v001a_2ca_6b_574
copyartifact:770.va_6c69e063442
coverage:2.3060.v035a_5557cdb_c
coverage-badges-extension:197.vb_390173d00ec
credentials:1453.v9b_a_29777a_b_fd
credentials-binding:702.vfe613e537e88
csp:1.43.vb_e0a_fd67d28d
customizable-header:251.251.v49b_7da_e6c61a_
dark-theme:574.va_19f05d54df5
data-tables-api:2.3.5-1482.vb_47a_3c19f78c
datadog:9.1.8
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:457.v0f62a_94f11a_3
docker-workflow:634.vedc7242b_eda_7
durable-task:639.vefb_3d8372f6d
ec2:2045.v06da_da_a_46422
echarts-api:6.0.0-1165.vd1283a_3e37d4
eddsa-api:0.3.0.1-19.vc432d923e5ee
embeddable-build-status:637.vd878e68178f8
flatpickr-api:4.6.13-18.vcf5f6a_5b_8468
font-awesome-api:7.1.0-882.v1dfb_771e3278
forensics-api:3.1772.v99ca_3d83b_9fa_
generic-tool:1.17.vb_8a_b_4e5600c9
git:5.8.1
git-client:6.4.2
git-forensics:3.2138.vf25ea_d549e33
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1917.v9ee8a_39b_3d0d
github-checks:634.v371dc6d978a_3
groovy:497.v7b_061a_a_de65d
gson-api:2.13.2-173.va_a_092315913c
http_request:1.24
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.1-423.v13951f6b_6532
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-10.v9b_7e1d1fc40b_
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jjwt-api:0.11.5-120.v0268cf544b_89
jnr-posix-api:3.1.21-186.vb_7ec9b_23ce83
joda-time-api:2.14.0-149.v1c3ce991d1b_9
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-173.v596efb_962a_31
json-path-api:2.10.0-202.va_9cc16c1e476
jsoup:1.21.2-66.v6ea_38164b_8a_2
junit:1369.v15da_00283f06
junit-attachments:330.v25180b_263160
junit-realtime-test-reporter:166.v889e7b_0e23a_a_
kubernetes:4398.vb_b_33d9e7fe23
kubernetes-client-api:7.3.1-256.v788a_0b_787114
kubernetes-credentials:206.vde31a_b_0f71a_c
ldap:793.v754d6b_41b_ea_4
lockable-resources:1438.v3c0f8c9e2060
mailer:525.v2458b_d8a_1a_71
matrix-auth:3.2.9
matrix-project:870.v9db_fcfc2f45b_
metrics:4.2.37-489.vb_6db_69b_ce753
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
mina-sshd-api-scp:2.16.0-167.va_269f38cc024
netty-api:4.1.128.Final-14.v3a_f80ce4b_522
node-iterator-api:72.vc90e81737df1
nord-theme:1.1
okhttp-api:4.12.0-195.vc02552c04ffd
oss-symbols-api:424.ved751e062911
parallel-test-executor:611.va_92b_d1852e52
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-github:2.8-162.382498405fdc
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-graph-view:661.v6003f4542123
pipeline-groovy-lib:787.ve2fef0efdca_6
pipeline-input-step:540.v14b_100d754dd
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.38
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
prism-api:1.30.0-630.va_e19d17f83b_0
scm-api:724.v7d839074eb_5c
script-security:1385.v7d2d9ec4d909
snakeyaml-api:2.5-143.v93b_c004f89de
solarized-theme:28.vfe25223f14fe
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1085.vc64d040efa_85
sshd:3.374.v19b_d59ce6610
structs:362.va_b_695ef4fdf9
theme-manager:327.v780d7096ec29
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
variant:70.va_d9f17f859e0
warnings-ng:12.9936.vda_5743ded29a_
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4238.va_6fb_65c1f699
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1559.va_a_533730b_ea_d
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1010.vb_b_39488a_9841

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux for controller with Windows and Linux EC2 agents

Reproduction steps

1. Launch a dynamically provisioned agent (ci.jenkins.io provisions based on requests for labels like maven-21)
2. Sometimes, the agent will not be connected but will remain in the list of nodes with the message that the SSH host key could not be read

Leaving the EC2 agent running but not connected as an agent is a waste of money.

If I delete the agent that cannot read the SSH key, a new agent is provisioned as needed and work continues

Expected Results

Agent should be deleted if it cannot be used

Actual Results

EC2 virtual machine remains running, without being usable as an agent

Anything else?

No response

Are you interested in contributing a fix?

No

[timja]

FWIW Azure VM agents has a feature for this:
https://github.com/jenkinsci/azure-vm-agents-plugin/blob/56c5e6fe755dcb6f7e2610f3756dad1b44e0c968/src/main/java/com/microsoft/azure/vmagent/AzureVMAgentCleanUpTask.java#L337

It will delete any VM that it "owns"[1] that it doesn't know about

[1]: based on tags it uses the Jenkins URL as a tag, we originally used the instance identity but found people were cloning VMs between environments so they had duplicate identities...

[craigcomstock]

I mentioned in https://community.jenkins.io/t/host-key-verification-for-ec2-instances-buggy/36109 that for me I can see that the hostkey IS verified in the logs BEFORE the timestamp in the UI that says that the hostkey is NOT verified:

e.g. I see this in the log:

Jan 28, 2026 9:44:21 PM hudson.plugins.ec2.EC2Cloud
INFO: Connection allowed after the host key has been verified

and then the UI says:

Jan 28, 2026, 9:43:49 PM
The instance SSH key was unexpected or impossible to check

Not sure if that helps or not. In the meantime I suppose the workaround is to change platforms which exhibit this behavior, for me Ubuntu 24 for some reason, to CHECK_NEW_SOFT.