Jenkins kills long-running jobs without notice #26365
Description
Jenkins and plugins versions report
Environment
Jenkins: 2.550
OS: Linux - 5.14.0-570.26.1.el9_6.x86_64
Java: 21.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ansicolor:536.v13fa_b_860c267
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.6-183.ve5a_8a_b_e71e59
asm-api:9.9.1-189.vb_5ef2964da_91
authentication-tokens:1.144.v5ff4a_5ec5c33
blueocean:1.27.25
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.25
blueocean-commons:1.27.25
blueocean-config:1.27.25
blueocean-core-js:1.27.25
blueocean-dashboard:1.27.25
blueocean-display-url:2.4.4
blueocean-events:1.27.25
blueocean-git-pipeline:1.27.25
blueocean-github-pipeline:1.27.25
blueocean-i18n:1.27.25
blueocean-jwt:1.27.25
blueocean-personalization:1.27.25
blueocean-pipeline-api-impl:1.27.25
blueocean-pipeline-editor:1.27.25
blueocean-pipeline-scm-api:1.27.25
blueocean-rest:1.27.25
blueocean-rest-impl:1.27.25
blueocean-web:1.27.25
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1280.v0d4e5b_b_460ef
build-user-vars-plugin:214.va_eed2ed849ca_
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:402.vca_263b_f200e3
cloudbees-bitbucket-branch-source:937.2.4
cloudbees-folder:6.1079.vc0975c2de294
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-3
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-210.v7480a_da_70b_9e
conditional-buildstep:1.5.0
copyartifact:795.ve8e151429b_27
credentials:1490.ve7532596f1fd
credentials-binding:717.v951d49b_5f3a_a_
customizable-header:285.v70da_c6f8a_420
dark-theme:574.va_19f05d54df5
display-url-api:2.217.va_6b_de84cc74b_
durable-task:651.v1f5e074fc83f
echarts-api:6.0.0-1165.vd1283a_3e37d4
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
email-ext:1933.v45cec755423f
envinject:2.934.vc674e76cf954
envinject-api:1.237.v82803a_511906
favorite:2.263.v941d21defef7
file-operations:409.v90cc23075442
flatpickr-api:4.6.13-32.v60a_51029c136
font-awesome-api:7.1.0-882.v1dfb_771e3278
git:5.10.0
git-client:6.5.0
github:1.46.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1967.vdea_d580c1a_b_a_
github-checks:676.v48c732b_52951
google-chat-notification:166.ve6b_de280f2e8
gson-api:2.13.2-173.va_a_092315913c
handy-uri-templates-2-api:2.1.8-38.vcea_5d521d5f3
htmlpublisher:427
http_request:1.24
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.1-423.v13951f6b_6532
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-12.vb_1833c1231d3
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-143.v5979df3304e6
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.25
jjwt-api:0.11.5-120.v0268cf544b_89
job-dsl:3654.vdf58f53e2d15
job-import-plugin:143.v044a_2e819b_27
jobConfigHistory:1356.ve360da_6c523a_
joda-time-api:2.14.0-177.vd7e9347b_e7d5
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20251224-185.v0cc18490c62c
json-path-api:2.10.0-202.va_9cc16c1e476
jsoup:1.22.1-76.v9cdb_2456c0e3
junit:1396.v095840ed8491
ldap:807.v7d7de30930cf
log-parser:3.0.2
login-theme:284.v95044396a_484
mailer:525.v2458b_d8a_1a_71
matrix-auth:3.2.9
matrix-project:870.v9db_fcfc2f45b_
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
okhttp-api:4.12.0-195.vc02552c04ffd
parameterized-trigger:873.v8b_e37dd8418f
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:254.v0f63a_a_447dca_
pipeline-graph-view:777.vdeb_113b_b_94da_
pipeline-groovy-lib:787.ve2fef0efdca_6
pipeline-input-step:540.v14b_100d754dd
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.39
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.39
pipeline-utility-steps:2.20.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
prism-api:1.30.0-703.v116fb_3b_5b_b_a_a_
pubsub-light:1.19
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
run-condition:276.v97298f3a_cd51
scm-api:728.vc30dcf7a_0df5
script-security:1399.ve6a_66547f6e1
simple-theme-plugin:230.v8b_fd91b_b_800c
snakeyaml-api:2.5-149.v72471e9c6371
solarized-theme:28.vfe25223f14fe
sse-gateway:1.29
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1097.v868116049892
ssh-steps:2.0.92.vb_a_0583935f9b_
sshd:3.384.vc89b_5e138cf9
structs:362.va_b_695ef4fdf9
testng-plugin:981.v1dc64d227855
theme-manager:327.v780d7096ec29
thinBackup:2.1.3
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
uno-choice:2.8.9
variant:70.va_d9f17f859e0
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1398.v67030756d3fb_
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4256.v991a_f1991c03
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1571.vb_423c255d6d9
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1015.v785e5a_b_b_8b_22
ws-cleanup:0.49
What Operating System are you using (both controller, and any agents involved in the problem)?
All Jenkins machines are running on Rocky Linux 9.6
The controller is running in Docker compose + nginx.
Reproduction steps
- Execute night unit test suite, usually runs between 24-28 hours.
Long running job, which does not print much into the Jenkins console, works in the background and produces other types of logs.
Expected Results
The triggered Job to finish and not to be stopped by the system.
Actual Results
Something sporadically stops the Job execution. We have observed that the "Sending interrupt signal to process" can be send on the 12; 15; or 20thet hour of the execution, every time is different.
Looking at the console we can see this:
21:10:08 [Pipeline] dir
21:10:08 Running in /projects/sandbox-unit-test/sandboxes/1891-94697299/build/
21:10:08 [Pipeline] {
21:10:08 [Pipeline] sh
21:10:08 + set -e
21:10:08 + ./test-all --timeout 3600 -c /projects/sandbox-unit-test/sandboxes/1891-94697299/......
09:39:25 Sending interrupt signal to process <-- This is the problem, we cannot understand what cause it.
09:39:28 Sending interrupt signal to process <-- Second attempt to stop the long running job.
09:39:31 Terminated
09:39:31 + kill 519067
09:39:31 + true
09:39:31 script returned exit code 143
09:39:31 [Pipeline] echo
09:39:31 Еxception catched: org.jenkinsci.plugins.workflow.steps.FlowInterruptedException: Body of block-scoped step failed
Anything else?
This problem was not observed in Jenkins version 2.530.
After we upgrade to 2.550 we start hitting this problem.
Are you interested in contributing a fix?
I'm not sure that I can help with that.