Build pipeline jobs stuck in "CurrentBuildDone" status, never achieve "Done"

[andrew-m-leonard]

Build pipeline jobs stuck in "CurrentBuildDone" status, never achieve "Done"

eg: https://ci.adoptium.net/job/build-scripts/job/openjdk17-pipeline/980/
Completed 11 days ago
TRSS _id = 685313e6e512ba006dae1234

==> Child job     "_id": "6856c94be512ba006daf07fa",
"url": "https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u",
"buildName": "jdk17u-linux-x64-temurin",
"buildNameStr": "jdk17u-linux-x64-temurin",
"buildNum": 649,
"rootBuildId": "685313e6e512ba006dae1234",
"parentId": "685313e6e512ba006dae1234",
"type": "Build",
"status": "CurrentBuildDone",

       ==> Child job {
"_id": "6856de88e512ba006daf0951",
"url": "https://ci.adoptium.net",
"buildName": "Test_Job_Auto_Gen",
"buildNameStr": "Test_Job_Auto_Gen",
"buildNum": 32420,
"rootBuildId": "685313e6e512ba006dae1234",
"parentId": "6856c94be512ba006daf07fa",
"type": "Build",
"status": "NotDone"

},

The Test_Job_Auto_Gen is left in "NotDone" for some reason??

[smlambert]

What is interesting and 'odd' is that report says it is expecting Test_Job_Auto_Gen buildNum 32420, https://ci.adoptium.net/view/Test_grinder/job/Test_Job_Auto_Gen/32420/ which doesn't actually exist...

The latest job build id is https://ci.adoptium.net/view/Test_grinder/job/Test_Job_Auto_Gen/33836/

[andrew-m-leonard]

Yeah the Test_Job_Auto_Gen gets called many many times during builds, and jobs very quickly get purged past 100

Could that be why, ie. Jenkins no longer has the job details? Would we need to store say 1000 Builds of Test_Job_Auto_Gen ?

[andrew-m-leonard]

I have up'd the Test_Job_Auto_Gen build to keep to 1000, since they have no artifacts it should be ok. I will try a new pipeline and see how TRSS sees it....

[andrew-m-leonard]

I've started a jdk17 pipeline to test https://ci.adoptium.net/job/build-scripts/job/openjdk17-pipeline/981/

[llxia]

Test_Job_Auto_Gen does not exist should not cause any problem. Once TRSS figures out the job does not exist anymore, it will move on.

[andrew-m-leonard]

Test_Job_Auto_Gen does not exist should not cause any problem. Once TRSS figures out the job does not exist anymore, it will move on.

that would just a guess, since the TRSS DB has a lot of the Test_Job_Auto_Gen jobs in state "NotDone", thus I believe preventing all the pipeline builds from getting to "Done", remaining in "BuildCurrentDone".

Should https://trss.adoptium.net/ GUI maybe say completed(not in progress) if the build is in "BuildCurrentDone" maybe? @llxia

[andrew-m-leonard]

TRSS is currently showing 18 pipeline builds "in-progress", when actually there are none....

[andrew-m-leonard]

i'm sure it wasn't like this a few weeks back... has something changed in this area?

[llxia]

What about the TRSS log? Does it contain any information? I do not have access TRSS server at Adoptium, so I cannot check.

[steelhead31]

This is from the trss log..

aqa-test-tools-server-1 | 10:01:46 AM - warn: Server returned unexpected status code: 403
aqa-test-tools-server-1 | 10:01:46 AM - warn: Try #1: connection issue [Arguments] { '0': 'Test_Job_Auto_Gen', '1': 29520 }
aqa-test-tools-server-1 | 10:01:46 AM - warn: status code: 403
aqa-test-tools-server-1 | 10:01:46 AM - warn: headers: {"server":"nginx","date":"Thu, 03 Jul 2025 10:01:46 GMT","content-type":"text/html;charset=utf-8","transfer-encoding":"chunked","connection":"close","vary":"Accept-Encoding","x-content-type-options":"nosniff","set-cookie":["JSESSIONID.4526b000=node05j9o41vr2ku6dcl0as2jcoks21118823.node0; Path=/; Secure; HttpOnly"],"expires":"Thu, 01 Jan 1970 00:00:00 GMT","x-hudson":"1.395","x-jenkins":"2.504.1","x-jenkins-session":"3ebdb50a"}
aqa-test-tools-server-1 | 10:01:46 AM - warn: Server returned unexpected status code: 403
aqa-test-tools-server-1 | 10:01:46 AM - warn: Try #2: connection issue [Arguments] { '0': 'Test_Job_Auto_Gen', '1': 29520 }
aqa-test-tools-server-1 | 10:01:46 AM - warn: status code: 403
aqa-test-tools-server-1 | 10:01:46 AM - warn: headers: {"server":"nginx","date":"Thu, 03 Jul 2025 10:01:46 GMT","content-type":"text/html;charset=utf-8","transfer-encoding":"chunked","connection":"close","vary":"Accept-Encoding","x-content-type-options":"nosniff","set-cookie":["JSESSIONID.4526b000=node02zp73fomqvf4f7k0shoq3n7n21118824.node0; Path=/; Secure; HttpOnly"],"expires":"Thu, 01 Jan 1970 00:00:00 GMT","x-hudson":"1.395","x-jenkins":"2.504.1","x-jenkins-session":"3ebdb50a"}
aqa-test-tools-server-1 | 10:01:47 AM - warn: Server returned unexpected status code: 403

[steelhead31]

im going to re-enable enhance logging on trss/nginx, and probe the nginx logs further

[andrew-m-leonard]

So I suspect those 403's for Test_Job_Auto_Gen are because Jenkins no longer has those builds.... which makes me think TRSS does not handle that error and "assume" the Test_Job_Auto_Gen is now "Done", and leaves them in "NotDone" ?

I have increased the Jenkins Test_Job_Auto_Gen max jobs to 1000 now, which may avert the problem in the future

[steelhead31]

Yup, I'll still add the extra logging for a bit

[llxia]

warn: Server returned unexpected status code: 403

Did the Jenkins credentials change?

[andrew-m-leonard]

warn: Server returned unexpected status code: 403

Did the Jenkins credentials change?

403 is what you get once Jenkins prunes to the max builds, eg. try https://ci.adoptium.net/job/Test_Job_Auto_Gen/31934/
and note this will typically happen before the top level pipeline finishes, or at least it did when max builds=100, since in an average pipeline about 110 Test_Job_Auto_Gen jobs were started...