Jenkins: audit of any retained individual build jobs to reduce disk usage

[sxa]

Ref: #3285

jenkins@jenkins-hetzner-ubuntu2004:~/jobs$ du -sk build-scripts/jobs/jobs/jobs/jdk* build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk* build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk*
38483916	build-scripts/jobs/jobs/jobs/jdk
40709196	build-scripts/jobs/jobs/jobs/jdk11u
31727760	build-scripts/jobs/jobs/jobs/jdk17u
9464336	build-scripts/jobs/jobs/jobs/jdk20u
10528416	build-scripts/jobs/jobs/jobs/jdk21
12237780	build-scripts/jobs/jobs/jobs/jdk21u
13770112	build-scripts/jobs/jobs/jobs/jdk22
25356428	build-scripts/jobs/jobs/jobs/jdk8u
3805584	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk
18372	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk11u
4906184	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk17u
5259572	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk20u
4542752	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk21
5025796	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk21u
2300000	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk22
557188	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk23
627552	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk8u
16899976	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk11u
13937892	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk17u
5823836	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk19u
8339712	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk20
6462948	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk20u
6126940	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk21
8507088	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk21u
7039280	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk8u

With about 270Gb of space used by build-scripts/jobs (i.e. the individual platform jobs) we should look at which ones are being retained in there and see if they are still required.

[sxa]

See also #3442 from the March upgrade cycle

[sxa]

Summary equivalent to the one in the initial description of this issue, in preparation for a clean-up session tomorow between myself and @steelhead31

42915908	build-scripts/jobs/jobs/jobs/jdk
46431992	build-scripts/jobs/jobs/jobs/jdk11u
38206776	build-scripts/jobs/jobs/jobs/jdk17u
1056	build-scripts/jobs/jobs/jobs/jdk20u
39546752	build-scripts/jobs/jobs/jobs/jdk21u
308	build-scripts/jobs/jobs/jobs/jdk22
1888	build-scripts/jobs/jobs/jobs/jdk22u
484	build-scripts/jobs/jobs/jobs/jdk23u
17509360	build-scripts/jobs/jobs/jobs/jdk24u
18523124	build-scripts/jobs/jobs/jobs/jdk25
488	build-scripts/jobs/jobs/jobs/jdk26
319665968	build-scripts/jobs/jobs/jobs/jdk8u
4913852	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk
1457116	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk11u
5785860	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk17u
5671880	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk21u
966084	build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk8u
17057984	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk11u
16290960	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk17u
12174160	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk21u
9153732	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk24u
8116900	build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk8u

The jdk8u is over 300GiB but that is mostly from the Solaris simple jobs which are retaining more things than they should so the job can be easily reconfigured:

26313284	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin-simple
19242656	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin-simplepipe
109826440	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin-simpletest
576	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin_SmokeTests
397104	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-hotspot
592	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-hotspot_SmokeTests
1139896	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin
60536000	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simple
36985096	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplepipe
24	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplepipebackup
28	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplepipestatic
68	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplestatic
38687888	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simpletest
24	build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simpleteststatic

Also noting that within the SPARC ones there are some large core files (>1GiB) being generated by the system tests - this example has 5 large core files in it making the archive for 91 end up as 91GiB in total. Noting that the x64 ones aren't producing cores but are still coming in at 1.3GiB per job archive and contain over 60,000 files. At the very least we should modify the process to compress them going forward (the 1.3Gib directory goes down to about 50M with xz and 170M with gzip

 53876835       4 drwxr-xr-x   2 jenkins  jenkins        4096 Jun 18 14:03 91/archive/output_17502006059704/langtools_all_0/work/classes/tools/javac/annotations/6365854/test/core
 53664120  785116 -rw-------   1 jenkins  jenkins   803954242 Jun 18 05:51 91/archive/output_17502144506801/MauveMultiThrdLoad_5m_0/20250617-225104-MauveMultiThrdLoad/results/core
 53668107 1144776 -rw-------   1 jenkins  jenkins  1172238666 Jun 18 06:33 91/archive/output_1750220679405/MiniMix_5m_0/20250617-232947-MixedLoadTest/results/core
 53634418 1278776 -rw-------   1 jenkins  jenkins  1309460234 Jun 18 06:37 91/archive/output_1750220679405/MiniMix_10m_0/20250617-233306-MixedLoadTest/results/core
 53646157 3671740 -rw-------   1 jenkins  jenkins  3759843010 Jun 18 06:45 91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-233725-MixedLoadTest/results/core
 53648916 2681144 -rw-------   1 jenkins  jenkins  2745476898 Jun 18 06:50 91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-234527-MixedLoadTest/results/core

[sxa]

Removing solaris core files has increased the space in the jobs file system from 36G to 111G - list of remove core files is in the twisty

jdk8u-solaris-sparcv9-temurin-simpletest/builds/75/archive/output_17441582146568/MiniMix_5m_0/20250408-192340-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/75/archive/output_17441582146568/MiniMix_10m_0/20250408-192702-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/75/archive/output_17441519076259/MauveMultiThrdLoad_5m_0/20250408-175509-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_17502144506801/MauveMultiThrdLoad_5m_0/20250617-225104-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_5m_0/20250617-232947-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_10m_0/20250617-233306-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-233725-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-234527-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_5m_0/20250609-191926-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_10m_0/20250609-192303-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_aot_5m_0/20250609-192949-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_aot_5m_0/20250609-193510-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495085158373/MauveMultiThrdLoad_5m_0/20250609-181234-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/60/archive/output_17371492448118/MauveMultiThrdLoad_5m_0/20250117-163900-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/79/archive/output_17461573923340/MauveMultiThrdLoad_5m_0/20250501-225330-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375768463742/MauveMultiThrdLoad_5m_0/20250122-154200-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375831371306/MiniMix_5m_0/20250122-155907-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375831371306/MiniMix_10m_0/20250122-160233-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375831371306/MiniMix_aot_5m_0/20250122-160704-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17483998949170/MauveMultiThrdLoad_5m_0/20250527-215856-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17484061333494/MiniMix_5m_0/20250527-232729-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17484061333494/MiniMix_10m_0/20250527-233213-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17484061333494/MiniMix_aot_5m_0/20250527-233704-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/59/archive/output_17371284426562/MauveMultiThrdLoad_5m_0/20250117-095106-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_1747787009862/MiniMix_5m_0/20250520-192338-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_1747787009862/MiniMix_10m_0/20250520-192935-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_1747787009862/MiniMix_aot_5m_0/20250520-193445-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_17477807639826/MauveMultiThrdLoad_5m_0/20250520-190018-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/93/archive/output_17526835348670/MauveMultiThrdLoad_5m_0/20250716-125906-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/93/archive/output_17526897767603/MiniMix_10m_0/20250716-132702-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496639255764/MiniMix_5m_0/20250611-125046-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496639255764/MiniMix_10m_0/20250611-125524-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496639255764/MiniMix_aot_5m_0/20250611-130244-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496576858763/MauveMultiThrdLoad_5m_0/20250611-122837-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/83/archive/output_1748631151270/MauveMultiThrdLoad_5m_0/20250530-145850-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/83/archive/output_17486373876362/MiniMix_5m_0/20250530-154149-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/83/archive/output_17486373876362/MiniMix_10m_0/20250530-154619-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_17371794076987/MauveMultiThrdLoad_5m_0/20250118-001654-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_1737185626472/MiniMix_5m_0/20250118-013903-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_1737185626472/MiniMix_10m_0/20250118-014217-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_1737185626472/MiniMix_aot_5m_0/20250118-014652-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_5m_0/20250721-161420-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_10m_0/20250721-161856-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_aot_5m_0/20250721-163129-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_aot_5m_0/20250721-162337-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531262467452/MauveMultiThrdLoad_5m_0/20250721-155749-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448304533232/MauveMultiThrdLoad_5m_0/20250416-151815-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448366901769/MiniMix_5m_0/20250416-155139-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448366901769/MiniMix_10m_0/20250416-155720-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448366901769/MiniMix_aot_5m_0/20250416-160854-MixedLoadTest/results/core

FYI @smlambert - I'm not sure if we've got anything covering this or plan to take any action on the number of core files being created during the Solaris/sparcv9 system test runs that are showing in the above list?

[sxa]

Space increased from 116G to 166G after enabling discarding of old builds (20 jobs being retained for now - 10 with artifacts):

jenkins@jenkins-hetzner-ubuntu2004:~/jobs/build-scripts/jobs/jobs/jobs/jdk8u/jobs$ du -sk *simpletest
5448780	jdk8u-solaris-sparcv9-temurin-simpletest
7807268	jdk8u-solaris-x64-temurin-simpletest

[sxa]

Noting that as mentioned in #3028 there are a significant number of old jobs - especially openjdk11-pipeline which has over 50GiB - which are explicitly locked from deletion. A number of them are ones from the AdoptOpenJDK days (including some from other variants such as OpenJ9) and the expectation is that they can be cleared too.

Note: job runs that have been explicitly marked as "keep forever" have <keepLog>true</keepLog> in their build.xml. At the time of eriting there at 805 such jobs - 242 of which have release in their name. I've split them into release jobs, other build jobs, test jobs and everything else:

keeplog=true.release.txt
keeplog=true.builds.txt
keeplog=true.test.txt
keeplog=true.others.txt