Clear externalDependency and testDependency cached files periodically

[Haroon-Khel]

The directories /home/jenkins/externalDependency/ and /home/jenkins/testDependency are cached dependency files used in perf or system tests. If these directories do not have the latest files, the tests will fail. Two solutions to this are

• Periodically delete these directories, or whenever a change to the files is detected upstream, delete the cache
• A solution on the test side is to check to see if the cached files are up to date before running the test

[sxa]

Personally I'd lean towards option 2.

See also adoptium/aqa-tests#5954 where checksumming of downloaded test artifacts are mentioned.

Also vaguely related:

• Some files may fail to download or download incompletely aqa-tests#4295 (Specifically mentions dacapo which we've had issues with on Solaris recently)
• Improve getDependency workflow aqa-tests#6461
  • Improved getDependency workflow aqa-tests#6694
  • Improved getDependency workflow TKG#758

[smlambert]

A solution on the test side is to check to see if the cached files are up to date before running the test

Test side we check the SHA when one is provided, and if it doesn't match, we exit, with a statement that the SHA is incorrect, and to please delete the file and rerun

if ($digest ne $expectedsha) {
					print "Expected sha is: $expectedsha,\n";
					print "Actual sha is  : $digest.\n";
					print "Please delete $filename and rerun the program!";
					die "ERROR: sha checksum error.\n";
				}

https://github.com/adoptium/TKG/blob/master/scripts/getDependencies.pl#L397-L402

This is not the case of mauve.jar, as there is no SHA set for that artifact.

We currently have an issue with mauve.jar, we used to pull it from sourceforge.org, but it is no longer available from there, so as a temp work-around we have a copy in an adoptium/aqa-triage-data repo where we pull from (atm without checking SHA).

Long story, but we tried to build from src, but the version of src we find for it does not match the version that the systemtests were using, so the changes made to build from src have been temporarily reverted.

[Haroon-Khel]

@smlambert While it's stored in adoptium/aqa-triage-data, does it sync with any upstream changes to the jar? If not and if the mauve.jar stays as it is then it sounds to me that this is just a one time error which can be solved by clearing the cache directories on the machines once

[smlambert]

this is just a one time error which can be solved by clearing the cache directories on the machines once

Correct, mauve is not changing upstream at all thus mauve.jar is not expected to change again (until we build from source, but the hope is that it will be a close or exact match to what we are using now.

[Haroon-Khel]

Ive cleared the externalDependency directory of alp320-2 and -3 and the mauve tests now pass
-3 https://ci.adoptium.net/job/Grinder/15403/
-2 https://ci.adoptium.net/job/Grinder/15399/console (ParallelStream tests still fail on -2, will raise infra issue on this)

[Haroon-Khel]

This issue can be closed

[Haroon-Khel]

In fact ill keep this issue open and try to clear the externalDependency directory of each machine just to be thorough