Graduated Testing & Test Numbers

Our goal is to employ graduated testing to optimize our use of the machine resources at our disposal. This is accomplished by tagging the tests by level, and choosing to run the 'sanity' level nightly, and the 'sanity' + 'extended' level against the release builds. Each test target may contain many tests, sometimes 1000's.

Some targets are rerun with variations of command-line options, in those cases, the number of tests are recounted, as the new input creates a unique test (a different code path is tested). In the case of system/load tests, some tests are run repeatedly (in some cases 100's to 10,000's of times).

The numbers vary slightly depending on the jdk version and platforms, but for the purpose of this wiki, we will document the average/common numbers and add notes for exceptional differences.

sanity = nightly
(Triggered by nightly pipelines) sanity + extended = per release (Triggered manually at present, eventually to be triggered by release pipelines when we have sufficient machine resources)

Test group	# of sanity targets	# of sanity tests	# of extended targets	# of extended tests	Platforms	Notes
openjdk	7	~3000	18		all	extended: 3 targets disabled (awt, swing, sound), 3 targets are hotspot specific (hotspot_all, hotspot_all_SE100, hotspot_jre)
system	89	4454 (bare number)	27	482 (bare number)	all	total: 4936 (bare number) / 25,040 (with variants) / 31,727,361 (variants with repetitions), extended: 11 targets all jdk impls, 16 are openj9 specific
external	8	~40,000	3		xLinux_docker	sanity targets are functional suites from applications, extended are microprofile tcks (which are actually being run nightly)
perf	2		1		xLinux	full benchmark suites currently disabled
functional	168	~20,000	295		all	not enabled in automated runs at AdoptOpenJDK, used only in Grinders for now (running nightly at openj9)

This wiki will not include the compliance tests in the listing.