Graduated Testing & Test Numbers

Our goal is to employ graduated testing to optimize our use of the machine resources at our disposal. This is accomplished by tagging the tests by level, and choosing to run the 'sanity' level nightly, and the 'sanity' + 'extended' level against the release builds. Each test target may contain many tests, sometimes 1000's.

Some targets are rerun with variations of command-line options, in those cases, the number of tests are recounted, as the new input creates a unique test (a different code path is tested). In the case of system/load tests, some tests are run repeatedly (in some cases 100's to 10,000's of times).

The numbers vary slightly depending on the jdk version and platforms, but for the purpose of this wiki, we will document the average/common numbers and add notes for exceptional differences.

sanity = nightly
sanity + extended = per release

Test group	# of sanity targets	# of sanity tests	# of extended targets	# of extended tests	Notes
openjdk	7	~3000	18		extended: 3 targets disabled (awt, swing, sound), 3 targets are hotspot specific (hotspot_all, hotspot_all_SE100, hotspot_jre)
system	89		27		total: 4936 (bare number) / 25,040 (with variants) / 31,727,361 (variants with repetitions)
external	8	~40,000	3		only run in xLinux Docker containers
perf	2		1		full benchmark suites currently disabled
functional	168	~20,000	295		not enabled in automated runs at AdoptOpenJDK, used only in Grinders for now (running nightly at openj9)

This wiki will not include the compliance tests in the listing.