Graduated Testing & Test Numbers

Our goal is to employ graduated testing to optimize our use of the machine resources at our disposal. This is accomplished by tagging the tests by level, and choosing to run the 'sanity' level nightly, and the 'sanity' + 'extended' level against the release builds. Each test target may contain many tests, sometimes 1000's.

Some targets are rerun with variations of command-line options, in those cases, the number of tests are counted again, as the new input create a unique test case (a different code path is tested).

The numbers vary slightly depending on the jdk version and platforms, but for the purpose of this wiki, we will document the average/common numbers and add notes for exceptional differences.

sanity = nightly
sanity + extended = per release

Test group	# of sanity targets	# of sanity tests	# of extended targets	# of extended tests	Notes
openjdk	7	~3000	18		3 targets disabled (awt, swing, sound), 3 targets are hotspot specific (hotspot_all, hotspot_all_SE100, hotspot_jre)
system	89		27
external	8		3		only run in xLinux Docker containers
perf	2		1		full benchmark suites disabled
functional	168		295		not enabled in automated runs at AdoptOpenJDK, used only in Grinders for now (running nightly at openj9)

This wiki will not include the compliance tests in the listing.