Skip to content

Commit ed541b4

Browse files
aschemanascheman
committed
Add golden master testing documentation
Document golden master testing methodology and add output sections to all example READMEs showing expected vs. actual results via dynamic includes. Update CI/CD pipeline to provide run results for documentation. Key changes: * README.adoc: Add [[golden-master-testing]] section explaining the testing approach, how verify.sh works, and why some examples are excluded from golden master testing * 35 examples WITH golden master: Add Output sections with include directives for expected-result/run.txt and run-result/run.txt * 5 examples WITHOUT golden master: Add explanatory notes with cross-references to main README section (Maven/Gradle/Spring examples excluded due to variable output: timestamps, paths, versions) * pom.xml: Configure AsciiDoctor logHandler to fail build on WARN severity to catch missing includes and other documentation issues * .github/workflows/build.yml: Always upload run-results artifacts (not just on failure), download Linux results for docs generation, and allow docs job to run even if tests fail Benefits: - Documentation shows actual program output via includes (no copy/paste) - Output automatically updates when test results change - Clear documentation of testing methodology and exclusions - Cross-references connect example docs to main README - Build fails fast on documentation errors (missing includes, etc.) - CI generates docs with real test output even on test failures This establishes a single source of truth for example outputs and ensures documentation stays synchronized with actual program behavior. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> Introduced in the course of support-and-care/maven-support-and-care#137
1 parent 22c6bb2 commit ed541b4

File tree

43 files changed

+729
-4
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+729
-4
lines changed

‎.github/workflows/build.yml‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -188,8 +188,8 @@ jobs:
188188
fi
189189
echo
190190
191-
- name: Upload run results on failure
192-
if: failure()
191+
- name: Upload run results
192+
if: always()
193193
uses: actions/upload-artifact@v4
194194
with:
195195
name: run-results-${{ matrix.os }}
@@ -200,7 +200,7 @@ jobs:
200200
name: Generate and Deploy Documentation
201201
runs-on: ubuntu-latest
202202
needs: build
203-
if: success()
203+
if: "!cancelled()"
204204

205205
permissions:
206206
contents: read
@@ -218,6 +218,12 @@ jobs:
218218
distribution: 'temurin'
219219
java-version: '17'
220220

221+
- name: Download Linux run results for documentation
222+
uses: actions/download-artifact@v4
223+
with:
224+
name: run-results-ubuntu-latest
225+
path: jigsaw-examples
226+
221227
- name: Generate documentation
222228
run: ./mvnw verify
223229

‎README.adoc‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,24 @@ If you run all stuff on *nix, use a colon : .
119119
* *Either* call one of the scripts on the top level, i.e., one of `allclean.sh`, `allcompile.sh`, `allcreatevis.sh` and `allrun.sh` (or `all.sh` for all in one step).
120120
* *Alternatively*, cd to one of the examples and call one of the scripts there (again `all.sh` for all in one step).
121121

122+
[[golden-master-testing]]
123+
=== Golden Master Testing
124+
125+
Most examples in this repository use golden master testing (also known as characterization testing or approval testing) to ensure output consistency.
126+
This technique captures the expected output of a program and compares it against actual output during subsequent runs.
127+
128+
*How it works in this repository:*
129+
130+
* Each example with golden master testing has an `expected-result/run.txt` file containing the captured expected output
131+
* When you run `./verify.sh` in an example directory, it executes the example and compares the actual output (saved to `run-result/run.txt`) with the expected output
132+
* The test passes if both outputs match exactly
133+
134+
*Why some examples don't have golden master testing:*
135+
136+
Examples that use build tools (Maven, Gradle) or frameworks (Spring Boot) typically produce output that varies between runs.
137+
This includes timestamps, absolute file paths, download progress messages, port numbers, and environment-specific information.
138+
For these examples, manual verification or integration tests are more appropriate.
139+
122140
=== Additional information
123141

124142
All examples have been tested with the <<recommended-jdk,recommended JDK version>> on Windows, Linux, and macOS.

‎jigsaw-examples/example_addExports_manifest/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,22 @@ For this to work
4343
* modmain has to be in the unnamed module
4444

4545
see: https://mail.openjdk.java.net/pipermail/jpms-spec-experts/2016-September/000391.html
46+
47+
=== Output
48+
49+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
50+
The expected output is compared with actual output using `verify.sh`.
51+
52+
==== Expected Output
53+
54+
[source]
55+
----
56+
include::expected-result/run.txt[]
57+
----
58+
59+
==== Actual Output
60+
61+
[source]
62+
----
63+
include::run-result/run.txt[]
64+
----

‎jigsaw-examples/example_addReads_addExports/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,22 @@ a) modb needs a read relationship to modc
4141
b) modc needs to export its package pkgc to modb
4242
c) modmain needs a read relationship to modb
4343
d) modb needs to exports it package pkgb to modmain
44+
45+
=== Output
46+
47+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
48+
The expected output is compared with actual output using `verify.sh`.
49+
50+
==== Expected Output
51+
52+
[source]
53+
----
54+
include::expected-result/run.txt[]
55+
----
56+
57+
==== Actual Output
58+
59+
[source]
60+
----
61+
include::run-result/run.txt[]
62+
----

‎jigsaw-examples/example_addReads_addExports_reflection/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,3 +45,22 @@ Adding cannot be done in modmain/Main directly, as the reflection method is call
4545
Note that for successful compilation, modmain/Main needs access to the exported BHelper.
4646
Therefore the option --add-exports is needed.
4747
Same thing during runtime.
48+
49+
=== Output
50+
51+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
52+
The expected output is compared with actual output using `verify.sh`.
53+
54+
==== Expected Output
55+
56+
[source]
57+
----
58+
include::expected-result/run.txt[]
59+
----
60+
61+
==== Actual Output
62+
63+
[source]
64+
----
65+
include::run-result/run.txt[]
66+
----

‎jigsaw-examples/example_annotations/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,22 @@ see https://openjdk.java.net/projects/jigsaw/spec/issues/#ModuleAnnotations[Modu
3737
* modmain looks up modules modb and mod.annotations and prints their annotations available at runtime
3838
* modb is deprecated via annotation in its module-info.java.
3939
* mod.annotations defines three annotations with ElementType module, one available at runtime, two only available at compile time
40+
41+
=== Output
42+
43+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
44+
The expected output is compared with actual output using `verify.sh`.
45+
46+
==== Expected Output
47+
48+
[source]
49+
----
50+
include::expected-result/run.txt[]
51+
----
52+
53+
==== Actual Output
54+
55+
[source]
56+
----
57+
include::run-result/run.txt[]
58+
----

‎jigsaw-examples/example_automatic-module-logging/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,22 @@ The example also shows:
4747
* during compile time, only the interfaces are needed
4848
* the mapping of names in automatic modules: The part behind the last "-" is interpreted as version and cut off.
4949
Each remaining "-" is substituted by a ".".
50+
51+
=== Output
52+
53+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
54+
The expected output is compared with actual output using `verify.sh`.
55+
56+
==== Expected Output
57+
58+
[source]
59+
----
60+
include::expected-result/run.txt[]
61+
----
62+
63+
==== Actual Output
64+
65+
[source]
66+
----
67+
include::run-result/run.txt[]
68+
----

‎jigsaw-examples/example_compile-target-jdk8/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,22 @@ This example is about compatibility of code compiled for jdk 8 / 9 and running i
3535
To run this example, set a correct JDK 8 as environment variable JAVA8_HOME.
3636

3737
Note: No JavaDoc generation for this example.
38+
39+
=== Output
40+
41+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
42+
The expected output is compared with actual output using `verify.sh`.
43+
44+
==== Expected Output
45+
46+
[source]
47+
----
48+
include::expected-result/run.txt[]
49+
----
50+
51+
==== Actual Output
52+
53+
[source]
54+
----
55+
include::run-result/run.txt[]
56+
----

‎jigsaw-examples/example_derived_private-package-protected/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,22 @@ E.g. a cast to InternalData would not work in modmain.
4747
May be called from modmain, as long as the type InternalData is not used.
4848
Some static methods (e.g. getName) may even be called, other like toString not.
4949
Not really obvious why.
50+
51+
=== Output
52+
53+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
54+
The expected output is compared with actual output using `verify.sh`.
55+
56+
==== Expected Output
57+
58+
[source]
59+
----
60+
include::expected-result/run.txt[]
61+
----
62+
63+
==== Actual Output
64+
65+
[source]
66+
----
67+
include::run-result/run.txt[]
68+
----

‎jigsaw-examples/example_exceptions/README.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,22 @@ image::moduledependencies.png[Example's Module Dependency Graph]
3636
* Both Exceptions and RuntimeExceptions are used.
3737
* Both cases of exported and non-exported exceptions are shown.
3838
* Chaining of non-exported Exceptions and RuntimeExceptions is shown.
39+
40+
=== Output
41+
42+
This example uses xref:../../README.adoc#golden-master-testing[golden master testing] to ensure output consistency.
43+
The expected output is compared with actual output using `verify.sh`.
44+
45+
==== Expected Output
46+
47+
[source]
48+
----
49+
include::expected-result/run.txt[]
50+
----
51+
52+
==== Actual Output
53+
54+
[source]
55+
----
56+
include::run-result/run.txt[]
57+
----

0 commit comments

Comments
 (0)