Review and rewrite regeneration.txt help file. #15385

Author: dweiss

help/regeneration.txt

@@ -7,10 +7,11 @@ resource (binary) files, others are Java source files that are stored
 
 If you're reading this, chances are that:
 
-1) you've hit a precommit check error that said you've modified a generated
+1) you've hit a build check error that said you've modified a generated
    resource and some checksums are out of sync.
 
-2) you need to regenerate one (or more) of these resources.
+2) you need to regenerate one (or more) of these resources because of
+   third party library updates or other reasons.
 
 In many cases hitting (1) means you'll have to do (2) so let's discuss
 these in order.
@@ -20,44 +21,58 @@ Checksum validation errors
 --------------------------
 
 LUCENE-9868 introduced a system of storing (and validating) checksums of
-generated files so that they are not accidentally modified. This checksums
+generated files so that they are not accidentally modified. This checksum-checking
 system will fail the build with a message similar to this one:
 
-Execution failed for task ':lucene:core:generateStandardTokenizerChecksumCheck'.
-> Checksums mismatch for derived resources; you might have modified a generated resource (regenerate task: :lucene:core:generateStandardTokenizerIfChanged):
-  Actual:
-    lucene/core/[...]/StandardTokenizerImpl.java=3298326986432483248962398462938649869326
+Execution failed for task ':lucene:core:regenerateStandardTokenizerChecksumCheck'.
+> Checksums mismatch for generated resources; you might have modified a generated resource (regenerate task: regenerateStandardTokenizer, checksum file: .../lucene/core/src/generated/checksums/regenerateStandardTokenizer.json).
 
-  Expected:
-    lucene/core/[...]/StandardTokenizerImpl.java=8e33c2698446c1c7a9479796a41316d1932ceda8
+  Current checksums:
+    - lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java=f4d6311e03a97cb55301cfa55fab3338b1fdb725
+
+  Expected checksums:
+    - lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java=f4d6311e03a97cb55301cfa55fab3338b1fdb724
+
+  Input files for this task are:
+    - .../lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
+    - .../lucene/gradle/regenerate/jflex/skeleton.disable.buffer.expansion.txt
+
+  Files generated by this task are:
+    - .../lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java
 
 The message shows you which resources have mismatches on checksums (in this case
 StandardTokenizerImpl.java) but also the *module* where the generated
-resource exists and the *task name* that should be used to regenerate this resource:
+resource exists, the *task name* that should be used to regenerate this resource
+and the inputs and outputs of that task:
+
+:lucene:core:regenerateStandardTokenizer
 
-:lucene:core:generateStandardTokenizerIfChanged
+To resolve the problem, try to "git diff" the changes that caused the build failure
+(to see why the checksums have changed). Then decide if the change was accidental
+or intentional and either:
 
-To resolve the problem, try to:
+1) revert any accidental changes made to inputs/outputs of generated resources,
 
-1) "git diff" the changes that caused the build failure (to see why the checksums
-changed) and then decide whether to update the generated resource's template (or whatever
-it is using to emit the generated resource);
+2) regenerate the derived resources, possibly saving new checksums.
 
-2) regenerate the derived resources, possibly saving new checksums. If you decide to
-regenerate, just run the task hinted at in the error message, for example:
+If you decide to regenerate, just running the task hinted at in the error message
+should work fine, for example:
 
-gradlew :lucene:core:generateStandardTokenizerIfChanged
+gradlew :lucene:core:regenerateStandardTokenizer
 
 This regenerates all resources the task "generateStandardTokenizer" produces
-and updates the corresponding checksums.
+and updates the corresponding checksums. If you want to bypass any of gradle's
+smart up-to-date checks and force regeneration, use the standard gradle's switch:
+
+gradlew :lucene:core:regenerateStandardTokenizer --rerun-tasks
 
 
 Resource regeneration
 ---------------------
 
-The "convention" task for regenerating all derived resources in a given
-module is called "regenerate" and you can apply it to all Lucene modules
-by running:
+Lucene's "convention" task for regenerating all derived resources in a given
+module is called "regenerate" and you can apply it to all Lucene modules at
+once by running:
 
 gradlew regenerate
 
@@ -105,18 +120,12 @@ gradlew -p lucene/analysis/common regenerate --console=plain
 ...
 
 This shouldn't worry you at all - the internal tasks are skipped because
-the inputs and outputs of these task have not changed. If they have changed,
-the task is re-run and followed up by other tasks, such as code-formatting (tidy).
-
-Of course, sometimes you may want to *force* the regeneration task to run, even if the
-checksums indicate nothing has changed. This may happen because of several reasons:
-
-- the generation task has outputs but no inputs or the inputs are volatile. In this case
-only the outputs have checksums and the task will be skipped if the outputs haven't changed.
+the inputs and outputs of these task are up to date. If changes were detected, the task
+would be re-executed, followed by any necessary cleanup tasks (like code formatting).
 
-- you may want to run the regeneration task just to see that it actually runs and produces
-the same checksums (git diff should be clean). This would be a wise periodic sanity check
-to ensure everything works as expected.
+Sometimes you may want to *force* the regeneration task to run, even if the
+checksums indicate nothing has changed. This may happen if you're paranoid or don't trust
+gradle (and there's no reason you should).
 
 If you want to force-run the regeneration, use gradle's "--rerun-tasks" option:
 
@@ -131,7 +140,7 @@ Scoping the call to a particular task will also work:
 gradlew -p lucene/analysis/common regenerateUnicodeProps --rerun-tasks
 
 Finally, if you do feel like force-regenerating everything, remember to exclude this
-monster...
+monster (it takes quite a long time and requires significant heap memory)...
 
 gradlew regenerate -x regenerateUAX29URLEmailTokenizer --rerun-tasks