Run expensive Traversal verification sanity check less often (#3442)

A sanity check to validate the traversal integrity was recently added in
#3435. That check alone seems to almost double the total time it takes
to run the `core-tests` during PRB from around 15 minutes to around 30
minutes. It's a useful check, but it is probably running too often. We
should be able to get away with only running it occaisionally and then
only enabling it more stringently if the need arises.

Author: alecgrieser

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/CascadesPlanner.java

@@ -437,11 +437,22 @@ private void planPartial(@Nonnull final Supplier<Reference> referenceSupplier,
                     Debugger.withDebugger(debugger -> debugger.onEvent(nextTask.toTaskEvent(Location.BEGIN)));
                     try {
                         nextTask.execute();
-                        Debugger.sanityCheck(() ->
-                                // This is a fairly expensive check, but it makes sure that the traversal
-                                // is up to date with the query DAG. This is used during memoization, so
-                                // it's important that it has an accurate view of the state of the world.
-                                traversal.verifyIntegrity());
+                        Debugger.sanityCheck(() -> {
+                            if (nextTask instanceof InitiatePlannerPhase) {
+                                //
+                                // Validate that the memo structure and the query graph are aligned.
+                                // This is an expensive check, but if it finds anything, then it may
+                                // be useful to enable this check so that it runs after every task
+                                // to root out the underlying issue.
+                                //
+                                // Once we have sanity check levels, we should consider enabling
+                                // this check all the time at certain levels.
+                                // See: https://github.com/FoundationDB/fdb-record-layer/issues/3445
+                                //
+                                traversal.verifyIntegrity();
+                            }
+                        });
+
                     } finally {
                         Debugger.withDebugger(debugger -> debugger.onEvent(nextTask.toTaskEvent(Location.END)));
                     }
@@ -472,6 +483,9 @@ private void planPartial(@Nonnull final Supplier<Reference> referenceSupplier,
                 pushInitialTasks();
             }
         }
+
+        // Also validate memo structure integrity at the end of planning
+        Debugger.sanityCheck(() -> traversal.verifyIntegrity());
     }
 
     private void pushInitialTasks() {