Explore idea to parse only those files present in Checkstyle report

[Anmol202005]

Problem

Recipes currently parses all files, which is slow. We only need files that appear in the checkstyle report.

proposed solution:

Use preconditions to filter files based on checkstyle report:

@Override
public TreeVisitor<?, ExecutionContext> getVisitor() {
    return Preconditions.check(
            Preconditions.not(...), // exclude files not in report
            new UpperEllVisitor()
    );
}

Need to implement proper condition logic to check against the report.

[romani]

@timtebeek, is there any approach to focus open rewrite to parse only special set of files?
in runtime of open rewrite run, not in configuration before run.

We experienced strange performance problems during compilation phase or rewrite. It is visible 20 seconds of some compilation, but we definitely don't need all files to be compiled before execution of rewrite itself.

[timtebeek]

Hi! There's two different ways to limit the results you're seeing from recipe outputs:

• exclusions, configured on the build tool, which prevent files from being parsed at all, which is the time consuming step
• preconditions, configured on the recipe, which prevents making changes to files after they have already been parsed

When you have say a file A, B and C, and A uses B and C, then excluding any of those files could lead to missing type information in the others, making recipe runs less reliable such as for instance a failure to match, or hesitant to remove imports. With preconditions this isn't a problem, but they do not significantly speed up a recipe run, just the results you'd see.

In short there's no guaranteed safe way to speed up parsing a project, and the effects you'd see with exclusions might be subtle and fragile. We do recommend building up a full LST model wherever possible, but recognize that might not be feasible for large projects. At Moderne specifically we have solved that through persistence of the model, but that part is proprietary to Moderne, yet free to use for OSS projects.

[romani]

exclusions

We can use this approach, unfortunately it requires extra script run to generate such content.
But it will be very beneficial. We can reuse limitation of checkstyle that all validation is done based on single file, no more complicated cases that general open rewrite is targeting is outside of checkstyle scope.

@timtebeek, why compilation is required for open rewrite? Does open rewrite put something in code during compilation? Or is just to be sure that sources are good and AST can be built in good way to let all recipes to execute well.

Can we disable compilation inside open rewrite? If we confident that we will not run it on non compile-able sources.

[timtebeek]

Compilation is required for type attribution, and type attribution is required for matching as well as preconditions like the following:
https://github.com/openrewrite/rewrite/blob/21e3bad16904dfc05383747794f8e51866b1dcca/rewrite-java/src/main/java/org/openrewrite/java/RemoveUnusedImports.java#L69-L71

Without compilation only a subset of recipes would work well, and it's hard to say exactly how each component might be affected. Just sharing for context here, you're welcome to do as you please of course (even if not advised), but for OpenRewrite to work well we do require that compilation step.