add support for repairing Qodana warnings

[monperrus]

It'd be great to add support for repairing Qodana bug warnings

Qodana: https://www.jetbrains.com/qodana/

Some processors already exist in https://github.com/MartinWitt/laughing-train by @MartinWitt

[monperrus]

This would come under the abstraction https://github.com/SpoonLabs/sorald/blob/master/src/main/java/sorald/rule/StaticAnalyzer.java

[MartinWitt]

So after some thoughts I believe there are 2 ways to integrate the laughing-train code or qodana in sorald. The fast way is to integrate the current refactorings with their own detection.
The more advanced way is using some kind of sarif parser using the qodana sarif output https://www.jetbrains.com/help/qodana/qodana-sarif-output.html. Qodana could be run with https://github.com/JetBrains/qodana-cli only requiring docker installed or requiring the sarif file provided by the user.

In the sarif file we could match the rules by their ID and fix the bad smell with their position. This would allow sorald to fix a lot more bad smells like value always null or other very advanced IntelliJ stuff. (A sarif parser could be https://github.com/Contrast-Security-OSS/java-sarif)

The first approach yields faster results the second needs more time and should fix more bad smells(assuming there is no legal stuff hindering us). Which approach do you prefer?

[MartinWitt]

I'm currently experimenting with the second option, and kinda works. The current results and automatic fixable bad smells for some projects can be found here https://github.com/MartinWitt/laughing-train/tree/gh-mining. The real benefit from using qodana are bad smells like https://github.com/MartinWitt/laughing-train/blob/gh-mining/spoon.md#unnecessary-return, where information flow is necessary to detect.

[MartinWitt]

So after some more experimenting:

How to integrate Qodana

More or less simple if we assume a docker host is running. Pull the image if it isn't available local and run it with the given parameters. After the Qodana run is finished, simply parse the SARIF file(there exist some good libs for it). Then convert the sarif rules to Sorald rules. A variant without docker does not exist to my knowledge. If creating an own container wrapping the Qodana container(docker in docker) or simple invocation is more or less a style question.

The Interface Problem

The Sorald interface for an analyzer is a list of files, some rules and classpath.
Collection<RuleViolation> findViolations(List<File> files, List<Rule> rule, List<String> classpath):
The Qodana interface is more or less
SarifFile analyze(String projectRoot, String pathToQodanaRulesFile, String pathToAnalyze)
As you see, Qodana is project based, Sorald is file based. Qodana needs the whole project and not only some files.
So Qodana requires a new interface, or we simply abuse the contract and say the first file is the project root.

The Qodana Runtime

Qodana is slow. This is because Qodana invokes more or less a full build before analyzing. For complex builds like https://github.com/IntellectualSites/FastAsyncWorldEdit this has horrible runtime. I haven't been able to finish the analyse step in 30 minutes for FAWE (on Windows with a gen 12 intel CPU). This could be either an WSL issue or Qodana is that slow. For spoon, it requires around 6 min in the CI. So we have to ask us does Qodana generate that much value, that this runtime is worth it? Has this more than scientific benefit and do we hope JetBrains can improve their runtime.

PS:
Interesting question I never really thought about, but is there a comparison how much worse a no-classpath analyze with Spoon is against a more complex like Qodana is using. Most important bad smells should be detectable without a classpath. Is there any research about it?

[monperrus]

Thanks a lot for the analysis.

the interface problem.

What about adding Collection<RuleViolation> findViolations(Path projectRoot, List<Rule> rule, List<String> classpath):? Would you PR this?

how much worse a no-classpath analyze with Spoon is against a more complex like Qodana is using

I'm not aware of such a comparison. But I'm note sure the Spoon no-classpath ls 'less complex' than Qodana's.

[MartinWitt]

What about adding Collection findViolations(Path projectRoot, List rule, List classpath):? Would you PR this?

I would add this interface after I created a working example for Qodana in Sorald. Otherwise, we have no proof if this fits.
I saw Qodana in Sorald is a GSoC task, shall we wait if someone applies to this task?

[MartinWitt]

As the GSoC proposal was rejected, this task is now again a hot topic. We should first define a goal for a first rule we have as initial target. The bad smell should be detected by Qodana and fixed by Sorald. I would propose using the redundant toString() check. From my current knowledge, this check only exists in sonarlint in C# but not for Java(see https://rules.sonarsource.com/csharp/RSPEC-1858). Furthermore, this bad smell is present in many projects and (should) never gets rejected if you create a PR fixing it. The code for fixing this bad smell is relatively short, and the PR focus keeps on the docker/Qodana integration.
Wdyt?

[algomaster99]

I agree with @MartinWitt that we should start with fixing a rule which has an easy repair so that we can focus on the docker/Qodana integration.

But before we do that, should we generalise the app using Java SPI as you have suggested here? It does sound like a good idea.

[MartinWitt]

This is the million-dollar question. It depends on the research scope, goal and future vision of the project. Are there any actual plans to integrate more analyzers into Sorald? Is this a long living project or paper and maintenance mode? Adding this new abstractions costs time with no direct benefit but would boost future development. Adding it later costs more time because it is more complex, but you have intermediate results. As it is hard for me to answer questions about your research, it's your choice. Depending on your other answers, we can choose the best option.

Edit: And we should clarify first if we can hard break with the current Sorald design and classes. Adding SPI is 100% a breaking API change.

[algomaster99]

Is this a long living project or paper and maintenance mode? Adding this new abstractions costs time with no direct benefit but would boost future development. Adding it later costs more time because it is more complex, but you have intermediate results

I think @monperrus can weigh in better on this. My opinion was to generalise the app first so that it is easier to integrate other static analyzers (as we have claimed in the paper). Moreover, we do not have any set deadlines as of now that we have to integrate Qodana. So if we plan to integrate another static analyzer, we do it in the most generic way possible.

And we should clarify first if we can hard break with the current Sorald design and classes

I was trying that, but I was having trouble designing the interfaces. I have summarised the gist over here. As a short term goal, I have decided to remove Checks.java because the map inside it is annoying me. See #722.

[monperrus]

Is this a long living project or paper and maintenance mode?

We don't know yet. For now, the highest priority is to get the the paper formally published, and we're very close to this (minor revision).

[algomaster99]

@monperrus @MartinWitt We realised in the meeting with SonarSource that we were a bit late in presenting our work because they had already started implementing code fixes. To my knowledge, I don't think so there is a repair tool by JetBrains to fix Qodana warnings. We could probably keep that on priority.

[MartinWitt]

To my knowledge, I don't think so there is a repair tool by JetBrains to fix Qodana warnings.

The repair tool is more or less IntelliJ. But you are correct, there is no headless/non-ide version.

[algomaster99]

The repair tool is more or less IntelliJ.

Yes, I agree. They have techniques to perform quick fixes, but not specifically for Qodana violations. This paper here cites two repair tools built as IntelliJ plugins. However, they focus on some code smells. I am not sure if they are dictated by Qodana or not.

[MartinWitt]

I'm currently testing the SPI refactoring inside my fork of the repo. It seems like the first goal to refactor Sorald as a platform for analyzers and repair is to use SPI for sonar-java and break the monolithic design.

This paper here cites two repair tools built as IntelliJ plugins. However, they focus on some code smells. I am not sure if they are dictated by Qodana or not.

From my short look at it, we would differentiate from them in the assumptions. They assume the usage of IntelliJ, and we assume the existence of a static analyzer. But I will read the paper closer and see how far they are.