Consider magically adding @Testable to JMH annotations

[sbrannen]

The README suggests that users manually add @Testable to benchmark methods.

That of course works; however, it would be nice if @Testable could be magically added to annotations such as @Benchmark transparently for the user -- for example, by modifying the byte code when such annotation classes are loaded, such as via a ClassFileTransformer.

Just a thought...

[mp911de]

Thanks for raising the issue. This would work for non-IDE environments such as Maven/Gradle plugins. In our case, we need to cheat the IDE.

Typically, IDE's use files under META-INF (such as Spring Boot) for static extension information that allow 3rd party implementors to provide metadata for components that are not under their control. JUnit platform assumes that test method/test class annotations are under control of the TestEngine author so she can add @Testable onto these.

Right now, @Testable is an initial cosmetic problem that requires to launch an entire benchmark first to then select the desired benchmark to run. Our JUnit 4 integration allows launching every method as soon as a custom Runner is associated (at least for IntelliJ).