The 'auto build' should not use Maven's target folder

[anthonyvdotbe]

The fact that the 'auto build' uses Maven's target folder is problematic. It causes independent Maven builds to fail in mysterious ways. Or even worse: the build might succeed, but the build artifacts may differ from the ones that are expected (e.g. ecj-compiled classes may end up in build artifacts, even though the Maven build itself uses javac).

Environment

• Operating System: Windows 10
• JDK version: JDK 14
• Visual Studio Code version: 1.44.0
• Java extension version: 0.59.1

Steps To Reproduce

As an example of the build that might fail:

1. git clone https://github.com/validator/htmlparser.git
2. Add Folder to Workspace... -> htmlparser
3. open a Terminal (inside VS Code) & set the JAVA_HOME environment variable to a JDK 8 installation, e.g. $Env:JAVA_HOME='C:\Program Files\Java\jdk1.8.0_202'
4. in the pom.xml, add the following under build.plugins & ignore any messages about build synchronization

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-javadoc-plugin</artifactId>
        <version>3.2.0</version>
        <configuration>
          <doclint>none</doclint>
        </configuration>
      </plugin>

5. run mvn clean source:jar javadoc:jar package repository:bundle-create in the terminal

Note: this will not fail consistently, and quite often it will also just work. But that's simply because it's about uncontrollable interference between 2 processes.

Current Result

I've seen the build fail with 3 different kinds of errors already (note that the first one explicitly mentions that the failure is possibly caused by ecj-generated classes):

[INFO] --- maven-bundle-plugin:2.3.7:bundle (default-bundle) @ htmlparser ---
[ERROR] Bundle nu.validator.htmlparser:htmlparser:bundle:1.4 : The default package '.' is not permitted by the Import-Package syntax.
 This can be caused by compile errors in Eclipse because Eclipse creates
valid class files regardless of compile errors.
The following package(s) import from the default package [nu.validator.htmlparser.io]
[ERROR] Error(s) found in bundle configuration
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ htmlparser ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 25 source files to C:\Users\me\Desktop\htmlparser\target\test-classes
[INFO] Some messages have been simplified; recompile with -Xdiags:verbose to get full output
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /C:/Users/me/Desktop/htmlparser/test-src/nu/validator/htmlparser/test/TokenPrinter.java:[44,8] nu.validator.htmlparser.test.TokenPrinter is not abstract and does not override abstract method endTag(ElementName) in nu.validator.htmlparser.common.TokenHandler
[ERROR] /C:/Users/me/Desktop/htmlparser/test-src/nu/validator/htmlparser/test/TokenPrinter.java:[140,28] cannot access Tokenizer
  class file for Tokenizer not found
[ERROR] /C:/Users/me/Desktop/htmlparser/test-src/nu/validator/htmlparser/test/TokenizerTester.java:[101,29] incompatible types: nu.validator.htmlparser.impl.ErrorReportingTokenizer cannot be converted 
to Tokenizer
[ERROR] /C:/Users/me/Desktop/htmlparser/test-src/nu/validator/htmlparser/test/JSONArrayTokenHandler.java:[41,8] nu.validator.htmlparser.test.JSONArrayTokenHandler is not abstract and does not override 
abstract method endTag(ElementName) in nu.validator.htmlparser.common.TokenHandler
[INFO] 4 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[INFO] --- maven-bundle-plugin:2.3.7:bundle (default-bundle) @ htmlparser ---
[ERROR] An internal error occurred
java.io.FileNotFoundException: C:\Users\me\Desktop\htmlparser\target\classes\nu\validator\htmlparser\annotation\package.html (The system cannot find the file specified)
    at java.io.FileInputStream.open0 (Native Method)
    at java.io.FileInputStream.open (FileInputStream.java:195)
    at java.io.FileInputStream.<init> (FileInputStream.java:138)
    at aQute.lib.osgi.FileResource.openInputStream (FileResource.java:15)
    at aQute.lib.osgi.FileResource.copy (FileResource.java:36)
    at aQute.lib.osgi.FileResource.write (FileResource.java:31)
    at aQute.lib.osgi.Jar.writeResource (Jar.java:445)
    at aQute.lib.osgi.Jar.write (Jar.java:224)
    at aQute.lib.osgi.Jar.write (Jar.java:192)
    at org.apache.felix.bundleplugin.BundlePlugin.execute (BundlePlugin.java:362)
    at org.apache.felix.bundleplugin.BundlePlugin.execute (BundlePlugin.java:264)
    at org.apache.felix.bundleplugin.BundlePlugin.execute (BundlePlugin.java:255)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Expected Result

The build always succeeds and none of the build artifacts contain any ecj-generated classes.

[wheezil]

+1

We hit this all the time on Windows, and after blaming our anti-virus, backup software and everything else we could think of, we finally stopped using VS.code for java and everything was better. This plugin in VS.code seems to interfere with maven builds on the command-line, leading to all sorts of weird errors, especially where there are code-generation steps involved. It often manifests as "generated class does not exist" or "generated class doesn't implement an abstract method" or "generated package doesn't exist", but when we go look at the offending generated item, it is always fine after the fact. I've also seen errors reported in target class files, as in "the file XXX.class contains class YYY".

[DeegC]

I have the same problem as wheezil on Linux. It seems worse in projects with code generation. On vanilla projects vscode is ok but in workspaces with code generation it is near unusable.

[testforstephen]

@wheezil and @DeegC thank you for reaching out. Would you mind sharing more about the specific problems you have with code generation in VS Code? Is that your 3rd party maven code generation plugin is not well recognized by VS Code Java extension?

[wheezil]

Hi @testforstephen. The issue doesn't really manifest in VS.code itself. It is more like VS code is continuously building and/or/analyzing the java code, and that activity (at least on Windows) seems to occur simultaneously with a "mvn install" performed on the command-line, and interferes with the maven build. I suspect that our code-generation causes updated files to be seen by VS code, and it responds immediately with its building/scanning operation to keep up to date. However, this operation (apparently) opens files in such a way that it prevents or delays visibility of files created by our code-generator or javac.

We use this maven plugin to integrate the generated code:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <version>3.2.0</version>
  <executions>
    <execution>
      <id>add-source</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>add-source</goal>
      </goals>
      <configuration>
        <sources>
          <source>${project.build.directory}/generated-sources/java/</source>
          <source>${project.build.directory}/generated-sources/resources/</source>
        </sources>
      </configuration>
    </execution>
  </executions>
</plugin>

Our code-generation step is custom, basically ingesting a service IDL and generating java.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <executions>
    <execution>
      <id>ipc-generator</id>
      <phase>generate-sources</phase>
      <configuration>
      ...
      </configuration>
      <goals>
        <goal>exec</goal>
      </goals>
    </execution>
  </executions>
</plugin>

This tool tries to do a "minimal update" of the target generated code, by first writing to a temporary file, and comparing to the target. If the target is missing or different, it then deletes the target and renames the temp to the target. I have seen before on Windows, that if someone has a file open, you are allowed to delete it and rename another file onto it, but global visibility of the change is somehow delayed until the file handle is closed. This may influence the issue.

However, while generated code is most often seen as being involved in the issue, this is not always the case. I've also seen that .class files created during the build are reported as corrupt by later stages that attempt to use them.

In all cases, when we investigate the error reported by the build, the files in question are perfectly fine. It is as if the build is getting a stale or transient snapshot of the build-product files, seeing incorrect content during a very brief window.

Finally, we don't have this issue on NetBeans, although it has other problems keeping up to date with generated code -- seems like nobody really has got this right.

[DeegC]

I don't have too much to add; wheezil covered most of it. I use Java Immutables. If I run mvn install (or test, etc) while vscode is open then the build never works. The errors can be different but it's usually the compiler complaining about a missing generated file. I have to turn off autobuild in vscode and run clean to get it maven to finish without errors.

The reverse is also a problem; running maven will screw up vscode so it can't find generated/compiled files. Any time I want to run maven commands I'm forced to go through a shuffle of:

1. Turn off vscode's autobuild.
2. mvn clean install
3. When I'm done, turn on autobuild back on.
4. Force clean and delete of env in vscode.

You guys would know better than me but it seems this could be avoided by having vscode use a different target directory.

[fbricon]

As a possible workaround, add the following bits to your project('s parent) pom.xml:

<properties>
   ...
  <build.output>target</build.output>
</properties>
....
<build>
  <directory>${build.output}</directory>
  ...
</build>
...
<profiles>
  <profile>
    <activation>
      <property><name>m2e.version</name></property>
    </activation>
    <properties>
      <build.output>target-ide</build.output>
    </properties>
  </profile>
...
<profiles>

this allows the build output directory to be overridden to target-ide when running in m2e (vscode-java, Eclipse IDE, vim...), while maintaining the default target during CLI builds. Don't forget to add target-ide to your .gitignore

[wheezil]

@fbricon thanks, that is promising. I have to admit I do not know what this is doing:

    <activation>
         <property><name>m2e.version</name></property>
    </activation>

Do the IDEs automatically inject this property or do I need to configure them to define the property?
Do you know if this also works with Netbeans? We have related but different problems there.
john

[fbricon]

It'll only work for IDEs based on Eclipse m2e. m2e injects the m2e.version property to the Maven reactor during incremental builds. So it's possible to activate certain profiles by checking for the presence of the m2e.version property in the build. In our case, that profile, when running in the context of m2e, overrides the build directory.
I don't know/use Netbeans, so I'm not aware of a similar mechanism in that IDE.

[testforstephen]

@wheezil @DeegC Thank you for sharing the specific details about this issue. Yes, the property m2e.version is automatically injected by VS Code Java tooling when importing your project into VS Code, no need to configure it by yourself. Could you have a try on @fbricon's suggestion? thanks.

[wheezil]

@testforstephen thanks for the update. I'm not the VS.code user in our development group, but I will relay the information. Since it involves a change to the pom.xml files and not just user settings.xml it might take some negotiation to make the change.

[wheezil]

Good news! The workaround by @fbricon seems to work for us.

[MarcelMerkle]

I have the exact same problem. It would be really nice to have a workaround, or better solution, that does not involve changing the actual build files.