OutOfMemory errors in PDE build since Tycho 5.0.1-SNAPSHOT

[HannesWell]

Since we upgraded to Tycho 5.0.1-SNAPSHOT in #3432, the PDE master build is failing constantly with OutOfMemoryErrors when building the o.e.pde.ui jar:

 [INFO] --- tycho-packaging:5.0.1-SNAPSHOT:package-plugin (default-package-plugin) @ org.eclipse.pde.ui ---
 [INFO] Building jar: /home/jenkins/agent/workspace/eclipse.pde_master/ui/org.eclipse.pde.ui/target/org.eclipse.pde.ui-3.16.300-SNAPSHOT.jar
 [578.399s][warning][gc,alloc] ForkJoinPool-1-worker-10: Retried waiting for GCLocker too often allocating 62502 words
 java.lang.OutOfMemoryError: Java heap space
 Dumping heap to jvmHeapDump.hprof ...
 Heap dump file created [1201636756 bytes in 8.056 secs]
 
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-9-SelectorManager"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-7-SelectorManager"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-10-SelectorManager"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-1-SelectorManager"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-0"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-8-SelectorManager"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-1"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-6-SelectorManager"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-2-SelectorManager"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
 Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-3-SelectorManager"

Running the PDE build with -Dtycho.version=5.0.0 makes it succeed.

Since one I-build also failed in a similar way, I create this general issue:

 [INFO] --- tycho-packaging:5.0.1-SNAPSHOT:package-plugin (default-package-plugin) @ org.eclipse.ant.optional.junit ---
 [ERROR] Java heap space -> [Help 1]
 java.lang.OutOfMemoryError: Java heap space
 [ERROR] java.lang.OutOfMemoryError: Java heap space
 java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
     at java.util.concurrent.FutureTask.report (FutureTask.java:122)
     at java.util.concurrent.FutureTask.get (FutureTask.java:191)
     at io.takari.maven.builder.smart.ProjectExecutorService.take (ProjectExecutorService.java:97)
     at io.takari.maven.builder.smart.SmartBuilderImpl.build (SmartBuilderImpl.java:123)
     at io.takari.maven.builder.smart.SmartBuilder.build (SmartBuilder.java:109)
     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
     at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
     at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
     at java.lang.reflect.Method.invoke (Method.java:580)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
 Caused by: java.lang.OutOfMemoryError: Java heap space

For PDE I captured a heap-dump and let it print all JVM flags for further investigation (by adding the environment variable MAVEN_OPTS = '-XX:+PrintFlagsFinal -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=jvmHeapDump.hprof' and archiving jvmHeapDump.hprof:

• https://ci.eclipse.org/pde/job/eclipse.pde/job/master/20
• https://ci.eclipse.org/pde/job/eclipse.pde/job/master/20/artifact/jvmHeapDump.hprof

Because the heap-dump is more than 1GiB in size, I just add that link and don't attatch it here. So everyone interested in investigatiting it, should download it before it's deleted (or I rerun a build with that option again).

[laeubi]

Might be because of

• Fix ThreadLocal memory leak in ParallelScatterZipCreator causing OOM codehaus-plexus/plexus-archiver#400

[HannesWell]

That sounds like a possible cause. At least the plexus-archiver is high in the list of objects retaining a lot of memory:

[HannesWell]

Might be because of

* [Fix ThreadLocal memory leak in ParallelScatterZipCreator causing OOM codehaus-plexus/plexus-archiver#400](https://github.com/codehaus-plexus/plexus-archiver/pull/400)

I'm trying to downgrade our build to use plexus-archiver 4.10.0, the version before the assumed cause of the bug was introduced, by setting it as explicit dependency of tycho-packaging.

[HannesWell]

The workaround implemented in #3437, seems to be successful.
Thanks to @laeubi for the hint about adding adependency.

Still I'll leave this open as a reminder to remove it again, when the fix arrived in Tycho.

[iloveeclipse]

The workaround implemented in #3437, seems to be successful.

I see constant OOM issues with platform ui build. Do we need some fix/workaround there, or why #3437 was OK for PDE but not for platform UI?

[laeubi]

I see constant OOM issues with platform ui build.

Please give example! Some builds use other mojos (e.g. for web-app building) that need separate workaround/fixes, so it is important to check at what stage we see the issue

[iloveeclipse]

Please give example!

See above link to eclipse-platform/eclipse.platform.ui#3460

[laeubi]

From the build output I suspect that tycho-source plugin is the culprit here and needs a downgrade of plexus as well ... in general we could consider to downgrade plexus lib in Tycho 5 ... let me check.

See

• Downgrade plexus-archiver to 4.10.0 eclipse-tycho/tycho#5539

[iloveeclipse]

general we could consider to downgrade plexus lib in Tycho 5

Is there some deployment step needed? There are still OOM's, see https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-3460/3/pipeline-overview/