Classloading Problems with Java 25 and Virtual (System?) Threads

[laeubi]

I encountered a strange problem recently that when a Virtual Thread or System Thread (I'm not sure i maybe both are the same) is trying to load some dependency it fails with ClassNotFoundException or ClassCastException on Java 25.

The strange thing is that one can "fix" it e.g. by having an activator in the bundle that loads triggers the classload, or any other kind of code. I first noticed this with parallel streams that suddenly throw an exception where I though it was a configuration problem and it vanished as soon as I added some debugging (outside) the loop. In one case the not found class was even in the same bundle and tried to be loaded via the Java service loader.

I have not yet a good reproducer here but maybe it rings a bell to someone (@tjwatson ?) what could be the case, maybe some API changed or is unavailable?

[tjwatson]

Does the scenario involve buddy class loading? I seem to remember the built-in executor used by the parallel streams in the JVM causing issues. I think buddy classloading was involved, but I am not sure. I was trying to find the bug/issue/discussion, but couldn't find it. I think it was reported by @martinlippert but I could be wrong. Martin, do you recall some issues you would having with the parallel streams APIs with Equinox?

[laeubi]

Does the scenario involve buddy class loading?

I don't think so or at least I'm not aware of it. I noticed it with this bundle when the first classload trying to load DefaultSystemOfUnitsService, for example one uses tech.units.indriya.unit.Units.HOUR somewhere inside a parallel stream it bails out with "class can not be found by bundle".

Adding an activator class that simply do var bogus = tech.units.indriya.unit.Units.HOUR; everything works.

I seem to remember the built-in executor used by the parallel streams in the JVM causing issues.

I'm only aware ob the issue with security manager what should not be an issue here hopefully and that they somewhat do not inherit context classloaders. Maybe that's the missing puzzle here? I might try to reproduce the issue again but though I have checked that already.

[tjwatson]

Maybe it isn't buddy classloading but I do think there is something that can go wrong with the use of the org.eclipse.osgi.internal.framework.ContextFinder and virtual threads. Equinox tries to make sure this is the thread context class loader is set as the thread context class loader for the framework managed threads. But if the built in executor for streams is being used it would not have the context class loader set to org.eclipse.osgi.internal.framework.ContextFinder.

[tjwatson]

It wasn't Martin (sorry for the noise Martin) that opened the bug I was thinking of: https://bugs.eclipse.org/bugs/show_bug.cgi?id=569434

[martinlippert]

No worries - even though blood pressure is still slightly going up when I read buddy classloading... 😂
Hope you can get to the bottom of this.

[laeubi]

I made some experiments now and in Java 21 the ForkJoinPool uses regular Threads:

In Java 25, they become "System Threads" (whatever that means)

The Javaodoc is not enlightening here it only says

If no thread factory is supplied via a system property, then the common pool uses a factory that uses the system class loader as the thread context class loader.

Comparing it with Java 21 the only difference seems to be the removal of the note about security manager what seems obvious.

As there is a java.util.concurrent.ForkJoinPool.common.threadFactory systempropety maybe Equinox can supply a OSGiThreadFactory here so users can enable it for Java 25 as well to get old behaviour?

[tjwatson]

As there is a java.util.concurrent.ForkJoinPool.common.threadFactory systempropety maybe Equinox can supply a OSGiThreadFactory here so users can enable it for Java 25 as well to get old behaviour?

I am unsure because I don't really know what classloading problem you are having or why the thread context class loader is being used in your scenario.

Have you confirmed what class loader is being used when you get classloading problems? Do you have the exception? Have you tried with equinox classloading trace enabled? Can you use a debugger when the problem happens?

[laeubi]

@tjwatson I now tried to generate a reproducer (or more ways to explore) the problem more systematically:

• Add reproducer tests and workaround demos for ContextFinder TCCL missing on ForkJoinPool threads (#1243) laeubi/equinox#152

There is a standalone maven project that can hopefully show the results better here, does it help in answer your questions? I was not able to really reproduce it with Java 21 vs 25 it seem to behave very similar here so I'm not totally sure why it only happens now.

Having a custom ThreadFactory activated with the system property seems useful in any case, so I'm currently lean to add such thing to equinox so users can just set a system property to mitigate such problems?

[tjwatson]

Having a custom ThreadFactory activated with the system property seems useful in any case, so I'm currently lean to add such thing to equinox so users can just set a system property to mitigate such problems?

I don't know how this would work in a general way where the framework itself is not on the JVM classpath. For example, the only thing on the JVM classpath for a typical Eclipse launch is the equionix launcher JAR. That is the only place where one could set the java.util.concurrent.ForkJoinPool.common.threadFactory to a class that the JVM can use for the built in Java ForkJoinPool instances. Therefore don't think the framework is the right place to do this. It has to happen way before the framework is activated AND the impl has to be loadable by the JVM system class loader.

Even for scenarios that do have the framework directly on the JVM system classpath, there is the issue of running multiple Framework instances on the same JVM. Or what to do if the framework is embedded and the java.util.concurrent.ForkJoinPool.common.threadFactory has already been set and created before the framework could interject.

[laeubi]

I don't know how this would work in a general way where the framework itself is not on the JVM classpath.

How is the framework started when it is not on the classpath? So maybe my explanation was just missleading, I just want ti to be in the jar that provides the framework e.g. in this case org.eclipse.osgi because that has to be on the classpath usually.

Even for scenarios that do have the framework directly on the JVM system classpath, there is the issue of running multiple Framework instances on the same JVM.

That's for sure not work for all cases anyways we have something similar for the URLHandlers that even work across instances.

[tjwatson]

How is the framework started when it is not on the classpath? So maybe my explanation was just missleading, I just want ti to be in the jar that provides the framework e.g. in this case org.eclipse.osgi because that has to be on the classpath usually.

Not sure what you mean by usually. While the framework CAN be on the classpath; there are many very popular usages of the framework where it is NOT on the classpath at all. One such case is the Equinox IDE when using the Equinox launcher. The launcher uses a StartupClassLoader to load up the framework:

equinox/bundles/org.eclipse.equinox.launcher/src/org/eclipse/equinox/launcher/Main.java

Lines 611 to 626 in d63de88

	URLClassLoader loader = new StartupClassLoader(bootPath, parent);
	Class<?> clazz = loader.loadClass(STARTER);
	Method method = clazz.getDeclaredMethod("run", String[].class, Runnable.class); //$NON-NLS-1$
	try {
	method.invoke(clazz, passThruArgs, splashHandler);
	} catch (InvocationTargetException e) {
	if (e.getTargetException() instanceof Error error) {
	throw error;
	} else if (e.getTargetException() instanceof Exception exception) {
	throw exception;
	} else {
	//could be a subclass of Throwable!
	throw e;
	}
	}
	}

[laeubi]

Not sure what you mean by usually. While the framework CAN be on the classpath; there are many very popular usages of the framework where it is NOT on the classpath at all. One such case is the Equinox IDE when using the Equinox launcher. The launcher uses a StartupClassLoader to load up the framework

Then such factory would be part of the launcher bundle. What I mean is that something must be on the classpath to boot up the Application and such thing can contain a factory class that maybe handle this more graceful.