StarDist3D

[stefanhahmann]

I tried running StarDist3D with a small dummy dataset:

import java.io.IOException;
import java.net.URISyntaxException;

import net.imglib2.img.Img;
import net.imglib2.img.array.ArrayImgs;
import net.imglib2.type.numeric.real.FloatType;

import org.apache.commons.compress.archivers.ArchiveException;

import io.bioimage.modelrunner.apposed.appose.MambaInstallException;
import io.bioimage.modelrunner.bioimageio.description.exceptions.ModelSpecsException;
import io.bioimage.modelrunner.exceptions.LoadEngineException;
import io.bioimage.modelrunner.exceptions.LoadModelException;
import io.bioimage.modelrunner.exceptions.RunModelException;
import io.bioimage.modelrunner.model.Stardist3D;

public class Stardist3DDummyPrediction
{
	public static void main( String[] args )
			throws MambaInstallException, IOException, URISyntaxException, ArchiveException, InterruptedException, ModelSpecsException,
			LoadEngineException, RunModelException, LoadModelException
	{
		Stardist3D.installRequirements();
		Stardist3D stardist3D = Stardist3D.fromPretained( "StarDist Plant Nuclei 3D ResNet", false );
		final Img< FloatType > dummyImg = ArrayImgs.floats( 10, 10, 10 );
		stardist3D.predict( dummyImg );
	}
}

When trying to run this, I get:

Exception in thread "main" Error loading a Deep Learning engine located at tensorflow-1.15.0-1.15.0-windows-x86_64-cpu-gpu.
java.lang.reflect.InvocationTargetException
	at io.bioimage.modelrunner.engine.EngineLoader.setEngineInstance(EngineLoader.java:330)
	at io.bioimage.modelrunner.engine.EngineLoader.<init>(EngineLoader.java:138)
	at io.bioimage.modelrunner.engine.EngineLoader.createEngine(EngineLoader.java:159)
	at io.bioimage.modelrunner.model.Model.setEngineClassLoader(Model.java:480)
	at io.bioimage.modelrunner.model.Model.<init>(Model.java:142)
	at io.bioimage.modelrunner.model.Model.createBioimageioModel(Model.java:319)
	at io.bioimage.modelrunner.model.Model.createBioimageioModel(Model.java:258)
	at io.bioimage.modelrunner.model.Model.createBioimageioModel(Model.java:236)
	at io.bioimage.modelrunner.model.Stardist3D.predict(Stardist3D.java:245)
	at main(Stardist3DInstallRequirements.java:28)

This behavior is unexpected. It happens on Windows machine with JDK-21 (not sure, if this OS related or Java-version related).

[stefanhahmann]

@carlosuc3m any ideas how to solve this? I tried investigating a bit, but did not get very far.

I am surprised a little bit that the above downloads dl-modelrunner-tensorflow-1-0.4.2-20250103.161616-23.jar instead of the latest release https://maven.scijava.org/service/local/artifact/maven/redirect?r=releases&g=io.bioimage&a=dl-modelrunner-tensorflow-1&v=0.4.1&e=jar

[stefanhahmann]

I found out that with the 0.5.8 release, when dl-modelrunner-tensorflow-1-0.4.1.jar was used the above mentioned example actually works. After that, it seems to be broken.

[carlosuc3m]

HEllo @stefanhahmann

First of all sorry for having forgotten about this topic for so long. Thanks to your reminder i am back on it.

What are you using to run this code snippet? I have just tried with the latest snapshot and it seems to work.

I am surprised a little bit that the above downloads dl-modelrunner-tensorflow-1-0.4.2-20250103.161616-23.jar instead of the latest release https://maven.scijava.org/service/local/artifact/maven/redirect?r=releases&g=io.bioimage&a=dl-modelrunner-tensorflow-1&v=0.4.1&e=jar

Totally normal to be surprised. I have just changed many many things on JDLL, so I ma keeping everything on SNAPSHOTS at the moment (hopefully for not too long).

[stefanhahmann]

HEllo @stefanhahmann

First of all sorry for having forgotten about this topic for so long. Thanks to your reminder i am back on it.

No worries. I also I did not work on it in the mean time.

What are you using to run this code snippet? I have just tried with the latest snapshot and it seems to work.

I am running this snippet from the IDE (IntelliJ in my case). The snippet misses a package on the first line. But there could be any package.

Totally normal to be surprised. I have just changed many many things on JDLL, so I ma keeping everything on SNAPSHOTS at the moment (hopefully for not too long).

This is also my interpretation. I think the InvocationTargetException may have the reason that something with Reflection goes wrong. E.g. a constructor that may need some parameters but is not provided any parameter when invoked using reflection. But I was not able to trace this down.

[carlosuc3m]

I am running this snippet from the IDE (IntelliJ in my case).

Is the snippet using dl-modelrunner as a dependency? Or did you directly pull it from JDLL github and are running the example in the Stardist3D class?

[stefanhahmann]

I am running this snippet from the IDE (IntelliJ in my case).

Is the snippet using dl-modelrunner as a dependency? Or did you directly pull it from JDLL github and are running the example in the Stardist3D class?

I have checked out JDLL locally and set it to this commit db3c3f7

When I do this, the snippet is working.

When I set JDLL to the current main, it is not working.

[carlosuc3m]

I have just added a commit for debugging. Could you please pull, try again and post the new error?

[stefanhahmann]

I have just added a commit for debugging. Could you please pull, try again and post the new error?

I have just pulled. Now I am getting a different error message:

[SERVICE-0] {"task": "1381c64b-21b1-4dad-880f-b4968d0ee182", "responseType": "UPDATE", "message": "extra module imported: 1.3110008239746094"}
[SERVICE-0] {"task": "1381c64b-21b1-4dad-880f-b4968d0ee182", "responseType": "UPDATE", "message": "Imports"}
[SERVICE-0] {"task": "1381c64b-21b1-4dad-880f-b4968d0ee182", "responseType": "UPDATE", "message": "method"}
[SERVICE-0] <INVALID> warning: empty list of points (returning background-only image)
[SERVICE-0] {"task": "1381c64b-21b1-4dad-880f-b4968d0ee182", "responseType": "UPDATE", "message": "Preparing outputs"}
[SERVICE-0] {"task": "1381c64b-21b1-4dad-880f-b4968d0ee182", "responseType": "COMPLETION", "outputs": {"output0": {"data": "wnsm_a8853886", "shape": [10, 12, 12], "appose_data_type__15012025_132609": "np_arr", "is_fortran": false, "dtype": "float32"}}}
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Error retrieving the image from Python
	at io.bioimage.modelrunner.runmode.RunMode.runOP(RunMode.java:176)
	at io.bioimage.modelrunner.model.Stardist3D.postProcessing(Stardist3D.java:195)
	at io.bioimage.modelrunner.model.Stardist3D.predict(Stardist3D.java:174)
	at net.stefanhahmann.stardist3d.jdll.StarDist3DDummyPrediction.main(StarDist3DDummyPrediction.java:33)
Caused by: java.lang.RuntimeException: Error retrieving the image from Python
	at io.bioimage.modelrunner.runmode.RunMode.createImgLib2ArrFromApposeOutput(RunMode.java:511)
	at io.bioimage.modelrunner.runmode.RunMode.recreateOutputObjects(RunMode.java:209)
	at io.bioimage.modelrunner.runmode.RunMode.runOP(RunMode.java:165)
	... 3 more
Caused by: java.nio.file.FileAlreadyExistsException: Shared memory segment already exists with different dimensions, data type or format. Size of existing shared memory segment: 8192, size of proposed object: 23040
	at io.bioimage.modelrunner.tensor.shm.SharedMemoryArrayWin.<init>(SharedMemoryArrayWin.java:224)
	at io.bioimage.modelrunner.tensor.shm.SharedMemoryArrayWin.readOrCreate(SharedMemoryArrayWin.java:337)
	at io.bioimage.modelrunner.tensor.shm.SharedMemoryArray.readOrCreate(SharedMemoryArray.java:189)
	at io.bioimage.modelrunner.runmode.RunMode.createImgLib2ArrFromApposeOutput(RunMode.java:508)
	... 5 more

[carlosuc3m]

Hello @stefanhahmann
It seems that Python and stardist has been run. The error might probably be because of the size of the volume.
I have tried with the following size and it works
RandomAccessibleInterval<FloatType> img = ArrayImgs.floats(new long[] {116, 120, 66})
Could you please try it?

Just so you know I am completely re-writing the API. As mentioned on the zulip thread I am trying first to create a Java wrapper for the Stardist original API, but if it is not worth it or too complex (too complex meaning that it takes more than today and tomorrow) I will just call everything in Python.
After the changes, you should be able to run a model directly without converting into the Bioimage.io format.

Does it suit you?

[stefanhahmann]

Hello @stefanhahmann It seems that Python and stardist has been run. The error might probably be because of the size of the volume. I have tried with the following size and it works RandomAccessibleInterval<FloatType> img = ArrayImgs.floats(new long[] {116, 120, 66}) Could you please try it?

I tried, but now I am getting again errors due to long classpath:

java.io.IOException: Cannot run program ""C:\Program Files\Java\jdk-21\bin\java"" (in directory "."): CreateProcess error=206, Der Dateiname oder die Erweiterung ist zu lang
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1170)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1089)
	at io.bioimage.modelrunner.apposed.appose.Service.start(Service.java:149)
	at io.bioimage.modelrunner.apposed.appose.Service.task(Service.java:180)
	at io.bioimage.modelrunner.tensorflow.v1.Tensorflow1Interface.launchModelLoadOnProcess(Tensorflow1Interface.java:218)
	at io.bioimage.modelrunner.tensorflow.v1.Tensorflow1Interface.loadModel(Tensorflow1Interface.java:192)
	at io.bioimage.modelrunner.model.Model.loadModel(Model.java:495)
	at io.bioimage.modelrunner.model.Stardist3D.predict(Stardist3D.java:247)
	at net.stefanhahmann.stardist3d.jdll.StarDist3DDummyPrediction.main(StarDist3DDummyPrediction.java:33)
Caused by: java.io.IOException: CreateProcess error=206, Der Dateiname oder die Erweiterung ist zu lang
	at java.base/java.lang.ProcessImpl.create(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:500)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1126)
	... 8 more

	at io.bioimage.modelrunner.tensorflow.v1.Tensorflow1Interface.loadModel(Tensorflow1Interface.java:194)
	at io.bioimage.modelrunner.model.Model.loadModel(Model.java:495)
	at io.bioimage.modelrunner.model.Stardist3D.predict(Stardist3D.java:247)
	at net.stefanhahmann.stardist3d.jdll.StarDist3DDummyPrediction.main(StarDist3DDummyPrediction.java:33)

This migh be a Windows specific problem. I am trying to workaround adding

private static String[] fixArgs( final String[] args )
	{
		try
		{
			String[] newArgs = new String[ args.length - 1 ];
			File classPathFile = Files.createTempFile( "classpath", "" ).toFile();
			classPathFile.deleteOnExit();
			String arguments = "-cp ";
			arguments += "\"";
			arguments += args[ 2 ];
			arguments += "\"";
			Files.write( classPathFile.toPath(), arguments.getBytes( StandardCharsets.UTF_8 ) );
			newArgs[ 0 ] = args[ 0 ].replaceAll( "\"", "" ) + " ";
			newArgs[ 1 ] = "@" + classPathFile.getAbsolutePath();
			newArgs[ 2 ] = args[ 3 ];
			return newArgs;
		}
		catch ( IOException e )
		{
			throw new RuntimeException( e );
		}
	}

to the Service class and the changing the Constructor to

public Service(File cwd, String... args) {
		this.cwd = cwd;
		this.args = fixArgs(args.clone());
		serviceID = serviceCount++;
	}

But, then I get

even though the class is on the classpath..

Just so you know I am completely re-writing the API. As mentioned on the zulip thread I am trying first to create a Java wrapper for the Stardist original API, but if it is not worth it or too complex (too complex meaning that it takes more than today and tomorrow) I will just call everything in Python. After the changes, you should be able to run a model directly without converting into the Bioimage.io format.

Does it suit you?

That sounds good. My feeling at the moment is that running everything in python and getting it into Java using https://github.com/imglib/imglib2-appose seems most promising as this would allow testing everything on the python site and once this is working connecting it to java, but this is just a gut feeling.

[carlosuc3m]

Hello @stefanhahmann I have completely reworked the Stardist api in JDLL.
Please can you try it?
Now it should mimic the Python one, you can proide models that are not in the bioimage.io format

REgards,
Carlos

[stefanhahmann]

Sorry for not testing this again for a while.
I pulled the latest changes and run the code again. Now, it is working.

I have problems though when trying to predict real images. But these problems are not JDLL related. They occur also in pure Python and seem to be related to too weak hardware, where I was testing this.

As far as this issue is concerned, it can be closed.

[carlosuc3m]

which problems do you have? maybe i can help you with something

[stefanhahmann]

I am trying to reproduce the StarDist prediction with the model published here: https://zenodo.org/records/8429205

I have the image data (1136_n_stain_TO-PRO-3.tif) that I downloaded from here: https://www.ebi.ac.uk/biostudies/files/S-BIAD1026/Nuclei_training_segmentation/Training/image/dataset_Tiff/Files.zip
It has the dimensions (1120, 1120, 268). When I run on full resolution, I get

From pure Python I tried to run StarDist with different tiling approaches with and without using GPU, cf.: kreshuklab/go-nuclear#10

The result was that I did not find any tiling combination that worked on my machine (32 GB RAM, but not all of it usable for the prediction). It was found that at least 27GB of free RAM are needed to run StarDist for such image data (cf. kreshuklab/go-nuclear#10 (comment)). So I guess, it is really a Hardware related problem, not so much a software related problem. It is really a pity that the tiling that StarDist actually has does not improve the situation.

Cellpose in contrast seems to be less greedy for Hardware resources.