Multithreading in the Java API

[andreabrduque]

I want to optimize some inferences I am doing using the Java API using multithreading, and there are several options available:
As I understand, they are:

1- Using a global shared threadpool when constructing OrtEnviroment
2- Controlling the session thread pool
3- Calling the same session concurrently using a Java controlled threadpool, since it seems that the session is thread safe.

What are the benefits of constructing the environment with a single shared thread pool, apart from reducing the memory footprint, in comparison to controlling the session threadpool? Moreover, are there benefits of calling the session through several Java threads, since the ONNX should already have its own thread pool?

Thank you for any pointers!

[Craigacp]

I believe 1 and 2 are equivalent when there is only a single session. 1 is advantageous when there are multiple sessions in the same process and you want them to share resources.

3 is separate as it allows you to have multiple inference calls in flight, which are then assigned to the threads using ORT's mechanisms. Also 3 will let you schedule multiple calls to your accelerators, whereas the threadpools only control CPU threading (as far as I'm aware).