Can you run two neural networks concurrently?

[elephantpanda]

Say you have two small neural networks, A and B.

I assume that the GPU calculates things one layer at a time.

If each of the layers were small enough, could it calculate a layer from A while simultaneously calculating a layer from B?

For example a GTX 3080 has 8704 cores. So if A was small enough it wouldn't use all the cores in a calculation.

Mind you for a 64x64x4 =16384 image, this is bigger than the number of cores. (But if A and B all had layers of size 32x32x4 could they be run together?)

So what I'm getting at it is if I two independent networks, is there any advantage in running the Inference session on different threads or should they just be run sequentially?

[RyanUnderhill]

Possibly. If your GPU isn't at 100% usage running the one model, then running a second at the same time in parallel should use the available power and you will come out ahead.