Performance optimization

[allo-]

Even with tensorflow-gpu, the program uses the CPU (one core) quite much.

Find out what are the slow parts and how good they can be optimized:

• Probably quite a few numpy operations could be in-place, but in general I would not expect the numpy operations to be slow for arrays with shape (1280, 720, 4).
• Passing around the frame should be a reference and not too slow by itself.
• How good is the performance of the cv2 operations?
• Is there a bottleneck between tensorflow and cv2/numpy?
• Is there data that should be global but is stored in each iteration of the main loop?

[schwab]

Some performance stats....
Running this on a 32 core thread ripper, and it's been up for about 10 hours with a web stream and a simple background image. Currently the process is broken up across ~20 subprocesses. When the process is running it takes the overall system load average from a baseline of around 1.79 to just over 3.0. This is with about 400MB of RAM. I'd estimate then that overall we are using about 1.2 to 1.3 cores at 100%. With that load the delay is just under 1 second.

If we can improve the delay time and bring it down closer to .5 second, even at the cost of more cpu usage I think it would be a good tradeoff especially for those that have cpu bandwidth to spare. Perhaps there could even be a cpu utilization/delay time tradeoff config value that allows allocating more resources to improve the video delay.

[allo-]

I get about 0.1s per mainloop iteration on a Ryzen 8-Core CPU with 100% load on one core.

Evaluating the network seems to be 0.01-0.02s, getting the images and scheduling them seems to be fast. Probably body-pix + filters just sum up. Maybe the segmentation for the next image could run in another thread before the current one is scheduled?

In principle you could process one image per core by just grabbing the next frame when the first core is idle again

For limiting the CPU usage, we could limit the framerate, but it should probably be at least 10 for a good stream.

[schwab]

How are you measuring the loop iteration? In my case I just hold up a stopwatch in front of the camera and observe how far apart the video is from the real one.

[allo-]

timestamp = time.time()
# code to benchmark
print(time.time() - timestamp)

This may not be the best profiler, but its enough to get a rough impression what may be the slow parts.

The latency between webcam and fake webcam is another issue, but for finding the slow parts the most important part is what happens between cap.read and fakewebcam.schedule_frame.

[schwab]

Right, so correctly stated, my latency is about a second. Just reviewed that mainloop code between cap.read and schedule_frame. I see there is a lot going on per frame there.

[allo-]

Some thoughts about performance

• I think copying a numpy array is probably fast. Numpy/Scipy do much in C and are in general optimized for performace I would think that the copy operation is implemented using something like memcpy in the backend.
• .astype is probably slow.
• Adding dimensions may be slow.
• OpenCV operations seem to be much slower than numpy
• Tensorflow operations are fast, but converting between tensorflow tensors and numpy arrays is probably slow
• Operations like cv2.dilate(mask, np.ones((dilate_value, dilate_value), np.uint8)) instantiate a new numpy array, convert an existing one to uint8 and require converting back to numpy afterwards.
• I am not sure if frame grabbing and frame scheduling may be slow. In basic tests it seemed okay, but maybe it is possible to already start processing the next frame before the current one is scheduled. I am not sure about when it should be grabbed, as this changes which frames are dropped in between.

[Nerdyvedi]

Using another thread for grabbing frames slightly increases the speed.

[allo-]

@Nerdyvedi Did you test it in some way?

I think it will have quite a bit of issues. When grabbing the next frame at the beginning of the loop you are dropping the in between frames automatically.

When grabbing in an own thread you need a buffer. When the filter thread is ready to process the next frame, the one in the buffer is already stale, so you need to constantly fill a stack and drop the older frames yourself when processing the next one from the top of the stack.

The benefit of avoiding the delay to grab a frame is probably not worth the issues with buffering and synchronization, if you don't have numbers that it will be a lot faster.

[allo-]

Maybe no stack is needed, but just double buffering and correct locking. I am still not convinced if this is the most important part that should be optimized at cost of complexity.

[Nerdyvedi]

@allo-
I employed the following tricks to increase the speed, It definitely results in better performance

1. Using a different thread to grab frame
2. Replacing .astype('uint8') with .view('uint8') [:,:]
3. Replacing the loop on output tensor with the fixed index
   for idx,name in enumerate(output_tensor_names): if name == "float_segments:0": segment_logits = results[idx]
   with segment_logits = results[6]
   6 is the idx for float_segments tensor

[allo-]

1. How do you handle that you can grab several frames before the first is sent to the fake cam? How do you select the next one?

2. Sounds good. The most important point that is not that clear in the code is when a frame should be copied or not. A view can speed up processing frames that are not changed,

3. I think this is needed because mobilenet and resnet are different. But this shouldn't be a bottleneck anyway.

[Nerdyvedi]

1. Using a separate thread allows frames to be read continuously from the I/O thread, all while our root thread processes the current frame. Once the root thread has finished processing its frame, it grabs the current frame from the I/O thread. But now that I think of it , I am not handling buffer size this way.
2. I used it only once, In the following line.
   mask = (mask * 255).astype(np.uint8)
3. It's faster than looping on the output tensors, for resnet it was 6th index, Similarly we can find it for Mobilenet. But yeah, minimal increase in performance.

[allo-]

1. I guess one would only need two frames. The last one and the one that is currently grabbed and some intelligent locking.
2. Looks good there. I thought about the filter loop, which is more complex. Some plugins convert back and forth and it matters if the plugin returns a copy or not. It is not that clear at which points you are allowed to modify the input. In the end this should be the same optimization, onöly copying the frame when needed, but on the other hand preventing plugins from messing with the input of another one.
3. It's a list of names after all. On the other hand, you cannot just plug in a new model anyway (e.g. without new preprocessing), so we can hardcode the indices as well.