How do I work with video from a camera?

[tusqasi]

I have a function which will take an Evision image, and return an Evision image.

I want to pass it a camera frame and then take the frame and display that frame.

How do I go about doing this?

I tried with livebook and kino render stuff, that did work but was very slow.

I want to use Wx Evision.Wx.imshow to render images.

Thanks for any help!

[cocoa-xu]

Hi @tusqasi, sorry I only have an Erlang demo for that at the moment, evision_erlang_dnn_demo.erl. I plan to rewrite it in Elixir next week or so.

I think the following code fragments might help you write an Elixir one, and they are basically converting Evision.Mat to wxBitmap and drawing the converted image on the canvas. Please free feel to reply here if you need any help!

https://github.com/cocoa-xu/evision/blob/main/examples/evision_erlang_dnn_demo.erl#L235-L244

                {ok, NewFrame} = evision:resize(ResultFrame, [W, H]),
                {ok, RGBFrame} = evision:cvtColor(NewFrame, evision:cv_COLOR_BGR2RGB()),
                {ok, Binary} = evision_mat:to_binary(RGBFrame),
                Img = wxImage:new(W, H, Binary),
                Bmp = wxBitmap:new(Img),
                Fun = fun(DC) ->
                    wxDC:clear(DC),
                    wxDC:drawBitmap(DC, Bmp, {0, 0})
                end,
                draw(State#state.canvas, State#state.bitmap, Fun),

https://github.com/cocoa-xu/evision/blob/main/examples/evision_erlang_dnn_demo.erl#L284-L293

draw(Canvas, Bitmap, Fun) ->
    MemoryDC = wxMemoryDC:new(Bitmap),
    Fun(MemoryDC),

    CDC = wxWindowDC:new(Canvas),
    wxDC:blit(CDC, {0,0},
	      {wxBitmap:getWidth(Bitmap), wxBitmap:getHeight(Bitmap)},
	      MemoryDC, {0,0}),    
    wxWindowDC:destroy(CDC),
    wxMemoryDC:destroy(MemoryDC).

[tusqasi]

Elixir rewrite is going to be a huge help!

I want to use Wx to render images.

To clarify, I meant the Evision.Wx.imshow function. Because the example seems to be using wx directly. I didn't mean the wx GUI thing directly.

PS. Do you get very low frame rates when using camera capture from livebook? Is that expected?
I get very bad performance when using the camera in livebook.

[cocoa-xu]

tl;dr

Yes, because rendering in livebook with kino is basically encoding each frame as a jpeg/png (depends on your settings) image and pass it to the browser, where it gets decoded and displayed. Therefore, it is expected to be slow at the moment.

As for Evision.Wx.imshow, perhaps you can do something like

def show_video(cap) do
  case Evision.VideoCapture.read(cap) do
    %Evision.Mat{}=frame ->
      # as along as you're using the same window title, 
      # the frame will be plotted in the same window
      Evision.Wx.imshow("window title", frame)
      show_video(cap)
    _ ->
      # video has ended or an error occurred
      :no_more_frames
  end
end

More information

And to achieve a smooth experience in livebook, it would effectively require at least a stream encoder (which encodes the video stream to HLS or other live format) on the Elixir side and serving it on another port. I'm not planing to implement this any time soon (either in Evision or as another standalone library).

However, if you're okay with gstreamer, then one possible way is to build it from source (OpenCV will detect and compile gstreamer related code if gstreamer library and headers are available on your machine). Then you can set up a gstreamer pipeline, and receive the video stream in another browser tab or other video player that supports it.

For example,

gst_str_rtp = "v4l2src device=/dev/video0 ! video/x-raw, format=YUY2, width=640, height=480, pixel-aspect-ratio=1/1, framerate=30/1 ! videoconvert ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4 " \
             " ! h264parse ! mpegtsmux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5000"

[tusqasi]

Wowww!!!! show_video works like a charm.

I don't know what was I thinking when spawning processes just to render images 😅