Improve qvh by adding a more stable video integration either using mkv + tcp or ffmpeg integration

[danielpaulus]

QVH is a rock solid tool, except for the flaky gstreamer integration.
I can see two options to improve this:

1. Integrate with ffmpeg
2. Implement the mkv container format and export a stream via tcp so any media framework can be integrated easily

Some pointers:
We get a series of CMSampleBuf objects from the device (https://github.com/danielpaulus/quicktime_video_hack/blob/main/screencapture/coremedia/cmsamplebuf.go) which I decoded to a Golang struct. Those contain either raw h264 NaLus or PCM audio. Sometimes with a format descriptor that contains PPS Nalus needed to start a stream.

To understand how to work with this, look at the current implementations of CmSampleBufConsumer (

quicktime_video_hack/screencapture/interfaces.go

Line 6 in d81396e

type CmSampleBufConsumer interface {

)

• quicktime_video_hack/screencapture/coremedia/avfilewriter.go

  Line 31 in d81396e

  func (avfw AVFileWriter) Consume(buf CMSampleBuffer) error {

• quicktime_video_hack/screencapture/gstadapter/gst_adapter.go

  Line 222 in d81396e

  func (gsta *GstAdapter) Consume(buf coremedia.CMSampleBuffer) error {

To solve this, we either need a MKVConsumer, that receives the above CMSampleBufs and turns them into an MKV stream or a FFMPEGConsumer that can use FFMPEG directly somehow to turn CMSampleBufs into a useful format.

If we have an MKVConsumer, we/me can hook it up easily to a local tCP server like f.ex. here: screencapture/gstadapter/tcp_server_adapter.go and stream it really fast to ffmpeg or gstreamer running in external processes which make everything much more stable.

[nnnpa31]

Have you seen gomedia?
It can easily wrap NALU and PCM data into fmp4.
It is possible to stream fmp4 using an external player, only a web server needs to be provided (or a socket connection would work, I'm not sure).

I can help with this work if I can :)
I have previous experience working on wrapping an h264 raw stream to fmp4 and mp4, it works fine and can be played using ffplay.

[danielpaulus]

I have not tried it no, do you have some example code you can share? @nnnpa31 If I know how to use nalus and pcm samples, I can probably do the rest pretty easily

[nnnpa31]

Here's an example of using gomedia to mux h264 streams to mp4 files: https://github.com/yapingcat/gomedia/blob/main/example/example_mux_mp4_memory_io.go
I'm currently working with it, just replacing the mp4 in it with fmp4 and overriding the Write function of the memory io.

[danielpaulus]

@nnnpa31 I checked the code and it does not support raw PCM 16 bit / wave audio.

[nnnpa31]

It looks like it was my amateurism that led to the misunderstanding of the content. It seems to me that everything is back to square one and I have to reconsider the option of enabling ffmpeg instead of a pure go implementation.