Restream problem after h264_v4l2m2m encoding

[warez4me]

While configuring Go2rtc for transcode RTSP streams on RPi4 with ffmpeg, I encountered a problem using the h264_v4l2m2m encoder.

If the libx264 encoder is used then everything functions normally and predictably. I can get as many copies of the transcoded video stream as I want adding different audio substreams.

But if to use the h264_v4l2m2m encoder that supports some hardware acceleration, then the features of the output video stream that I don't understand begin to appear, which do not allow parallel use of its copy or even use it with added transcoded audio track(s).

Actually, the RTSP output is produced but its video substream is unsuitable for playback.

For example, when using VLC for playback, the codec info is

• on the left - Ok, starter, link - rtsp://[IP]:8554/stream
• on the middle - bad video, starter, link - rtsp://[IP]:8554/stream?video=all&audio=all
• on the right - bad video, follower, link - rtsp://[IP]:8554/stream

ffprobe for "good" case when consumer is a 1st who opens the stream

$ ffprobe -v error -hide_banner -of default=noprint_wrappers=0 -print_format flat  -select_streams v:0 -show_entries frame=pict_type rtsp://raspberrypi:8554/8m_2
[rtsp @ 0x5555a84fd370] method SETUP failed: 461 Unsupported transport
frames.frame.0.pict_type="I"
frames.frame.1.pict_type="P"
frames.frame.2.pict_type="P"
...
frames.frame.2.pict_type="P"
frames.frame.12.pict_type="I"
frames.frame.13.pict_type="P"
...
^C

for "bad" case when the stream is already opened by another consumer

$ ffprobe -v error -hide_banner -of default=noprint_wrappers=0 -print_format flat  -select_streams v:0 -show_entries frame=pict_type rtsp://raspberrypi:8554/8m_2
[rtsp @ 0x5555a6b1d370] method SETUP failed: 461 Unsupported transport
[h264 @ 0x5555a6b23770] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0x5555a6b23770] decode_slice_header error
[h264 @ 0x5555a6b23770] no frame!
[h264 @ 0x5555a6b23770] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0x5555a6b23770] decode_slice_header error
[h264 @ 0x5555a6b23770] no frame!
...
^C

I've tried to compare the libx264 and h264_v4l2m2m generated streams both with capturing live RTSP stream by tcpdump or by redirecting ffmpeg output to file and then using stream analyzer to examine the differences.

Unfortunately my skill isn't good enough to see the source of problem.
Moreover, the "m2m" h264 stream looks more "canonical" to me.
But the stream from libx264, even without the visible presence of SPS/PPS headers, can be remuxed again, and the stream from h264_v4l2m2m turns out to be unusable with any attempt to use it for remux or copy.

I think this problem is a kinda synthetic and related to both Go2rtc and ffmpeg itself. It's not HW-related because I can use multiple transcoded streams if they were configured the same way, but under different names.

• The Go2rtc is under suspicion because the 1st transcoded instance is good.
• The ffmpeg is under suspicion because the libx264 codec's output stream doesn't have problems

If the Go2rtc is a rock-solid and its way to manipulate streams with ffmpeg is the only possible one - Ok, I'll agree if it can be clearly explained.

I will list the ffmpeg related questions below hoping on the experience and knowledge of the go2rtc community.

• is there a fundamental possibility of obtaining the output h264 video stream after h264_v4l2m2m with parameters similar to those obtained after libx264?
• perhaps the problem is related to the way the memory or CPU is used during the encoding process?
• is there documentation for h264_v4l2m2m encoder with information on how to control it through ffmpeg parameters?

P.S. if required I can share any captured or redirected data.

[AlexxIT]

• I still don't understand what "bad video" means in your description.
• non-existing PPS 0 referenced is OK error for both - direct RTSP restreaming and transcoding. If everything is OK with the stream, the video should appear in a few seconds.
• You will never get the same transcoding parameters with software and hardware methods. This applies to Intel, Nvidia, AMD and other video cards. Hardware algorithms are highly optimized and differ from software ones.

[warez4me]

• related to VLC "bad video" means black screen - video stream is present in properties and can be captured by tcpdump but it's not visible. Also the properties don't have info about resolution, f_rate and so on. BTW the "good" stream props also don't have color info kinda "ITU-R BT.709" or "Planar 4:2:0..." despite the used ffmpeg option "-pix_fmt yuv420p". But such info exists in camera's rtsp.
• if you mean that it's need to wait for I-frame then it's not so. AFAIK this mean that consumer can't get SPS/PPS info from I-frame (or "key-frame" maybe). Just look - 1st who opens the stream - can obtain global header with this info. Others who trying to connect to already opened stream - can't extract it from there.
  And another effect - it's not possible to get on output the "good" stream with any added or replaced audio track. IMO this tracks are "trying to connect" to once transcoded stream - similar to previous case.
• Under the "same parameters" I mean logical structure - to set all this abbreviations "SPS, PPS, SDP, POC... what else" to the same places. I think it is the source of problem - encoders output the streams with different logical structure. It has no deal with HW algorithms. Again, afaik the "m2m" - it's not so "hard" ware encoder.

sorry for my "bad French" (English also).. Too many manuals and RFC-s for still short time.

[AlexxIT]

Show a stream probe (from go2rtc web) for a good stream. And also a stream probe for a bad stream. And show yaml config for both streams.

[warez4me]

Thanks!

Sorry, but it's not so clear for me how to"Show a stream probe (from go2rtc web)".
Do you mean ffprobe? In what file format? What link to use from "streams"? RTSP or HTTP ?
What cmdline will be the best?
Or You mean tcpdump capturing?

I can propose you try webtorrent links.
I've tried external access via mobile GSM 4G - one consumer was the phone, and another - PC.
Effects was the same - in "bad" case the "Net" links were the same but stream was visible only for 1st consumer.

What do you prefer?

[AlexxIT]

On the main go2rtc web page you can see "probe" word. I don't need access. A probe and config are enough for now.

[warez4me]

"Semyon Semyonych!"

Here it is.

alexxit.zip

Naming:

• m2m is for 8m_w_264
• libx is for 8m_x_264
• 1st is for the case when "probe" taken without any concurrent consumers
• 2nd is for the case when "probe" taken after the VLC was already successfully opened & shows this stream in separate window
• try# just for sure

BTW I mostly using the Opera which doesn't have the menu for "probe" Json save. Only "pretty print". That's why I never know about such posibilities.
These savings are from Firefox (which in turn doesn't have "pretty print").

[warez4me]

What info You exactly need ?
Each tab shows in other way...
And Json "Save" creates only some short form...

---

Tried to compare savings - didn't see mean diffs...

[warez4me]

Here the raw results from Opera. Firefox doesn't allow to get in such way (non pretty-print). Just for case. Can be easily removed.

alexxit_2.zip

"m2m" doesn't allow "-profile high" prm.

Result will be as follows:

[rtsp] error=streams: exec/rtsp
[hevc @ 0x556c6aba60] Could not find ref with POC 21
[hevc @ 0x556c6aba60] frame_post_process: Decode fail
[h264_v4l2m2m @ 0x556c6dbbb0] [Eval @ 0x7fdba076a8] Undefined constant or missing '(' in 'high'
[h264_v4l2m2m @ 0x556c6dbbb0] Unable to parse option value "high"
[h264_v4l2m2m @ 0x556c6dbbb0] Error setting option profile to value high.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
, exec/rtsp
rtsp://127.0.0.1:8554/8m_w_264?audio&source=ffmpeg:8m_w_264%23audio%3Dopus: Invalid data found when processing input
stream=8m_w_264

P.S. noticed that VLC is visible in probe as 1st consumer

[warez4me]

And another thing - in all cases probe result appears with some delay. Except for m2m_2nd - this appears almost instantly, without any delay.

[AlexxIT]

I see you are using non-standard ffmpeg and non-standard ffmpeg parameters for transcoding.
Repeat your tests on standard ffmpeg and on built-in transcoding parameters for m2m.
The situation when VLC shows a black screen.

[warez4me]

Well, here are probes using go2rtc container.
I decided not to destroy my environment.
Previous camera outputs some non-yuv420 stream which isn't acceptable by h264_v4l2m2m if to follow your demand for go2rtc's "standard" presets.
This probes are for another camera, that has another "feature" - it doesn't give a sound track.
My config sets "#audio=copy" anyway because of some different, "more tolerate for restreaming" behavior. Don't remember how I got such experience but even this probes show some differences: go2rtc's log for "-c:a copy" at the end "gracefully" informs about disconnecting 2nd concurrent stream (1st logged stream is VLC). And in case when config without "#audio=copy" was used - the log ignores appearance of 2nd concurrent stream. Maybe this fact also mean something.
And another thing - the output quality with go2rtc's standard "#hardware=h264_v4l2m2m" preset is very poor.
At least it lacks of "-b" option to set adequate bitrate. My custom config isn't unreasonable.

v380.zip

[warez4me]

Sorry.
Checked again - info in the log about 2nd consumer has no deal with probe. Seems it's for another VLC - timestamps are later for ~20 min.
Well, the both logs ignore probes.

[AlexxIT]

I didn't understand much from your message.
You have your own transcoding configuration that is causing a black screen in VLC.
How do the built-in ffmpeg and go2rtc's built-in transcoding parameters work in this situation?

[warez4me]

@AlexxIT
Seems my English is very bad. Or comments are too long.
Your last request was

I see you are using non-standard ffmpeg and non-standard ffmpeg parameters for transcoding. Repeat your tests on standard ffmpeg and on built-in transcoding parameters for m2m. The situation when VLC shows a black screen.

I did exactly as You requested

• stopped go2rtc binary service and started go2rtc container.
• edited go2rtc.yaml to set standard transcoding configuration
• repeated probes 2 times : with and without copying audiostream because of the "silent" camera.
• copied probe's logs as .txt files
• copied go2rtc's logs as html pages
• made VLC screenshots of codec info for both "good" and "bad" case

I don't see any reason to state that this issue is related only to my "own transcoding configuration that is causing a black screen in VLC"
Last probes clearly show that this issue doesn't depend of ffmpeg version and cmd options but only depend of used encoder.
And the "black screen" it's only the visual tool to see the problem.

v380.zip