[Question]: OBS: Rate control method to encode static scenes efficiently and allow specified peak bitrate

[pbaykalov]

I would like rate control method to encode static scenes efficiently and allow specified peak bitrate.
I get something similar to what I want with HQVBR in both OBS and FFMPEG. But only if I use "recording" tab in OBS.

If I set exactly same parameters in "streaming" tab then then OBS always uses full HQVBR bandwidth to encode static image.

This behaviour is 100% repeatable for me. Settings look exactly the same in both tabs. Basically, same settings result in different codec behaviour.

Should I create an issue in OBS repository too?

OBS 32.0.4, 7900XTX, W11 24H2

[rhutsAMD]

Yes, using the same settings between the recording and streaming tabs in OBS but seeing different codec behaviour is more of a discussion for the OBS forum but I will provide some information that may help you understand why this is happening.

The AMF encoder plugin is agnostic to whether it is being used from the streaming or recording OBS tab. There are some suggested streaming service settings that OBS sets, which may be disabled by checking the “Ignore streaming service setting recommendations” checkbox in Settings > Stream. There are also some settings that OBS enforces just for streaming which may not be disabled. For example, I believe the CBR rate control method is still enforced in OBS for streaming, regardless of which rate control method is selected in settings.
https://github.com/obsproject/obs-studio/blob/master/plugins/rtmp-services/rtmp-common.c#L729

[pbaykalov]

@rhutsAMD thank you, it makes sense.
After more thorough checking I found out that HQVBR is basically behaving like CQP for me very consistently - producing small bitrate for static scenes and upwards of 200 Mb/s for moving scenes.
Started a clean portable installation of OBS to double check.
This is what log says:

10:15:50.320: [texture-amf-h264: 'advanced_video_recording'] settings:
10:15:50.320: 	rate_control: HQVBR
10:15:50.320: 	bitrate:      6000
10:15:50.320: 	cqp:          20
10:15:50.320: 	keyint:       250
10:15:50.320: 	preset:       quality
10:15:50.320: 	profile:      high
10:15:50.320: 	level:        4.2
10:15:50.320: 	b-frames:     0
10:15:50.320: 	width:        1920
10:15:50.320: 	height:       1080
10:15:50.320: 	pre-analysis: false
10:15:50.320: 	params:       (none)

I never set CQP anywhere.
And these are literally all the settings I made:

And also adding MaxQP=50 MinQP=50 changes codec behaviour (output becomes very small) when it should not in HQVBR.
Sounds like OBS setting some default QP somewhere which is not ignored by AMF as it should.

I can override RateControlMethod to any value and it clearly works - I definitely get CBR and VBR with =1 and =2 respectively. But setting RateControlMethod=5 (which must be HQVBR according to an enum) does not change behaviour at all, I still get what looks to be CQP.
as per

AMF/amf/public/include/components/VideoEncoderVCE.h

Line 106 in d0b3e6d

AMF_VIDEO_ENCODER_RATE_CONTROL_METHOD_HIGH_QUALITY_VBR,

Is HQVBR working for you in OBS?

---

And naturally a simple FFMPEG command seems to do exactly what documents tell:

ffmpeg -i 'D:\docs\OBS\static pic.mkv' -c:v h264_amf  -rc hqvbr -g 120 -b 10000000 output3.mp4

both encoding static image efficiently and not exceeding 10 Mb/s.
And similarly, if I also specify qmax it "breaks" HQVBR which is probably useful to be able to limit QP. Additionally, it would be nice if it is documented in
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/wiki/Rate%20Control%20Methods
because it sounds important.

But then OBS should account for that and call "ClearProperty" if CQP is not used and it is explicitly set here:
https://github.com/obsproject/obs-studio/blob/f4aaa5f0417c5ec40a3799551e125129fce1e007/plugins/obs-ffmpeg/texture-amf.cpp#L1167

[pbaykalov]

@rhutsAMD I think this is a comprehensive description: obsproject/obs-studio#12994

[TV4ELP]

The log is a bit misleading here. Yes CQP is set to 20, but it's never fed to the encoder.

This is just a sane default for when someone sets nothing.

Default settings:
https://github.com/obsproject/obs-studio/blob/902f6d17e5a65742d16baf3cd30032979866dae7/plugins/obs-ffmpeg/texture-amf.cpp#L1163

Those overwrite the current config, see the cqp 20.

Here however you can see that cqp is only set when CQP rate control is actually used:

Additionally, Bitrate is multiplied by 1.5 when using VBR methods so keep that in mind. It's probably okay tho.
https://github.com/obsproject/obs-studio/blob/902f6d17e5a65742d16baf3cd30032979866dae7/plugins/obs-ffmpeg/texture-amf.cpp#L1322

Here is just the info you see in the log. If you scroll up you see CQP being read from the config, but never actually being set (see the snipped above)
https://github.com/obsproject/obs-studio/blob/902f6d17e5a65742d16baf3cd30032979866dae7/plugins/obs-ffmpeg/texture-amf.cpp#L1570

Additionally, Filler Data seems to be only added when using CBR. Aka the rate control is not forced to up the bitrate and can undershoot it.
https://github.com/obsproject/obs-studio/blob/902f6d17e5a65742d16baf3cd30032979866dae7/plugins/obs-ffmpeg/texture-amf.cpp#L1322

[pbaykalov]

@TV4ELP thanks for the input, it makes sense. I would love if you told me if you see HQVBR working for you as expected.

I observe the obvious CQP behaviour and tried looking if it is related to those defaults (and it clear now that it is not ).

Additionally, Filler Data seems to be only added when using CBR.

I see the property being set but that does not seem to guarantee constant bitrate regardless of other settings in my setup (unrelated to my issue).

However, while it should work like you described at first glance this also cannot work reliably when settings are changed as I described in OBS issue: the encoder is affected by values unrelated to HQVBR and default QP values are not restored when rate control method is changed.

[pbaykalov]

I actually found it! OBS sets RateControlPreanalysisEnable=True for all rate control modes and it definitely breaks HQVBR.
Setting RateControlPreanalysisEnable=False restores the correct behavior.
This is either a bug in AMF or an incomplete documentation.

But now I cannot understand why OBS encoding is severely lagging for static images, like 90% frames dropped.
HQVBR
6 Mbps
RateControlPreanalysisEnable=False custom option, nothing else.
Even an empty scene lags severely but frame drops stop a few seconds after there is fast movement in the image.

[pbaykalov]

I have no idea what's the underlying problem but specifying PAEngineType=11 (DX12) completely eliminates it.