[Bug]: AV1 encodes which require using alignment mode `1080p` or `unrestricted` don't signal the correct resolution

[RFCArrow]

Describe the bug
It seems that when using the AMF_VIDEO_ENCODER_AV1_ALIGNMENT_MODE set to AMF_VIDEO_ENCODER_AV1_ALIGNMENT_MODE_NO_RESTRICTIONS, the video is extended by adding rows/columns of black pixels to fit into a whole number of transforms. However, the height and width signalled in the Sequence Header OBU is set to the extended resolution rather than the original height/width. This means that when a video is encoded using a non-64x16 alignment and then decoded, it will feature a black strip at the edge of the video at some resolutions (for example 1920x1080, 960x540, 320x180 etc). Instead the Sequence Header OBU should signal the original resolution so that the frame is cropped appropriately to avoid the black edges being displayed. It is also important that the original resolution is signalled because some applications require that the original input resolution is maintained in the output.

To Reproduce
Steps to reproduce the behavior:

• Build the AMF TranscodeHW sample.
• Run with -WIDTH 320 -HEIGHT 180 -INPUT C:\Users\rob.arrow\assets\test_input\other\input_short_320x180_8bit_420p.yuv -OUTPUT output.mp4 -CODEC AV1 -TRACE_LEVEL 4 -Av1AlignmentMode NORESTRICTIONS
• Play/inspect the output
• Note the black strip at the bottom
• Note the resolution is signalled as 320x192.

Setup (please complete the following information):

• OS: Windows 11 Pro (22H2 / 22621.2283)
• Driver Version: 21.9.3
• GPU: RX 7900 XTX
• AMF: 1.4.30 (2f32635)
• Which component has the issue: Encoder

Debug Log (please upload or paste):
Attached here

Expected behavior
The output video has resolution 320x180 (the same as the input video). No black edge extension is visible while playing the output video.

Additional context
Note a similar effect occurs when using the 64x16A1080P alignment mode one 1080p video. In that case, the video is extended to 1920x1082 and a similar 2-pixel tall row of black pixels is added to the bottom of the video.

[namoen0301]

ref: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9185

an AMD staff said:

It is embarrassing and sorry to report that this AV1 issue is a confirmed HW problem in this VCN generation (VCN4), the problem will be fixed in next VCN generation, and for VCN4, it is required to have the alignment of 64x16 as mentioned in this ticket description. And for the common 1080 height case, to minimize the impact, we adopted an idea to use a HW trick which happens to use the closest height 1082 instead of 1080, which is the reason you see 1082 for the 1080p cases.

It looks like it's a hardware issue, There's very little AMF can do.

[MikhailAMD]

Yes, this Wiki describes the behavior and mitigation: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/wiki/AV1%20Encoder#av1-specific-api
Note: FFmpeg adds container level crop support which would help. there are several patches starting with this: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231007162503.1057-2-jamrial@gmail.com/

[namoen0301]

I noticed that there is some new code in Mesa repository about rdna 3.5 (gfx11.5) in the Linux kernel, will this issue be fixed in rdna 3.5?

[MikhailAMD]

Cannot comment on unreleased HW or SW, sorry.

[DimkaTsv]

Hmm... For me Adrenaline software (7800XT) records not 1920*1082 for AV1, but 1920x1088 instead. And 1082 cannot even be aligned to x16 as 1082/16=67.625, while 1088/16=68.
Is documentation correct on specified 1082 resolution?

I also just tested ways to not fix, but at least make result more... compatible with sources like players or YouTube for example in terms of their interpretation of source.
By default they take 1920x1088 and interpret it as 1906x1080, which is... Well, not what expected, as they crop different dimension from what was filled.
I also tried to use container level crop - 0,0,0,8 which didn't help them to interpret result as 1920x1080.
But what DID help, is using MkvToolNix and set to container forced property of "Set aspect ratio" to 16/9, which did help and both MPC-HC and YouTube started to scale result as 1920x1080 (instead of 1906*1080 that they used before). As this also doesn't require video reencoding process is quite fast.

So... Maybe it is possible to add metadata compatability trick to compensate for HW issue?

Here is example:
Source - AV1 1920x1088 - https://drive.google.com/file/d/1-cphqrCzDaP0OEjBVwxZo4Kh2OWaI6dk/view?usp=sharing
Packed into container with forced 16/9 aspect ratio - https://drive.google.com/file/d/1YeiUH0Y1DhQu_KZ5kNRCKUvI0Dmy6JaH/view?usp=sharing
Uploaded on YouTube source - https://youtu.be/55YOSX6pecM
Uploaded on YouTube packed with forced 16/9 aspect ratio - https://youtu.be/IV2ctkqJC94

Sadly, using container level crop doesn't work, but for some reason forced aspect ratio does?
Attempt to use container level crop [0,0,0,8]- https://drive.google.com/file/d/1dpBoSxVDkCCnQrKN6SZbaw9-LaP3F52h/view?usp=sharing
Uploaded to YouTube container level crop - https://youtu.be/dreIa_QlsxE

Small UPD: Container level crop seems to be working, but you MUST specify resolution too, for some reason.

Container crop + forced 1920x1080 - https://drive.google.com/file/d/1YX4I4sHLQO_t_msfhOU7baM-ow59iKtA/view?usp=sharing
Uploaded to YouTube container crop + forced 1920x1080 - https://youtu.be/rrqWDusBA0w

Also, based on YouTube sample and amount of pixels left with force 16:9 and crop and force 1920x1080, forcing aspect ration if overall better, unless i cropped wrong row. Simply because under row of quick slots gap between numbers and end of frame is noticeably smaller in crop+force resolution sample compared to force aspect ratio.

Force aspect ratio

Crop and force resolution

[MikhailAMD]

• Did you use AMD ReLive app? It uses AMF_VIDEO_ENCODER_AV1_ALIGNMENT_MODE = MODE_NO_RESTRICTIONS, and asks for 1088.
• From AMF API perspective, if 1080 is requested, two modes (64X16_1080P_CODED_1082 or MODE_NO_RESTRICTIONS) would produce 1082 and the third (64X16_ONLY) will fail. 1082 is not aligned to 16 but is the closest possible to the requested.
• You can experiment with AMF DVR sample.
• Regarding metadata: AV1 spec doesn't have cropping metadata like other codecs (H264 and HEVC have VUI header). Custom metadata (a custom NAL unit) will be not recognized by any player/decoder. This is why the only option is to use cropping in containers.
• We are working with FFmpeg as a starting point, but it moves slowly. First, general FFmpeg changes for cropping need to be merged, and then we can put AMF changes on top. Once the changes are in FFmpeg, we will work with other ISVs to set the crop properly. The result should be the same is you achieved with MkvToolNix

[DimkaTsv]

1. Yes, it was recording made by AMD Adrenaline built-in Screen Capture. So i assume it was ReLive recoding.

2. That what i expected to see - 1920×1082, bevause which should be basically undistinguishable for end user, but was surprised to get 1088 instead from ReLive.

[AMF_VIDEO_ENCODER_AV1_ALIGNMENT_MODE_NO_RESTRICTIONS, videos with any resolution can be coded. However, for those videos whose resolution is not 64x16 aligned, their output resolution will be extrapolated to be 64x16 aligned and padded with black pixels. The exception is for resolution of 1080p, which will be padded to 1082p, as in the case of MODE 2.]

3. Hmm... Can you help me guide to said AMF DVR? But i think to use it daily can be more bothersome than practical from accessibility of hotkeys to start/stop recording quickly.

4. I understand that AV1 doesn't support native metadata, but as video is recorded in MP4 container, it should be possible to do on container level? At least until it will be worked out with ISV's? It is not hard neither long to do it with MKVToolNix (literally copying file speed), but having innate container level aspect ratio metadata without needing to take additional steps already makes compatibility with different players much higher based on how MPC-HC AND YouTube interpreted such videos despite being simple change not affecting video or codec data. At least within ReLive, because screen capture is one of the most used way to get such AV1 videos, and also because AMF as whole is more flexible compared to screen capture. But... I know that unlike .mp4 .mkv is universal container that can hold anything, and not sure if .mp4 supports enough flexibility.

P.S. Ah yeah, also a question. Where are these 2 or 8 row of pixels are being added relative to video? Should be at bottom but by cutting bottom 8 rows i lost part of real video it seems, and not filled data. Unless container cut didn't apply at all, and fixed resolution just worked this way unlike aspect ratio one?

Congratulations with upcoming New Year!

[MikhailAMD]

#2 in NO_RESTRICTIONS mode ask for 1080 will generate 1082, ask for 1088 will generate 1088. ReLive implemented the latter.
#3 DVR is not the app to use in day-to-day live. It is a sample to show details of desktop recording to use AMF capture, AMD encoder, timing etc.
#4 Yes, it is possible to do at container level. But AMF doesn't implement container code. It relies on FFmpeg as most of apps. Also writing crop information is only half of the work. There is need that decoding application read and used this crop info. And some containers have crop in spec (MKV, MP4), some don't (FLV). In addition, crop from MP4 rarely used.
I agree, ReLive should write cropping info but currently it relies on version of FFmpeg without it. We will update once we push cropping to FFmpeg.

The black rows from 2 to 8 are added at the bottom of the frame. Ideally, you don't need to play with aspect ratio. If reading app respects container cropping, it should just cut these lines.

[DimkaTsv]

2. Hm... Is there any particular reason why ReLive asks for 1088p instead of 1080(1082)? I tried OBS yesterday and it did produce 1082 video, so can confirm that this part works, just not for ReLive.

3. And that is strange part. I did try to write container level crop before trying other options, and i agree, it was completely ignored for some reason even for .mkv... Forcing an aspect ratio, on other hand did work extremely well.

4. At the end they are at the bottom indeed... Hmm. Interesting, i couldn't pinpoint them with eye easily, and just assumed, but bottom line under hotbar always felt shorter for some reason in MPC-HC or YouTube default interpretation (one that goes 1906x1080). Main issue with going 1906x1080 and not forcing interpreters (players) to align on width and not height is that aligning becomes too... Strange as it cuts or resizes both width AND height, instead of only one of those. For some reason such videos on playback gave me a feel as if original recording was from just slightly wider screen than 1920...
   Well, i guess for time being i will utilize MKVToolNix lifehack to quick interpretation fix, then. Will wait for crop changes, i guess and hope for the best?).

Thanks for answer!

[MikhailAMD]

#2 - this was ReLive team design choice.
We are trying to push the cropping to FFmpeg ASAP.

[ByteYIFFF]

We are trying to push the cropping to FFmpeg ASAP.

Has the patch been updated? Or Could you tell me when it will be updated?
Thanks

[DimkaTsv]

Has the patch been updated? Or Could you tell me when it will be updated?

Well, there was some work done in ffmpeg patch history https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=9848
But it is not in my capabilities to guess further progress (i am not a maintainer, but random guy from internet)

[vlhm83]

We tested the Nvenc AV1 encoder, QuickSync AV1 encoder and AMD AV1 encoder. According to our tests, Nvenc and QuickSync work properly with 1920x1080 resolution with no padding. AV1 headers indicate the correct 1920x1080 size (FrameWidth/FrameHeight, RenderWidth/RenderHeight, max frame width/height fields) in both cases.
AMD AV1 encoder (with any alignment mode except 64x16 only mode) has 2 rows padding and AV1 headers indicate 1920x1082 resolution.

1. How can it be that NVidia and Intel can actually encode 1920x1080 without padding?
2. If there is some issue with doing this in AMD AV1 encoder, is it possible to provide original resolution in RenderWidth/RenderHeight fields with hope that decoder/application will use them and properly crop? Did you try to do this?
3. As I understood from this thread, the mp4 container has capability to set crop settings for video. But there can also be other containers that may not have such capability. How about rtmp protocol? (in case we stream video encoded with AV1). From another point of view, if codec rounds up frame size, it is logical to have the ability to crop frames back somehow.

[MikhailAMD]

1. This is a limitation in AMD hardware ( https://github.com/GPUOpen-LibrariesAndSDKs/AMF/wiki/AV1%20Encoder#av1-specific-api )
2. We discussed possible use of RenderWidth/RenderHeight fields in AV1 spec but consensus was that they intended for different purpose: render vs crop. Cannot highjack them.
3. Yes, mitigation of this issue depends on presence/absence of crop fields in the container and application support for these fields.

Recently FFmpeg added generic way to propagate crop info and implemented the support in some containers.