Request to add 4:4:4 chroma subsampling encoding.

[PmNz8]

Hello.

I do not wknow where to check it, but to my eyes (I might be wrong) it looks like for now Sunshine is doing 4:2:0 chroma subsampling encoding. Modern GPUs have no issue encoding and decoding 4:4:4 video stream - it would greatly improve visual clarity of text, small elements and saturation.

I would like to request this feature to be a optional toggle to enable if our hardware supports it.

For example, I have RTX 3090 in source PC

and Rocket Lake i5 with UHD 730 in client PC that supports HEVC 4:4:4 decoding. I guess many people would benefit greatly from this feature. Thanks for considering implementing it.

[cgutman]

Yeah, this is on my list. It will require changes on the Moonlight side too, so it's more work than just support it in Sunshine. I may try to do it together with AV1 support when I work on that.

[PmNz8]

Okay, thanks for quick reply. I will be patiently waiting then.

[Inventatorul1]

Any news?

[forianzsiga]

yep we'd be so glad :,)

[Sharptsa]

Hi, any news on this? 🙂

[TomiWebPro]

Still waiting! People are hyped about this feature coming into play!

[n00mkrad]

Bump, there is NO free streaming solution right now that supports 4:4:4.

I would even pay money for this - just not willing to give into a subscription (Parsec).

[Ysabelle12]

I agree we need to support this. It's a must in graphic designing, Parsec is simply expensive and is also using NVENC for 4:4:4. I'd honestly say that it can be more useful than 10bit because using 4:4:4 in design like illustrator is useful.

[hyyz17200]

Yeah I do hope so! And I'd rather pay a reasonable price for this!

[n00mkrad]

Bump :)

[cgutman]

I did look into this and there are a few issues with the current hardware and software ecosystems that make this challenging to support well, which is why I didn't end up doing this work together with AV1 enablement.

First, the hardware decoding side is a complete mess. The standard Microsoft DXVA/D3D11VA API for video decoding does not define decoding profiles for HEVC Main 4:4:4, so some vendors (Intel) have made up their own (which is actually the best case) and others (Nvidia) require use of their own vendor-specific APIs to decode 4:4:4 content. AMD seems to lack 4:4:4 decoding support entirely. FFmpeg, which we use for encoding and decoding, doesn't support these custom 4:4:4 DXVA vendor extensions either, so we'd also need to write a FFmpeg patches to enable them. On non-PC platforms, the situation is even worse regarding hardware support (and non-PC clients don't support software fallback either).

The best hope for a reasonable cross-vendor (and cross-platform!) API for decoding HEVC 4:4:4 is probably Vulkan Video, which Moonlight now supports via libplacebo in the latest pre-release code. However, I haven't tested to see if HEVC 4:4:4 decoding is supported in the current release of Intel and AMD drivers.

On the software decoding side, SDL doesn't support consuming YUV 4:4:4 texture formats at all, so we don't really have any way to support software decoding using any of the existing code. Fortunately, the libplacebo renderer may come to the rescue again here, since it seems to support importing software decoding frames for rendering.

And on top of all this, AV1 support for 4:4:4 encoding in current generation hardware is a total no-show. Even on hardware that supports both HEVC Main 4:4:4 and AV1, none of it supports AV1 4:4:4.

There are probably more issues I'm forgetting or haven't run into yet. None of these issues are complete showstoppers, but all this is why it wasn't just as simple as plumbing a few new codec options like AV1 was.

[KWottrich]

It seems like Parsec's limitations agree completely with your assessment, given their hardware acceleration requirements for the 4:4:4 feature.

Maybe I'm misinterpreting, but it sounds like most of the limitations are on the DEcoding side. Since that's more on the Moonlight side of the equation -- and acknowledging that getting 4:4:4 will require changes to both sides -- is adding 4:4:4 HEVC ENcoding support something that could be done on the Sunshine side? Does FFmpeg support hardware encoding for 4:4:4 HEVC from Intel/Nvidia/AMD? I can only find explicit documentation of support from Nvidia here: https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

None of these issues are complete showstoppers, but all this is why it wasn't just as simple as plumbing a few new codec options like AV1 was.

Sounds like you weren't saying "this won't happen", but rather "this is why this hasn't happened yet" 😄

[n00mkrad]

Let's hope HW-accelerated HEVC 4:4:4 can be an option at least

[amazingfate]

Yeah even desktop platforms lack 4:4:4 support. For embedded devices situation is even worse: almost all arm devices do not supoort 4:4:4 decoding.
Microsoft RDP combines two 4:2:0 to one 4:4:4 to overcome the hardware limitation, but that is not a good method because of software processing: https://techcommunity.microsoft.com/t5/security-compliance-and-identity/remote-desktop-protocol-rdp-10-avc-h-264-improvements-in-windows/ba-p/249588

[nyanmisaka]

Does FFmpeg support hardware encoding for 4:4:4 HEVC from Intel/Nvidia/AMD?

Intel QSV:

• HEVC 8/10-bit 4:4:4 decoding & encoding are supported on 10th Gen Ice Lake and newer.

Nvidia NVENC/NVDEC:

• HEVC 8/10-bit 4:4:4 decoding is supported on GTX10 series and newer.
• HEVC 8/10-bit 4:4:4 encoding is supported on GTX16 series and newer.

AMD AMF:

• No 4:4:4 support in all existing AMD GPU hardware, as of the Radeon RX 7900 series.

Windows DXVA/D3D11VA:

• Vendor DDI specific: HEVC 8/10-bit 4:4:4 decoding is supported on 10th Gen Ice Lake and newer.
• DXVA specs: DXVA_PicParams_HEVC_RangeExt has only been recently defined and does not yet have driver support. Intel and Nvidia may support this via driver updates in the future. FFmpeg patches are need too.

Linux Libva/VA-API:

• On Intel, HEVC 8/10-bit 4:4:4 decoding & encoding are supported on 10th Gen Ice Lake and newer.
• On AMD: no support in hardware.

[PhytochromeFr]

https://github.com/microsoft/DirectX-Headers/blob/main/include/directx/d3d12video.h
d3d12va have HEVC 4:4:4 profile.

[InsaneDoggo]

I know Cross-platform compatibility is important, although it make sense to implement such feature as "for advanced users" or "for enthusiasts" i.e. make available only if HW supports it, maybe even put some "validation responsibility" onto users e.g. to check that HW actually support feature. I see no reason to skip/delay feature for Desktop users with high-end PCs just because there are exists some percent of "smartphone enjoyers".

[mrpops2ko]

I agree completely and I think that a use case scenario is important. The primary basis a lot of people want 4:4:4 support is for the creation of thin desktop clients that operate at high hz, in order to virtualise their desktops from a server. It was mine at least until I realised that its not worth going down this path until some legible text patch for 4:4:4 exists which currently it doesn't.

[amazingfate]

All the limitations of cross-platform compatibility that cgutman has mentioned like Microsoft DXVA/D3D11VA API, ffmpeg, SDL are widely used in sunshine and moonlight. If you want to skip them, developers have to write an absolutely new and vendor api based streaming framework. How much you can pay to let opensource developers develop software with the way you like?

[InsaneDoggo]

@amazingfate we compare two scenarios:

1. Support subset of cases
2. Support all cases

I don't see how "1" may be more expensive then "2". My take was about avoiding extra efforts to support some exotic cases if it helps to save resources.

[amazingfate]

Take a look at current code and read cgutman's reply, you should know why.

[InsaneDoggo]

@amazingfate I appreciate that you trying to help, but it looks like you misunderstood my original message. My statement is completely "implementation details"-agnostic, i.e. I'm not suggesting to use or not use specific technology/lib/framework. I'm saying that I as "end user" will not mind if this software would have "conditionally available" features. e.g. if you have "GPU X " then you can turn on "4:4:4 feature", otherwise you will see popup "HW requirements are not met", that's it. Please do not ascribe to me statements I have never said!

[KWottrich]

FYI all LizardByte/Sunshine#2533 and moonlight-stream/moonlight-qt#1282

[labor4]

FWIW, I did some testing on a Pascal Quadro, and...

chlimeo@chlimeo:~$ ffmpeg -h encoder=hevc_nvenc
...
Encoder hevc_nvenc []:
    General capabilities: dr1 delay hardware 
    Threading capabilities: none
    Supported hardware devices: cuda cuda 
    Supported pixel formats: yuv420p nv12 p010le yuv444p p016le yuv444p16le bgr0 bgra rgb0 rgba x2rgb10le x2bgr10le gbrp gbrp16le cuda
...

... yuv444p16le plays surprisingly well on Macos/ Browser. I think it was a WebRTC test.
If I play it in Macos/VLC, it seems to utilize my "Intel UHD Graphics 630".

I attached the test file (mp4, below 1MB):
https://github.com/LizardByte/Sunshine/assets/18475375/3dbbb9f3-96be-46e3-a6fb-71c69b17222d
(it does show 10bit 444 in Mediainfo, though...)

[ns6089]

Windows encoding of native 4:4:4 is ready for testing LizardByte/Sunshine#2533
Should work on nvidia and intel gpus (haven't tested intel yet though)

Vulkan decoding of native 4:4:4 in moonlight-qt is ready for testing moonlight-stream/moonlight-qt#1282
Again should work on nvidia and intel gpus

Emulated 4:4:4 through recombining doublesized 4:2:0 should be ready soon™ (I've encountered an inconsistency in vulkan/libplacebo that mess things up a bit in some specific cases)

Native and recombined 4:4:4 encoding should be possible on Linux and MacOS, but I probably won't work on it myself.

Recombined 4:4:4 decoding won't be coming to moonlight-android any time soon, too many things need to be implemented there first.

[WinkelCode]

Just some feedback/updates:

I downloaded some recent CI builds, modified the iOS app and unlocked the bitrate options.

My "source" PC is running a Nvidia RTX 4070 (no-super, non-ti)

As target devices I used an M2 Pro Mac Mini and an M1 iPad Pro 12.9" (5th gen). The target devices were both connected directly to the source PC via a gigabit Ethernet cable (no switches/routers in between), on the Mac I used the built-in port, on the iPad I used a hub with Ethernet.

Note: In all cases I verified (and counter-verified) the 4:4:4 chroma setting with a test pattern: https://www.rtings.com/images/test-materials/2017/chroma-444.png - when doing this test yourself, don't forget to disable HiDPI as it can falsify the result.

• I don't remember everything I changed in Sunshine, I just configured stuff to what I thought probably reduces latency to the minimum. I also disabled things like error correction since the devices are wired directly. I'll need to do some more settings tuning down the road, this was just for a quick test of what's possible.
• On the Mac Mini I used this configuration: 3440x1440@144hz, 4:4:4 chroma, HDR, 800Mbit/s bitrate, 10 bit HEVC.
  • It's really impressive that this works! The quality is really close to what I see on the screen normally, I couldn't spot any compression artifacts ever (it only reaches the 800Mbit/s budget with lots of motion). Text to me looked indistinguishable from connecting directly to the GPU (finally possible due to 4:4:4 chroma). The downsides, latency and sharpness, were really minimal. If I didn't have the direct comparison, I probably wouldn't notice.
  • Note regarding sharpness: Distant objects still look ever so slightly (very slightly, like FXAA is applied) blurry in games, regardless of motion (maybe unavoidable with lossy encoding). However, text on the desktop looks great, which was my main pet-peeve before.
• On the iPad, I used this configuration: 2732x2048@120hz (native to the display), 4:4:4 chroma, HDR, 150Mbit/s bitrate, 10 bit HEVC.
  • To do this I had to modify the moonlight-ios code by first updating the moonlight-common-c submodule and then having VIDEO_FORMAT_H265_REXT10_444 be added to _streamConfig.supportedVideoFormats. While I was at it I also changed the bitrate slider to have higher bitrates.
  • As a side note, the latest commit: moonlight-stream/moonlight-ios@022352c crashes due to Failed to load Legacy provider resulting in the p12 variable being a null pointer. I just used the one before that moonlight-stream/moonlight-ios@557765f which worked fine for me. If someone wants to open an issue for that, please feel free to do so. I am not familiar enough with the project/error to know if I maybe have something incorrectly configured.
  • On the iPad, bitrates greater than 150/200Mbit resulted in stutters so I didn't pursue that any further, this user: BitrateUnlock moonlight-stream/moonlight-qt#1377 mentions that they can stream at higher bitrates on their M1 (some type of Mac) device, so I guess decoding is either nerfed on iPads or there are some missing optimizations in Moonlight there.
  • Other than that, the stream quality was once again fantastic, HDR worked great, 4:4:4 is wonderful to have and makes the iPad really useful for creative work.

Really happy to see the progress being made and I am excited to see these features make it into the official releases hopefully soon. The Apple Media Engine is also a beast being able to decode ~80MB/s of HEVC video.

[Mekkermb]

Hi WinkelCode,

Would you be able to share your build with the updated functionality for the iPad? It would be incredibly helpful.

Thanks in advance!

[WinkelCode]

@Mekkermb

Sharing a build won't really work well for iPad because of Apple sideloading restrictions.

But I can tell you which lines I changed, note that I haven't tested this in a while, and the project was update since then, but maybe it's helpful for you:

Connection.m in Limelight/Stream:

in

-(NSString*) getActiveCodecName
{
    switch (activeVideoFormat)
[...]

I added

        case VIDEO_FORMAT_H265_REXT8_444:
            return @"HEVC RExt 4:4:4 8-bit";
        case VIDEO_FORMAT_H265_REXT10_444:
            return @"HEVC RExt 4:4:4 10-bit";

MainFrameViewController.m in Limelight/ViewControllers:

        if (streamSettings.enableHdr && (AVPlayer.availableHDRModes & AVPlayerHDRModeHDR10) != 0) {
            _streamConfig.supportedVideoFormats |= VIDEO_FORMAT_H265_MAIN10;
            _streamConfig.supportedVideoFormats |= VIDEO_FORMAT_H265_REXT8_444; // I added this
            _streamConfig.supportedVideoFormats |= VIDEO_FORMAT_H265_REXT10_444; // and this
        }

Edit: This could maybe be improved, you don't necessarily need HDR for 10 bit or 444 color.

finally in SettingsViewController.m in Limelight/ViewControllers:

static const int bitrateTable[] = {
[...]

add/replace your own bitrates, i.e.

[...]
    150000,
    200000, // Higher rates than the previous max
    300000,
    400000,
    500000,
[...]

Keep in mind it's presented as a slider, so too many values can be tricky to actually select in the UI.

Edit 2: You also need to make sure the moonlight-common-c submodule is updated so the new formats like VIDEO_FORMAT_H265_REXT10_444 are available.

[Mekkermb]

@WinkelCode

Thanks for the help! I eventually figured it out. It’s impressive how much better the experience is with 4:4:4 and HDR on my iPad M4. Even though it’s decoding via software, I think it’s a worthy tradeoff.

[lc-guy]

Is there any update on native/recombined YUV4:4:4 host support on Linux? I couldn't see any MR or discussion related to it around the repos or in the discord.