All encoders fail whether using nvfbc or KMS capture methods

[cjvirtucio87]

Hi, I need help troubleshooting an issue where all the encoders fail on Fedora 40. I've tried the following:

1. Building off of the lizardbyte/sunshine:latest-fedora-40 image and running a docker container.
2. Running this release directly on the host (fedora 40).
3. Building locally from master and installing and running that.
4. Setting capture to nvfbc.
5. Setting capture to kms.

In all cases, all encoders fail:

[2024:07:28:21:13:41]: Info: Trying encoder [nvenc]
[2024:07:28:21:13:41]: Info: Encoder [nvenc] failed
[2024:07:28:21:13:41]: Info: Trying encoder [vaapi]
[2024:07:28:21:13:41]: Info: Encoder [vaapi] failed
[2024:07:28:21:13:42]: Info: Trying encoder [software]
[2024:07:28:21:13:42]: Info: Encoder [software] failed
[2024:07:28:21:13:42]: Fatal: Unable to find display or encoder during startup.
[2024:07:28:21:13:42]: Fatal: Please check that a display is connected and powered on.
[2024:07:28:21:13:42]: Error: Video failed to find working encoder

I also see this output above the encoder failures:

[2024:07:28:21:13:41]: Error: Unable to initialize capture method
[2024:07:28:21:13:41]: Error: Platform failed to initialize

No CAP_SYS_ADMIN related failures (I've made sure to run setcap cap_sys_admin+p "$(readlink -f "$(command -v sunshine)")" before starting up).

Here's my nvidia-smi output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3070 Ti     Off |   00000000:0A:00.0  On |                  N/A |
|  0%   33C    P8             13W /  310W |     451MiB /   8192MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Installed nvidia packages:

$ sudo dnf list --installed | grep nvidia
akmod-nvidia.x86_64                                  3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
kmod-nvidia-6.9.11-200.fc40.x86_64.x86_64            3:555.58.02-1.fc40                  @@commandline
kmod-nvidia-6.9.6-200.fc40.x86_64.x86_64             3:555.58.02-1.fc40                  @@commandline
kmod-nvidia-6.9.9-200.fc40.x86_64.x86_64             3:555.58.02-1.fc40                  @@commandline
libnvidia-container-tools.x86_64                     1.16.0-1                            @nvidia-container-toolkit
libnvidia-container1.x86_64                          1.16.0-1                            @nvidia-container-toolkit
nvidia-container-toolkit.x86_64                      1.16.0-1                            @nvidia-container-toolkit
nvidia-container-toolkit-base.x86_64                 1.16.0-1                            @nvidia-container-toolkit
nvidia-gpu-firmware.noarch                           20240709-1.fc40                     @updates
nvidia-modprobe.x86_64                               3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
nvidia-persistenced.x86_64                           3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
nvidia-settings.x86_64                               3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia.x86_64                           3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda.x86_64                      3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda-libs.i686                   3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda-libs.x86_64                 3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-kmodsrc.x86_64                   3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-libs.i686                        3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-libs.x86_64                      3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-power.x86_64                     3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates

Thanks!

[ReenigneArcher]

Our fedora 40 rpm / image doesn't have support for cuda. This is due to Fedora shipping an incompatible version of GCC.

[cjvirtucio87]

ah, interesting. I tried debian-bookworm. not sure what's missing but now I'm running into Couldn't load cuda: -1 errors.

[2024:08:01:00:02:20]: Error: Couldn't load cuda: -1
[2024:08:01:00:02:20]: Info: /dev/dri/card1 -> nvidia-drm

<a bit of debug logging>

[2024:08:01:00:02:20]: Info: Found connector ID [88]
[2024:08:01:00:02:20]: Info: Found cursor plane [38]
Cannot load libcuda.so.1
[2024:08:01:00:02:20]: Error: Couldn't load cuda: -1
Cannot load libcuda.so.1
[2024:08:01:00:02:20]: Error: Couldn't load cuda: -1
[2024:08:01:00:02:20]: Info: Encoder [nvenc] failed

[ReenigneArcher]

Did you install cuda?

[cjvirtucio87]

yeah:

$ sudo nvidia-smi
Sat Aug  3 15:14:30 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 555.58.02      CUDA Version: 12.5     |

...

$ sudo dnf list --installed | grep 'cuda'
xorg-x11-drv-nvidia-cuda.x86_64                      3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda-libs.i686                   3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda-libs.x86_64                 3:555.58.02-1.fc40                  @rpmfusion-nonfree-updates

[brian6932]

I can reproduce this on Windows 10 with nvfbc, but not on the same release, for me this is a regression in 287ac4c0fbb0df96cd7d83f1c16cf84070eb50d1, 2d706d31041899eac5af0b5d242faa38e2c879ab and before works. Should I open a separate issue?

sunshine.conf

upnp = on
address_family = both
capture = nvfbc
encoder = nvenc
nvenc_twopass = disabled
nv_rc = cbr
encoderCscMode = 1
dynamicRange = 0
chromaSamplingType = 1

❯ nvidia-smi
Wed Aug 21 00:20:46 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.81                 Driver Version: 560.81         CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1080 Ti   WDDM  |   00000000:0D:00.0  On |                  N/A |
|  0%   28C    P8             22W /  126W |    7848MiB /  11264MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

[ReenigneArcher]

Nvfbc isn't an option on Windows (and has never been). You should use the WebUI to configure Sunshine.

Before the commit you mentioned, we just automatically set the capture option on Windows so it didn't really matter what you had in your config. Now there are choices, so the value must be a valid selection.

[brian6932]

I see, is it not possible to use on Windows, or just lacking server support? I'm aware it's "deprecated", but Steam Remote Play has support, and you can patch drivers on Windows just the same.

[ReenigneArcher]

It's possible, but we're not implementing a deprecated protocol.

[ReenigneArcher]

LizardByte now has copr repos you can use to install Sunshine. Currently fedora 39-41 and rawhide are supported for aarch64 and x86_64. All builds also have cuda support (v12.6.2).

https://docs.lizardbyte.dev/projects/sunshine/en/master/md_docs_2getting__started.html#fedora

Note: stable will be empty until the next stable release.

Please let me know if these packages work better for you.

[cjvirtucio87]

Hi, thanks for the update. Latest fedora image has been working well for the past few weeks. Marking this as the answer since it also points to other options.

[ReenigneArcher]

Thanks! Hopefully there's no regressions in the new packages, but let us know if you find any.

Note: the existing Docker builds are removed (as they put too much strain on our GitHub runner allocation)