Sunshine "Cannot load libcuda.so.1" and default to software encoding after upgrading to 0.21.0

[PierreBeucher]

I upgraded to Sunshine 0.21.0 from 0.19.x, but nvenc encoding fails with Cannot load libcuda.so.1 and related errors, eventually falling back to software encoding (with the performance you can imagine). These errors didn't show up before and nvenc encoding was working fine.

I'm using Nvidia A10G Tensor GPU (on AWS g5 instance) with driver v545.29.02 and CUDA 12.3.

Here are full log outputs:

Started Sunshine server.
[origin_web_ui_allowed] -- [lan]
[2023:12:03:20:04:39]: Info: Sunshine version: 0.21.0
[2023:12:03:20:04:39]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2023:12:03:20:04:39]: Info: Detecting monitors
[2023:12:03:20:04:39]: Info: Detected monitor 0: DVI-D-0, connected: true
[2023:12:03:20:04:39]: Info: Detected monitor 1: DVI-D-1, connected: false
[2023:12:03:20:04:39]: Info: Detected monitor 2: DVI-D-2, connected: false
[2023:12:03:20:04:39]: Info: Detected monitor 3: DVI-D-3, connected: false
[2023:12:03:20:04:39]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2023:12:03:20:04:39]: Info: Trying encoder [nvenc]
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x33f7240] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x33f7240] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: System tray created
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x3403580] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x3403580] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x340d040] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x340d040] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x334ea40] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x334ea40] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Encoder [nvenc] failed
[2023:12:03:20:04:39]: Info: Trying encoder [vaapi]
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Encoder [vaapi] failed
[2023:12:03:20:04:39]: Info: Trying encoder [software]
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] profile High, level 4.2, 4:2:0, 8-bit
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] frame I:1     Avg QP: 6.10  size:  1182
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] mb I  I16..4: 100.0%  0.0%  0.0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] 8x8 transform intra:0.0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] i16 v,h,dc,p: 99%  0%  1%  0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 50%  0% 50%  0%  0%  0%  0%  0%  0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] i8c dc,h,v,p: 100%  0%  0%  0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] kb/s:567.36
[2023:12:03:20:04:39]: Info:
[2023:12:03:20:04:39]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2023:12:03:20:04:39]: Info:
[2023:12:03:20:04:39]: Info: Found H.264 encoder: libx264 [software]
[2023:12:03:20:04:39]: Info: Adding avahi service Sunshine
[2023:12:03:20:04:39]: Error: Failed to add _nvstream._tcp service: Not permitted
[2023:12:03:20:04:39]: Info: Configuration UI available at [https://localhost:47990]

I suppose I'm missing some OS packages, but I don't know which ones as NVidia drivers and CUDA are installed and were working fine before.

For context: I'm using NixOS and upgrading from 0.19.x to 0.21.0 also had me upgrade from NixOS 23.05 to 23.11 so it may be a NixOS Sunshine packaging issue. For more context and source code, this is part of Cloudy Sunshine, a Sunshine in the Cloud project I'm working on.

I also had another "system package missing" problem causing segfault (NixOS/nixpkgs#271333 solved by NixOS/nixpkgs#271352, which I implemented locally), so it may be a similar issue.

Understand what's happening (which piece of software or lib Sunshine is missing) may help me solve it.

Thanks by advance !

[ReenigneArcher]

Please open this as an issue on NixOS.