Help needed - Plex LXC (Unpriviledged) HW Transcoding

[archonitex]

Hi all, I'm at my witt's end trying to fix this and I'm looking for some help.

I have my Plex LXC (Unpriviledged) container that I'm trying to make work with my Quadro P2000.
Plex pass, HW transcoding enabled, tried HDR tonemap on/off, still doesn't work.

If I monitor nvidia-smi, I see it kickoff a transcoding process and immediately stop. I don't see any errors in the logs of Plex, even with Verbose on or details that could help.

I've tried manually trying a HW transcode with FFMPEG from the plex LXC and it works fine, shows up in nvidia-smi process and uses the GPU.

ffmpeg -hwaccel nvdec -i <myfile.mp4> -c:v h264_nvenc -f null -

These are the steps I've taken for the setup:

On Proxmox Host:

• Installed NVIDIA driver ( v570.158.01 ) & nvidia-persistenced. Confirmed working with nvidia-smi, and confirmed with server reboot, and nvidia-smi still works.

• Mapped the mount entry points for the GPU in the /etc/pve/lxc/.conf

lxc.cgroup.devices.allow: c 195:* rwm
lxc.cgroup.devices.allow: c 234:* rwm
lxc.cgroup.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card1 dev/dri/card1 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1 usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1 none bind,ro,create=file
lxc.mount.entry: /usr/lib/x86_64-linux-gnu/libnvcuvid.so.1 usr/lib/x86_64-linux-gnu/libnvcuvid.so.1 none bind,ro,create=file

Inside the LXC:

• Installed nvidia-utils-570-server. Confirmed working with nvidia-smi, with the same driver version v570.158.01

Please help

[archonitex]

Logs I see when trying to transcode

Jul 26, 2025 14:53:53.121 [123805888576312] DEBUG - [Req#5ca/Transcode] Codecs: hardware transcoding: testing API nvdec for device 'pci:0000:0a:00.0' (NVIDIA GP106GL [Quadro P2000])
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] Codecs: Testing with profile 'Main 10'
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - nal_unit_type: 20(IDR_N_LP), nuh_layer_id: 0, temporal_id: 0
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Decoding VPS
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Main 10 profile bitstream
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Decoding SPS
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Main 10 profile bitstream
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Decoding VUI
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Decoding PPS
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Format cuda chosen by get_format().
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - Format cuda requires hwaccel initialisation.
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - CUDA texture alignment: 512
Jul 26, 2025 14:53:53.167 [123805888576312] DEBUG - [Req#5ca/Transcode] [FFMPEG] - CUDA texture alignment: 512
Jul 26, 2025 14:53:53.172 [123805888576312] VERBOSE - [Req#5ca/Transcode] [FFMPEG] - NVDEC capabilities:
Jul 26, 2025 14:53:53.172 [123805888576312] VERBOSE - [Req#5ca/Transcode] [FFMPEG] - format supported: yes, max_mb_count: 262144
Jul 26, 2025 14:53:53.172 [123805888576312] VERBOSE - [Req#5ca/Transcode] [FFMPEG] - min_width: 144, max_width: 8192
Jul 26, 2025 14:53:53.172 [123805888576312] VERBOSE - [Req#5ca/Transcode] [FFMPEG] - min_height: 144, max_height: 8192
Jul 26, 2025 14:53:53.178 [123805888576312] DEBUG - [Req#5ca/Transcode] Codecs: dummy-frame test succeeded
Jul 26, 2025 14:53:53.450 [123805888576312] DEBUG - [Req#5ca/Transcode] MDE: E9 - REDACTED: Audio Direct Streaming is disabled, so video's audio stream will be transcoded
Jul 26, 2025 14:53:53.450 [123805888576312] DEBUG - [Req#5ca/Transcode] MDE: Cannot direct stream audio stream due to profile or setting limitations
Jul 26, 2025 14:53:53.450 [123805888576312] DEBUG - [Req#5ca/Transcode] MDE: REDACTED: selected media 0 / 1257
Jul 26, 2025 14:53:53.450 [123805826788152] DEBUG - [Req#5ca/Transcode] Killing job.
Jul 26, 2025 14:53:53.450 [123805826788152] DEBUG - [Req#5ca/Transcode] Signalling job ID 862 with 9
Jul 26, 2025 14:53:53.450 [123805826788152] DEBUG - [Req#5ca/Transcode] Job was already killed, not killing again.
Jul 26, 2025 14:53:53.450 [123805826788152] DEBUG - [Req#5ca/Transcode] Stopping transcode session ywnxg6om0qhmmv8g9ssrnzi7
Jul 26, 2025 14:53:53.450 [123805888576312] DEBUG - [Req#5ca/Transcode] Streaming Resource: Session 0x7099c989a368:rsffnzkjnpqhopbwreb4m7jg changed from using 3265kbps to 432kbps of WAN bandwidth. Used bandwidth is now 432kbps
Jul 26, 2025 14:53:53.450 [123805888576312] DEBUG - [Req#5ca/Transcode] Streaming Resource: Session 0x7099c989a368:rsffnzkjnpqhopbwreb4m7jg changed transcode slot usage to used. Used slots is now 1
Jul 26, 2025 14:53:53.450 [123805884357432] DEBUG - [Req#5ca/Transcode] Cleaning directory for session ywnxg6om0qhmmv8g9ssrnzi7 (/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-ywnxg6om0qhmmv8g9ssrnzi7-c94122a7-1db4-4169-952f-463f2b5c45e0)

[archonitex]

Well if anyone stumbles upon this,

I changed everything to driver version 535.247.01 and everything worked as expected.
550 or 570 did not work.

[ThorpeJosh]

@archonitex you are a legend. I was chasing my tail with this issue. I even tried different version drivers but I was always using newer ones 580, 570, 550.

You prompted me to try 535. Going back to 535.261.03 resolved the ffmpeg cuda issue.