Cannot transcode HEVC in docker without privileged

[jerry-yuan]

I deployed a Jellyfin container in Docker, utilizing the RKMPP of the RK3588s platform for hardware encoding and decoding. I found that when I do not enable the container's privileged mode, Jellyfin cannot properly transcode HEVC-encoded videos, although H.264 videos seem to work fine. I tried directly running the FFmpeg command that reported an error and got the following content:

jerry@nanopi-m6:~$ docker exec -it Jellyfin /usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -f matroska -init_hw_device rkmpp=rk -hwaccel rkmpp -hwaccel_output_format drm_prime -noautorotate -afbc rga -i file:"/media/somehevc.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 hevc_rkmpp -tag:v:0 hvc1 -b:v 1116000 -maxrate 1116000 -bufsize 2232000 -profile:v:0 main -level 150 -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,vpp_rkrga=w=1280:h=536:format=p010,hwmap=derive_device=opencl,tonemap_opencl=format=nv12:p=bt709:t=bt709:m=bt709:tonemap=bt2390:peak=100:desat=0,hwmap=derive_device=rkmpp:reverse=1,format=drm_prime" -codec:a:0 libfdk_aac -ac 2 -vbr:a 5 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "bb37758ae779479a49e83565a78f582e-1.mp4" -start_number 0 -hls_segment_filename "/cache/transcodes/bb37758ae779479a49e83565a78f582e%d.mp4" -hls_playlist_type vod -hls_list_size 0 -hls_segment_options movflags=+frag_discont -y "/cache/transcodes/bb37758ae779479a49e83565a78f582e.m3u8"
ffmpeg version 7.1.3-Jellyfin Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 14 (Debian 14.2.0-19)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=arm64 --cross-prefix=/usr/bin/aarch64-linux-gnu- --toolchain=hardened --enable-cross-compile --enable-rkmpp --enable-rkrga --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, matroska,webm, from 'file:/media/somehevc.mkv':
  Metadata:
    encoder         : libebml v1.4.5 + libmatroska v1.7.1
    creation_time   : 2025-11-21T02:44:45.000000Z
    MOVIE/ENCODER   : Lavf59.27.100
  Duration: 01:58:15.86, start: 0.000000, bitrate: 4684 kb/s
  Stream #0:0(chi): Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1608 [SAR 1:1 DAR 160:67], 24 fps, 24 tbr, 1k tbn (default)
      Metadata:
        BPS             : 3209383
        DURATION        : 01:58:15.834000000
        NUMBER_OF_FRAMES: 170300
        NUMBER_OF_BYTES : 2846656756
        _STATISTICS_WRITING_APP: mkvmerge v95.0 ('Goodbye Stranger') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2025-11-21 02:44:45
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:1(chi): Audio: eac3, 48000 Hz, 5.1(side), fltp, 448 kb/s (default)
      Metadata:
        title           : Zh-cn [Dolby Digital Plus 5.1]
        BPS             : 448000
        DURATION        : 01:58:15.840000000
        NUMBER_OF_FRAMES: 221745
        NUMBER_OF_BYTES : 397367040
        _STATISTICS_WRITING_APP: mkvmerge v95.0 ('Goodbye Stranger') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2025-11-21 02:44:45
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(chi): Audio: dts (dca) (DTS), 48000 Hz, 5.1(side), fltp, 768 kb/s (default)
      Metadata:
        title           : Zh-cn [DTS ]
        BPS             : 768000
        DURATION        : 01:58:15.840000000
        NUMBER_OF_FRAMES: 665235
        NUMBER_OF_BYTES : 681200640
        _STATISTICS_WRITING_APP: mkvmerge v95.0 ('Goodbye Stranger') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2025-11-21 02:44:45
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(chi): Audio: aac (LC), 44100 Hz, stereo, fltp (default)
      Metadata:
        title           : Zh-cn [AAC 2.0]
        BPS             : 256000
        DURATION        : 01:58:15.855000000
        NUMBER_OF_FRAMES: 305593
        NUMBER_OF_BYTES : 227067380
        _STATISTICS_WRITING_APP: mkvmerge v95.0 ('Goodbye Stranger') 64-bit
        _STATISTICS_WRITING_DATE_UTC: 2025-11-21 02:44:45
        _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
[hevc_rkmpp @ 0xffff86b72300] Failed to init MPP context: -1
[vist#0:0/hevc @ 0xffff875a1380] [dec:hevc_rkmpp @ 0xffff87180280] Error while opening decoder: Generic error in an external library
Error opening output file /cache/transcodes/bb37758ae779479a49e83565a78f582e.m3u8.
Error opening output files: Generic error in an external library
jerry@nanopi-m6:~$ 

I also tried granting all container capabilities, but the issue remains unresolved.

I noticed the following error when starting Jellyfin in a container without privileged mode enabled:

mpp[28]: mpp_soc: open /proc/device-tree/compatible error
mpp[28]: mpp_platform: can not found match soc name: 
mpp[28]: mpp_platform: client 0 driver is not ready!
mpp[28]: mpp_platform: client 17 driver is not ready!

I would like to ask for advice on what might be causing this. Below is the content of my docker inspect:

[
    {
        "Id": "72a7aceed2ba09bf2cf90c81f944ce59c5399a1f7e32f84786f62adc5aefdae8",
        "Created": "2026-02-07T09:11:14.746203563Z",
        "Path": "/jellyfin/jellyfin",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 71691,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2026-02-07T09:11:15.363619089Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "starting",
                "FailingStreak": 0,
                "Log": []
            }
        },
        "Image": "sha256:6efff9c7241e94350cac13dc88ee969faff2ffc06235ef58c62d075d246a4582",
        "ResolvConfPath": "/var/lib/docker/containers/72a7aceed2ba09bf2cf90c81f944ce59c5399a1f7e32f84786f62adc5aefdae8/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/72a7aceed2ba09bf2cf90c81f944ce59c5399a1f7e32f84786f62adc5aefdae8/hostname",
        "HostsPath": "/var/lib/docker/containers/72a7aceed2ba09bf2cf90c81f944ce59c5399a1f7e32f84786f62adc5aefdae8/hosts",
        "LogPath": "/var/lib/docker/containers/72a7aceed2ba09bf2cf90c81f944ce59c5399a1f7e32f84786f62adc5aefdae8/72a7aceed2ba09bf2cf90c81f944ce59c5399a1f7e32f84786f62adc5aefdae8-json.log",
        "Name": "/Jellyfin",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "unconfined",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "JellyfinConfig:/config",
                "DSM_PT:/media:ro",
                "/usr/share/fonts:/usr/share/fonts:ro",
                "JellyfinCache:/cache"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "5",
                    "max-size": "10m"
                }
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": [
                "AUDIT_WRITE",
                "CHOWN",
                "DAC_OVERRIDE",
                "FOWNER",
                "FSETID",
                "KILL",
                "MKNOD",
                "NET_BIND_SERVICE",
                "NET_RAW",
                "SETFCAP",
                "SETGID",
                "SETPCAP",
                "SETUID",
                "SYS_CHROOT",
                "SYS_NICE",
                "SYS_RAWIO",
                "IPC_LOCK",
                "SYS_ADMIN"
            ],
            "CapDrop": [
                "AUDIT_CONTROL",
                "BLOCK_SUSPEND",
                "DAC_READ_SEARCH",
                "IPC_OWNER",
                "LEASE",
                "LINUX_IMMUTABLE",
                "MAC_ADMIN",
                "MAC_OVERRIDE",
                "NET_ADMIN",
                "NET_BROADCAST",
                "SYSLOG",
                "SYS_BOOT",
                "SYS_MODULE",
                "SYS_PACCT",
                "SYS_PTRACE",
                "SYS_RESOURCE",
                "SYS_TIME",
                "SYS_TTY_CONFIG",
                "WAKE_ALARM"
            ],
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [
                {
                    "PathOnHost": "/dev/mali0",
                    "PathInContainer": "/dev/mali0",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/rga",
                    "PathInContainer": "/dev/rga",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/mpp_service",
                    "PathInContainer": "/dev/mpp_service",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/dri",
                    "PathInContainer": "/dev/dri",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/dma_heap",
                    "PathInContainer": "/dev/dma_heap",
                    "CgroupPermissions": "rwm"
                }
            ],
            "DeviceCgroupRules": null,
            "DeviceRequests": [],
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null,
            "Init": false
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/266e1d3def92cfb21bb40367b2dddbab55c9019e842bdda0ccb09d05a19b9e0c-init/diff:/var/lib/docker/overlay2/57d4fb15af456e6d74399f512b94a281a718c7cea2aacc9cc63b8e0dbcc6de35/diff:/var/lib/docker/overlay2/f5963598d47e000e8fc4b423e4abcd20ad7ab1212e8f71cdc4f4a3e7ec03a92f/diff:/var/lib/docker/overlay2/c8bc2dce2c9e1c8a9a7b362c157898d002804e678818f9b877d77f49bcacca87/diff:/var/lib/docker/overlay2/9d01540f5b762c1520ec298dbe6443807532a409cb6b9b11f2559840a05af2af/diff:/var/lib/docker/overlay2/892b8a4484507a39ffefd35a8392bdcef0e77d60c352f2f6a7a2d367d1760357/diff:/var/lib/docker/overlay2/f3d2d24a601db11eb5e82a5c56425f839eb0688f44dd763c5cea784877692ac3/diff:/var/lib/docker/overlay2/74e2e7d3d900db8653a1eface5850770662ee9247a9f33ad74bc0435e523650b/diff:/var/lib/docker/overlay2/b68b48cde76d2211efc50d62cb1b8c8e3acb2f999ef6156d964521bc18c9df4b/diff:/var/lib/docker/overlay2/21043b1d7f8c7d7c4f29fc3e719e3c6b7ccebc87481e3c7a307bc4c950e693da/diff:/var/lib/docker/overlay2/9641b6d2daf4b628c6afc942bd5f80e4c4cb30afe9cbf93c1f19363c63e1da58/diff",
                "MergedDir": "/var/lib/docker/overlay2/266e1d3def92cfb21bb40367b2dddbab55c9019e842bdda0ccb09d05a19b9e0c/merged",
                "UpperDir": "/var/lib/docker/overlay2/266e1d3def92cfb21bb40367b2dddbab55c9019e842bdda0ccb09d05a19b9e0c/diff",
                "WorkDir": "/var/lib/docker/overlay2/266e1d3def92cfb21bb40367b2dddbab55c9019e842bdda0ccb09d05a19b9e0c/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "JellyfinConfig",
                "Source": "/var/lib/docker/volumes/JellyfinConfig/_data",
                "Destination": "/config",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "DSM_PT",
                "Source": "/var/lib/docker/volumes/DSM_PT/_data",
                "Destination": "/media",
                "Driver": "local",
                "Mode": "ro",
                "RW": false,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/usr/share/fonts",
                "Destination": "/usr/share/fonts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "JellyfinCache",
                "Source": "/var/lib/docker/volumes/JellyfinCache/_data",
                "Destination": "/cache",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "3059d1265833",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8096/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HEALTHCHECK_URL=http://localhost:8096/health",
                "DEBIAN_FRONTEND=noninteractive",
                "LC_ALL=en_US.UTF-8",
                "LANG=en_US.UTF-8",
                "LANGUAGE=en_US:en",
                "JELLYFIN_DATA_DIR=/config",
                "JELLYFIN_CACHE_DIR=/cache",
                "JELLYFIN_CONFIG_DIR=/config/config",
                "JELLYFIN_LOG_DIR=/config/log",
                "JELLYFIN_WEB_DIR=/jellyfin/jellyfin-web",
                "JELLYFIN_FFMPEG=/usr/lib/jellyfin-ffmpeg/ffmpeg",
                "XDG_CACHE_HOME=/cache",
                "MALLOC_TRIM_THRESHOLD_=131072",
                "NVIDIA_VISIBLE_DEVICES=all",
                "NVIDIA_DRIVER_CAPABILITIES=compute,video,utility",
                "LD_PRELOAD=/usr/lib/jellyfin/libjemalloc.so.2"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "curl --noproxy 'localhost' -Lk -fsS \"${HEALTHCHECK_URL}\" || exit 1"
                ],
                "Interval": 30000000000,
                "Timeout": 30000000000,
                "StartPeriod": 10000000000,
                "Retries": 3
            },
            "Image": "jellyfin/jellyfin:latest",
            "Volumes": {
                "/cache": {},
                "/config": {},
                "/media": {},
                "/usr/share/fonts": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/jellyfin/jellyfin"
            ],
            "OnBuild": null,
            "Labels": {
                "org.opencontainers.image.description": "The Free Software Media System",
                "org.opencontainers.image.documentation": "https://jellyfin.org/docs/",
                "org.opencontainers.image.source": "https://github.com/jellyfin/jellyfin-packaging",
                "org.opencontainers.image.title": "Jellyfin",
                "org.opencontainers.image.url": "https://jellyfin.org",
                "org.opencontainers.image.version": "10.11.6"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "ee50610a4db6596aa5d299b5cdc4495f4d08b04bccdf41bb8e911c7437f9c60e",
            "SandboxKey": "/var/run/docker/netns/ee50610a4db6",
            "Ports": {
                "8096/tcp": null
            },
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "dc13c2bd793f2538a9c500bbf26e209032b671a33f72fcae05559721be9f2119",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": null,
                    "MacAddress": "02:42:ac:11:00:02",
                    "NetworkID": "8d18dced0b1f740f9ed77f9e4b6de52ffb21296e0fbbeed52c6e0311426d7a87",
                    "EndpointID": "dc13c2bd793f2538a9c500bbf26e209032b671a33f72fcae05559721be9f2119",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null,
                    "DNSNames": null
                }
            }
        }
    }
]

[nyanmisaka]

ARM devices using device trees behave inconsistently with common x86 devices. For example accessing /proc/device-tree/compatible to get board/SoC info. They typically require additional settings to avoid using sudo, as stated in the documentation. https://jellyfin.org/docs/general/post-install/transcoding/hardware-acceleration/rockchip#configure-with-linux-virtualization

 --security-opt systempaths=unconfined
 --security-opt apparmor=unconfined

[jerry-yuan]

ARM devices using device trees behave inconsistently with common x86 devices. For example accessing /proc/device-tree/compatible to get board/SoC info. They typically require additional settings to avoid using sudo, as stated in the documentation. https://jellyfin.org/docs/general/post-install/transcoding/hardware-acceleration/rockchip#configure-with-linux-virtualization

 --security-opt systempaths=unconfined
 --security-opt apparmor=unconfined

Oh no, the portainer which I'm using to manage container does not support set security options. Anyway, thanks for your reply, I had refered this document and use docker command for the first start. Maybe the root cause of this problem is the security option was lost during the recreation opted by Portainer. I'll try to send an issue to ask Portainer supports this feature.

[jerry-yuan]

This has solved my problem but I have to use privileged mode before the portainer has supported --security-opt options