[Feature request] Support AMD VCE encoder which is supported by official HandBrake

[HexStan]

Idea

Seems that Docker HandBrake only supports for Intel QSV encoder. Since the upstream official HandBrake release already supports AMD VCE encoder, could we please add support for this in Docker HandBrake? There are many users with AMD GPU who would benefit from this.

Thanks!

[jlesage]

With the latest release, I enabled VCE support in HandBrake. I don't know however if it's enough or if additional dependencies are needed. Please let me know.

[HexStan]

With the latest release, I enabled VCE support in HandBrake. I don't know however if it's enough or if additional dependencies are needed. Please let me know.

Thanks for your hard work! I've pulled the latest image and tried modifying my Docker Compose, but it doesn't seem to be working. The logs indicate that the container has detected the GPU, but the relevant options aren't appearing in the Handbrake GUI.

Docker Compose:

services:
  handbrake:
    image: jlesage/handbrake:latest
    group_add:
      - video
    networks:
      - network
    ports:
      - "5800:5800"
      - "5900:5900"
    environment:
      - INSTALL_PACKAGES=linux-firmware-amdgpu mesa-dri-gallium mesa-va-gallium mesa-vdpau-gallium mesa-vulkan-layers xf86-video-amdgpu
      - USER_ID=0
      - GROUP_ID=0
      - LANG=zh_CN.UTF-8
      - TZ=Asia/Shanghai
      - KEEP_APP_RUNNING=1
      - WEB_LISTENING_PORT=5800
      - VNC_LISTENING_PORT=5900
      - ENABLE_CJK_FONT=0
      - WEB_AUDIO=1
    volumes:
      - "/opt/amdgpu:/opt/amdgpu"
      - "/opt/amdgpu-pro:/opt/amdgpu-pro"
      - "/opt/rocm-6.4.4:/opt/rocm-6.4.4"
      - "/mnt/Local/docker/handbrake/config:/config:rw"
      - "/mnt/Local/docker/handbrake/fonts:/usr/share/fonts"
    devices:
      - /dev/dri:/dev/dri
      - /dev/kfd:/dev/kfd

[cont-init   ] 15-check-gpu.sh: executing...
[cont-init   ] 15-check-gpu.sh:   [ OK ]   the device /dev/dri/renderD128 has proper permissions.
[cont-init   ] 15-check-gpu.sh:   [ OK ]   the container can write to device /dev/dri/renderD128.
[cont-init   ] 15-check-gpu.sh:   [ OK ]   the device /dev/dri/card1 has proper permissions.
[cont-init   ] 15-check-gpu.sh:   [ OK ]   the container can write to device /dev/dri/card1.
[cont-init   ] 15-check-gpu.sh:   [ OK ]   a GPU node has been found.
[cont-init   ] 15-check-gpu.sh:   [ OK ]   a render node has been found.
[cont-init   ] 15-check-gpu.sh: terminated successfully.

I noticed that the container runs check-qsv.sh during the init stage, but there isn't a corresponding detection script for AMD GPUs. Could that be the issue?

[cont-init   ] 54-check-qsv.sh: executing...
[cont-init   ] 54-check-qsv.sh: Processor: AMD Ryzen 7 5800H with Radeon Graphics         
[cont-init   ] 54-check-qsv.sh: Microarchitecture: AMD_ZEN3
[cont-init   ] 54-check-qsv.sh: Kernel: 6.14.0-37-generic
[cont-init   ] 54-check-qsv.sh: Intel Quick Sync Video not supported: video adapter not using i915 driver.
[cont-init   ] 54-check-qsv.sh: terminated successfully.

[jlesage]

54-check-qsv.sh is just a helper to troubleshoot issues with QSV, but does not affect HandBrake at all.

Can you share the output of docker exec handbrake HandBrakeCLI ?

Also, don't you have to run docker with the AMD container runtime ?
See https://instinct.docs.amd.com/projects/container-toolkit/en/latest/container-runtime/quick-start-guide.html

[HexStan]

54-check-qsv.sh is just a helper to troubleshoot issues with QSV, but does not affect HandBrake at all.

Can you share the output of docker exec handbrake HandBrakeCLI ?

Also, don't you have to run docker with the AMD container runtime ? See https://instinct.docs.amd.com/projects/container-toolkit/en/latest/container-runtime/quick-start-guide.html

Sometimes HandBrakeCLI output looks like this, which seems to indicate that the container has detected the AMD driver and is attempting to invoke it:

[23:42:34] Compile-time hardening features are enabled
Cannot load libnvidia-encode.so.1
0125-11-19 15:22:02 62306A00 [AMFDevicePALImpl]   Error: ../../../../../runtime/src/core/DevicePALImpl.cpp(436):Assertion failed:Pal::IPlatform::EnumerateDevices() returned 0 devices
0125-11-19 15:22:02 62306A00 [AMFContextImpl]   Error: ../../../../../runtime/src/core/ContextImpl.cpp(2016):GetDevicePAL() AMFDevicePAL::InitFromAdapterInfo() failed with res =1
0125-11-19 15:22:02 62306A00 [EncodeQueuePalImpl]   Error: ../../../../../runtime/src/components/EncoderCore/EncodeQueuePalImpl.cpp(892):Assertion failed:Pal retuned zero instances
0125-11-19 15:22:02 62306A00 [EncodeQueueService] Warning: Failed to create encode queue serivce.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreBaseImpl] Warning: Failed to initialize EncodeQueueFactory.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreBaseImpl] Warning: Failed to create encode queue serivce.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreImpl] Warning: Encode Core dll not found
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreH264] Warning: CreateServices failed.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreH264] Warning: *****Failed to create H264 encoder component.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreImpl]   Error: ../../../../../runtime/src/components/EncoderCore/EncoderCoreImpl.cpp(1182):Assertion failed:DestroyEncodeServiceBase() - pFn == NULL
[15:22:02] vcn: not available on this system
[15:22:02] qsv: not available on this system
0125-11-19 15:22:02 62306A00 [AMFDevicePALImpl]   Error: ../../../../../runtime/src/core/DevicePALImpl.cpp(436):Assertion failed:Pal::IPlatform::EnumerateDevices() returned 0 devices
0125-11-19 15:22:02 62306A00 [AMFContextImpl]   Error: ../../../../../runtime/src/core/ContextImpl.cpp(2016):GetDevicePAL() AMFDevicePAL::InitFromAdapterInfo() failed with res =1
0125-11-19 15:22:02 62306A00 [EncodeQueuePalImpl]   Error: ../../../../../runtime/src/components/EncoderCore/EncodeQueuePalImpl.cpp(892):Assertion failed:Pal retuned zero instances
0125-11-19 15:22:02 62306A00 [EncodeQueueService] Warning: Failed to create encode queue serivce.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreBaseImpl] Warning: Failed to initialize EncodeQueueFactory.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreBaseImpl] Warning: Failed to create encode queue serivce.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreImpl] Warning: Encode Core dll not found
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreAv1] Warning: CreateServices failed.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreAv1] Warning: *****Failed to create AV1 encoder component.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreImpl]   Error: ../../../../../runtime/src/components/EncoderCore/EncoderCoreImpl.cpp(1182):Assertion failed:DestroyEncodeServiceBase() - pFn == NULL
0125-11-19 15:22:02 62306A00 [AMFDevicePALImpl]   Error: ../../../../../runtime/src/core/DevicePALImpl.cpp(436):Assertion failed:Pal::IPlatform::EnumerateDevices() returned 0 devices
0125-11-19 15:22:02 62306A00 [AMFContextImpl]   Error: ../../../../../runtime/src/core/ContextImpl.cpp(2016):GetDevicePAL() AMFDevicePAL::InitFromAdapterInfo() failed with res =1
0125-11-19 15:22:02 62306A00 [EncodeQueuePalImpl]   Error: ../../../../../runtime/src/components/EncoderCore/EncodeQueuePalImpl.cpp(892):Assertion failed:Pal retuned zero instances
0125-11-19 15:22:02 62306A00 [EncodeQueueService] Warning: Failed to create encode queue serivce.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreBaseImpl] Warning: Failed to initialize EncodeQueueFactory.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreBaseImpl] Warning: Failed to create encode queue serivce.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreImpl] Warning: Encode Core dll not found
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreHevc] Warning: CreateServices failed.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreHevc] Warning: *****Failed to create HEVC encoder component.
0125-11-19 15:22:02 62306A00 [AMFEncoderCoreImpl]   Error: ../../../../../runtime/src/components/EncoderCore/EncoderCoreImpl.cpp(1182):Assertion failed:DestroyEncodeServiceBase() - pFn == NULL
[15:22:03] hb_init: starting libhb thread
[15:22:03] thread 7097604eb640 started ("libhb")

And sometimes it looks like this:

[23:42:34] Compile-time hardening features are enabled
Cannot load libnvidia-encode.so.1
[23:42:34] vcn: not available on this system
[23:42:34] qsv: not available on this system
[23:42:34] hb_init: starting libhb thread
[23:42:34] thread 71a3b8246b30 started ("libhb")

Depending on the interplay between kernel versions, driver versions, and other factors, one of the two outputs above will appear. The exact mechanism remains unclear, and the result is the same regardless of whether the AMD container runtime is installed.

I believe you've done everything correctly, as the official Flatpak version of HandBrake produces the same output. I haven't tried the native version yet (as it requires manual compilation and I haven't found the time), so I haven't been able to verify if this is an issue specific to containerized environments.

Under the correct software and hardware environment, VCN should be invoked as expected. However, AMD's proprietary drivers on Linux are a mess and lack consistency, making it difficult to define exactly what constitutes a "correct software and hardware environment."

Fortunately, according to VAAPI Encoder (2nd branch/attempt) by sgothel · Pull Request #7467 · HandBrake/HandBrake, HandBrake developers seem to have decided to add VAAPI support in version 1.11. Once that happens, we won't have to deal with AMD's proprietary drivers anymore.

Finally, feel free to close this issue at any time : )