hardware acceleration for Rockchip RK3588 mainline kernel support

[jimmyhon]

Rockchip BSP kernel

The current -rk docker images require the use of Rockchip's BSP kernel (5.10 or 6.1). They provide hardware acceleration by using their rkmpp library, and rknn library.

Mainline kernel

Support for that hardware is also getting into upstream mainline kernel.

• https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md

NPU

The RK3588 NPU can be used through as a TFLite Delegate by using Mesa's Teflon library with upcoming Rocket support in Mesa 25.3 and Kernel 6.18-rc1

• https://docs.mesa3d.org/teflon.html
• https://blog.tomeuvizoso.net/2025/07/rockchip-npu-update-6-we-are-in-mainline.html
• https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29698
• https://lore.kernel.org/linux-rockchip/20250721-6-10-rocket-v9-0-77ebd484941e@tomeuvizoso.net/
• https://web.git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip.git/log/?h=v6.18-armsoc/dts64
• The pull request for frigate support is Add Mesa Teflon as a TFLite detector #18310

Video Decoders

The RK3588 has two types of video decoders. The hantro_vpu has been upstream since kernel 6.12 for h.264.
The rkvdec (with H.264 and HEVC for resolutions of 1080p+) is being added in Kernel 7.0. https://lore.kernel.org/linux-rockchip/20260120222018.404741-1-detlev.casanova@collabora.com/
https://lore.kernel.org/linux-media/379e2cfb0ac6640936446b5e76dd24f854e7800c.camel@collabora.com/
Both of these hardware accelerators can be accessed by the v4l2-request API. FFmpeg support for the v4l2-request has been submitted, but the latest development version can be found at
https://code.ffmpeg.org/Kwiboo/FFmpeg/commits/branch/v4l2request-v3
https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20847
Adding more presets to Frigate https://github.com/jimmyhon/frigate/tree/v4l2request #22102
There are test binaries for ffmpeg at https://github.com/jimmyhon/frigate-builds/releases/tag/ffmpeg-8.0.1

To configure ffmpeg to build with the v4l2request hwaccel, the system will need to include libudev and have linux-libc-dev from a linux-6.1. The statically linked ffmpeg builder currently defaults to target linux 4.18, and does not build libudev yet. https://github.com/BtbN/FFmpeg-Builds

Configuration

config.yaml

detectors:
  npu:
    type: teflon_tfl

ffmpeg:
  hwaccel_args: preset-v4l2request

docker-compose.yml

devices:
      - /dev/dri/:/dev/dri/ # accel
      - /dev/accel/:/dev/accel/ # rocket
      - /dev/media0 # rkvdec
      - /dev/video1 # rkvdec

On the RK3588, the hantro_vpu has issues with can cause ffmpeg to crash. Use rkvdec for video decoding.

$ v4l2-ctl -A -z platform:rkvdec
/dev/media0
/dev/video1

ℹ️ I will try to keep this first post updated with current status

[jimmyhon]

Does ffmpeg have the ability to use the Rockchip RGA2 from the mainline kernel? The support has been in the kernel since 6.12, and it's the same IP that's used by the rk3288 and rk3568.
https://lore.kernel.org/linux-rockchip/20240604163408.1863080-1-liujianfeng1994@gmail.com/

It might be able to used by the scale_v4l2m2m
https://github.com/jc-kynesim/rpi-ffmpeg/blob/test/8.0.1/main/libavfilter/vf_deinterlace_v4l2m2m.c
Will need modification to work on IP besides the bcm2835-codec-isp.

[jimmyhon]

Would it be okay to include a non-upstream ffmpeg build, similar to how the -rk images ships with https://github.com/nyanmisaka/ffmpeg-rockchip via https://github.com/MarcA711/Rockchip-FFmpeg-Builds?

[jimmyhon]

Hacky Development

For my current proof of concept testing, I didn't bother trying to create a statically linked ffmpeg build with v4l2request hwaccel yet. I just built a dynamically linked ffmpeg on debian bookworm (to match frigate 0.16-beta2).

Use a debian bookworm docker image, and compile the mesa rocket development branch https://gitlab.freedesktop.org/tomeu/mesa/-/tree/rocket
to get the libteflon.so

Use a debian bookworm docker image, and compile the ffmpeg v4l2request development branch
https://github.com/jernejsk/FFmpeg/commits/v4l2-request-n7.1/
to get ffmpeg and ffprobe

Create a new image from frigate:0.16.0-beta2-standard-arm64
add another frigate/detectors/plugins by copying the edgetpu_tfl.py and make it load the /usr/lib/teflon/libteflon.so instead.
add hwaccel preset in frigate/ffmpeg_presets for v4l2request
install debian bookworm's ffmpeg, so all the dependent libraries are installed.
Dump the dynamically linked ffmpeg and ffprobe in to /usr/lib/ffmpeg/7.0/bin and mask the statically linked ones from the 0.16.0-beta-standard-arm64n layer.

[NickM-27]

Frigate already supports running custom ffmpeg builds (statically linked). I believe there would need to be a marked benefit to creating a second distribution for the rockchip hardware, as this would be potentially confusing for users and more support burden depending on how it is implemented.

[jimmyhon]

I concur. Only people who don't want to use the Rockchip BSP 6.1 kernel, and want to use newer kernel features would want a second option to choose from.

My hope is that we don't need another docker image to choose from. All this stuff will become part of upstream projects and users will be able to get hardware acceleration from the -standard-arm64 image when using a mainline kernel.

None of these additions to frigate are rockchip/rk3588 specific. mesa-teflon-delegate started with the vivante NPU in the Amlogic A311D. There are other v4l2request hardware accelerators from AllWinner too. https://wiki.pine64.org/wiki/Mainline_Hardware_Decoding

[tilk]

For the record, I am a person who installed Frigate on a RK3588 machine with a mainline kernel (I run unmodified Debian Jessie) and found out the hard way that I won't get hardware decoding that way. Hope that upstream ffmpeg will work with that soon.

[jimmyhon]

Slightly less Hacky

Starting with Frigate 0.17 beta, there is support for teflon. However, the libteflon.so TFLite delegate in bookworm-backports is only 25.0 which only support etnaviv.

Mesa 25.3's teflon has support for rocket (Rockchip NPU) and ethos (ARM NPU). You can get a manually compiled 25.3.x libteflon.so for bookworm from https://github.com/jimmyhon/frigate-builds/releases.

In the docker-compose.yaml, you can override the existing libteflon.so by adding it to the volumes

volumes:
    - ./libteflon.so:/usr/lib/teflon/libteflon.so:ro

In the docker-compose.yaml, you can override the existing ffmpeg_presets.py (#22102) by adding it to the volumes

volumes:
    - ./ffmpeg_presets.py:/opt/frigate/frigate/ffmpeg_presets.py:ro

In the config.yaml, you can override the FFmpeg used. Now frigate will use the binaries placed at /config/custom_ffmpeg/bin/
https://github.com/jimmyhon/frigate-builds/releases/tag/ffmpeg-8.0.1

ffmpeg:
  path: /config/custom_ffmpeg
  hwaccel_args: preset-v4l2request

...so you can use a stock frigate:0.17.0-standard-arm64 image.

[jsalatiel]

Slightly less Hacky

Starting with Frigate 0.17 beta, there is support for teflon. However, the libteflon.so TFLite delegate in bookworm-backports is only 25.0 which only support etnaviv.

Mesa 25.3's teflon has support for rocket (Rockchip NPU) and ethos (ARM NPU). You can get a manually compiled 25.3.x libteflon.so for bookworm from https://github.com/jimmyhon/frigate-builds/releases.

In the docker-compose.yaml, you can override the existing libteflon.so by adding it to the volumes

volumes:
    - ./libteflon.so:/usr/lib/teflon/libteflon.so:ro

@jimmyhon Would you mind to share which base container are you using for frigate 0.17 ?
ghcr.io/blakeblackshear/frigate:0.17.0-beta1-standard-arm64?
ghcr.io/blakeblackshear/frigate:0.17.0-beta1-rk?
other ?

[jimmyhon]

Using the ghcr.io/blakeblackshear/frigate:0.17.0-beta1-standard-arm64 is enough.

The -rk images add drivers that rely on the vendor/BSP kernel.

[jsalatiel]

@jimmyhon I can see that you have
devices:
- /dev/dri/:/dev/dri/ # accel
- /dev/accel/:/dev/accel/ # rocket
but I do not see anything in /dev/accel

ls -la /dev/dri/ /dev/accel* ls: cannot access '/dev/accel*': No such file or directory /dev/dri/: total 0 drwxr-xr-x 3 root root 80 Jan 19 16:00 . drwxr-xr-x 19 root root 3840 Jan 19 16:39 .. drwxr-xr-x 2 root root 60 Jan 19 16:00 by-path crw-rw---- 1 root video 226, 0 Jan 19 16:00 card0
I am using kernel 6.18.0

My frigate instance just dies:

2026-01-21 05:46:46.033224456  Couldn't open kernel device
2026-01-21 05:46:46.034066778  [2026-01-21 05:46:46] frigate.detectors.detector_utils ERROR   : No NPU was detected. If you do not have a TFLite device yet, you must configure CPU detectors.
2026-01-21 05:46:46.040539649  tflite_plugin_destroy_delegate: NULL delegate!
2026-01-21 05:46:46.040842687  Process frigate.detector:npu:
2026-01-21 05:46:46.040845603  Traceback (most recent call last):
2026-01-21 05:46:46.040847353    File "/usr/local/lib/python3.11/dist-packages/tflite_runtime/interpreter.py", line 166, in load_delegate
2026-01-21 05:46:46.040852312      delegate = Delegate(library, options)
2026-01-21 05:46:46.040853770                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-21 05:46:46.040855228    File "/usr/local/lib/python3.11/dist-packages/tflite_runtime/interpreter.py", line 104, in __init__
2026-01-21 05:46:46.040882353      raise ValueError(capture.message)
2026-01-21 05:46:46.040883228  ValueError
2026-01-21 05:46:46.040884103  
2026-01-21 05:46:46.040885269  During handling of the above exception, another exception occurred:
2026-01-21 05:46:46.040886144  
2026-01-21 05:46:46.040887019  Traceback (most recent call last):
2026-01-21 05:46:46.040888186    File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2026-01-21 05:46:46.040908894      self.run()
2026-01-21 05:46:46.040910352    File "/opt/frigate/frigate/object_detection/base.py", line 143, in run
2026-01-21 05:46:46.040911811      object_detector = LocalObjectDetector(detector_config=self.detector_config)
2026-01-21 05:46:46.040913269                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-21 05:46:46.040924936    File "/opt/frigate/frigate/object_detection/base.py", line 62, in __init__
2026-01-21 05:46:46.040926102      self.detect_api = create_detector(detector_config)
2026-01-21 05:46:46.040926977                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-21 05:46:46.040928436    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2026-01-21 05:46:46.040929019      return api(detector_config)
2026-01-21 05:46:46.040940102             ^^^^^^^^^^^^^^^^^^^^
2026-01-21 05:46:46.040941269    File "/opt/frigate/frigate/detectors/plugins/teflon_tfl.py", line 32, in __init__
2026-01-21 05:46:46.040942435      interpreter = tflite_load_delegate_interpreter(
2026-01-21 05:46:46.040943310                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-21 05:46:46.040944769    File "/opt/frigate/frigate/detectors/detector_utils.py", line 55, in tflite_load_delegate_interpreter
2026-01-21 05:46:46.040945935      tf_delegate = load_delegate(delegate_library, device_config)
2026-01-21 05:46:46.040947394                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-21 05:46:46.040948852    File "/usr/local/lib/python3.11/dist-packages/tflite_runtime/interpreter.py", line 168, in load_delegate
2026-01-21 05:46:46.040950019      raise ValueError('Failed to load delegate from {}\n{}'.format(
2026-01-21 05:46:46.040964310  ValueError: Failed to load delegate from /usr/lib/teflon/libteflon.so

Any ideas ?

[jimmyhon]

Oh, which board do you have?

To get the device nodes in /dev/accel, you'll need to have the rocket kernel module loaded. The kernel config needs to have CONFIG_DRM_ACCEL_ROCKET enabled.

in dmesg, you'd see something similar to

$ dmesg | grep rocket
[    5.510569] [drm] Initialized rocket 0.0.0 for rknn on minor 0
[    5.511024] rocket fdab0000.npu: Rockchip NPU core 0 version: 1179210309
[    5.512751] rocket fdac0000.npu: Rockchip NPU core 1 version: 1179210309
[    5.513487] rocket fdad0000.npu: Rockchip NPU core 2 version: 1179210309

Looking at the kernel device tree log
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/arch/arm64/boot/dts/rockchip?h=v6.18
The following boards had the NPU enabled in their device trees in 6.18

• Radxa Rock 5B
• QuartzPro64
• Orange Pi 5, 5B, 5 Plus, 5 Max, 5 Ultra
• NanoPi R6

and in 6.19 (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/arch/arm64/boot/dts/rockchip?h=v6.19-rc1):

• Gameforce Ace
• Indiedroid Nova

and in 6.20 (https://git.kernel.org/pub/scm/linux/kernel/git/mmind/linux-rockchip.git/log/?h=v6.20-armsoc/dts64):

• NanoPC T6
• FriendlyELEC CM3588
• Turing RK1
• ArmSoM Tiger
• ArmSoM Jaguar

[jsalatiel]

Thank you!
I have an orange pi 5 pro with my own 6.18 kernel. I missed that module during compilation.
Thanks! It is working great now!

[rolsch]

Radxa Rock 5B 16Gb.

• what is the prefered OS and config settings for frigate?

Actual i use armbian:

v25.11.2 for Rock 5B running Armbian Linux 6.1.115-vendor-rk35xx
Packages: Debian stable (trixie)

root@rock5b01:# ls -la /dev/dri/ /dev/accel*
ls: cannot access '/dev/accel*': No such file or directory
/dev/dri/:
total 0
drwxr-xr-x 3 root root 140 Feb 4 20:07 .
drwxr-xr-x 20 root root 3900 Feb 4 20:07 ..
drwxr-xr-x 2 root root 120 Feb 4 20:07 by-path
crw-rw---- 1 root video 226, 0 Feb 4 20:07 card0
crw-rw---- 1 root video 226, 1 Feb 4 20:07 card1
crw-rw---- 1 root render 226, 128 Feb 4 20:07 renderD128
crw-rw---- 1 root render 226, 129 Feb 4 20:07 renderD129

root@rock5b01:# dmesg | grep rocket
root@rock5b01:#

[jimmyhon]

Oh, you're using the Rockchip BSP kernel. So this discussion won't apply to you. You'll want to use the -rk docker images as mentioned at the beginning of the discussion.

You'll want to configure frigate following

• https://docs.frigate.video/configuration/object_detectors#rockchip-platform
• https://docs.frigate.video/configuration/hardware_acceleration_video#rockchip-platform

[flappyjet]

I'm a hobbyist working on a script to detect the NPU environment for ARM64 edge devices. Since I only have limited hardware, I’m looking for volunteers to run a quick test and see what is the environment on different NPU-equipped boards.

This is a personal experimental project with no guarantees, just a survey to see what works and what doesn't.

GitHub Repository: npu-toolbox
Test Script: curl -sSL https://github.com/flappyjet/npu-toolbox/raw/refs/heads/main/scripts/npu_probe.sh | bash

Report Results: Please drop a comment in this survey issue: flappyjet/npu-toolbox#1

If you run the script, you will get a toml file like this

[platform]
model=xxxx
soc_id=yyyy
arch="aarch64"
kernel="6.12.59-ophub"

[modules]
etanviv="/lib/modules/6.12.59-ophub/kernel/drivers/gpu/drm/etnaviv/etnaviv.ko"
galcore=""
rocket="/lib/modules/6.12.59-ophub/kernel/drivers/accel/rocket/rocket.ko"

[devices]
npu_device="/dev/dri/renderD128" # Driver: etnaviv

[libraries]
delegate="/usr/local/lib/aarch64-linux-gnu/libteflon.so"

Post updated: support NPU benchmark by npu-toolbox benchmark --delegate DELEGATE

[nyok92]

Hi @jimmyhon !
Thanks for all these info.
I managed to setup mainline npu with frigate 0.17.0 rc2 standard arm64.
Now I want to try H264/hevc video decode/encode through rk3588, what do you advise ?
Is there a static ffmpeg binary with v4l2-request to download somewhere ?
is there a docker image with your presets included somewhere too to try ?
Thanks ! 🙏

[jimmyhon]

h264 and hevc decoding for rk3588's rkvdec is going into mainline in 7.0, so for the time being, I've been testing using collabora's https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/tree/rockchip-v6.19 branch

I haven't tried testing lately with a ffmpeg with v4l2-request support in frigate because there's no support for using the rga2 in ffmpeg. So there's still a lot of CPU usage when converting to yuv420p format to be used for object detection.

However, I have used LibreELEC 13's development branch with ffmpeg 8 and kodi 22 patches to get rk3588 hardware acceleration for use in Kodi.
https://forum.libreelec.tv/thread/29953-le13-testing-for-rk3288-rk3328-rk3399-rk3566-rk3568-rk3576-rk3588/

[jimmyhon]

Feels like the v4l2request uAPI is getting more stable now.

I was able to use h264 decoding with hardware acceleration. On a 1080p h264 stream, without hwaccel the CPU is 15%, with hwaccel the CPU drops to 8%.

I made ffmpeg/ffprobe binaries at https://github.com/jimmyhon/frigate-builds/releases/tag/ffmpeg-8.0.1
I added a preset in #22102

#22102 (comment)

Drop the ffmpeg binaries to /config/custom_ffmpeg/bin

in the config.yaml

ffmpeg:
  path: /config/custom_ffmpeg
  hwaccel_args: preset-v4l2request

also, add the devices to docker-compose.yaml

$ v4l2-ctl -A -z platform:rkvdec
/dev/media0
/dev/video1

    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/accel/accel0:/dev/accel/accel0
      - /dev/video1:/dev/video1
      - /dev/media0:/dev/media0

Note 1: /dev/video0 in my case is the for platform:rga
Note 2: I purposely left out the device nodes that comes from hantro_vpu. The h264 decoder has trouble with simultaneous streams.

I am testing using a kernel built from https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/tree/rockchip-v6.19
But now that 7.0-rc1 has been released, it should have everything needed.

I haven't tested with HEVC streams yet.

[el-bakkali]

this is really good news, im glad to see we can use mainline kernel with our rockchip rk3588
all the stuff is soon to be supported if not majority already
way better than 6.1 BSP