video: introduction of STM32 VENC driver for H264 hardware video compression #92884

hfruchet-st · 2025-07-09T08:59:36Z

This PR introduces H264 hardware video compression thanks to VENC peripheral of STM32N6.

A new stm32 video encoder driver has been introduced which relies on vc8000nanoe software stack [1].
This is a first basic porting to enable H264 encoding use-case.
RAM memory resources required by VENC encoding process are allocated in external PSRAM, camera frames
captured by DCMIPP and encoded by VENC are also allocated in external PSRAM.
The video encoder driver currently lack of controls to configure the encoding process, default configuration
targets H264 1080p 2Mb/s constant bitrate.

In order to test, the tcpserversink video sample has been enhanced to support video compression in order to stream small H264 compressed video chunks instead of big raw uncompressed images.

To build sample: (flash using ST-Link and boot with FSBL [2])

zephyr$> west build -p -b stm32n6570_dk//fsbl --shield st_b_cams_imx_mb1854 samples/drivers/video/tcpserversink

The default configuration allows to capture and stream 1080p camera content from STM32N6 which can be received, decoded and displayed by an host PC using a GStreamer command line:

host$> gst-launch-1.0 tcpclientsrc host=192.0.2.1 port=5000 ! queue ! decodebin ! queue ! fpsdisplaysink sync=false

Tests have been made using ethernet over USB between STM32N6 and host PC thanks to the temporary last 2 commits.
When using ethernet over USB, set the ethernet interface through USB before starting the GStreamer command line:

host$> sudo ifconfig enx00005e005301 192.0.2.2

To find the right interface, search for "enx" in kernel log

host$> dmesg | grep enx
host$> [1316045.847337] cdc_ether 3-6.3.2:1.0 **enx00005e005301**: renamed from eth0

[1] zephyrproject-rtos/hal_stm32#295
[2] https://docs.zephyrproject.org/latest/boards/st/stm32n6570_dk/doc/index.html

Video buffers being often quite big, it might not be possible to allocate them from the default heap. For that reason, rely on the mem sw attr heap allocator in order to design from where to allocate buffers using which attributes. Signed-off-by: Alain Volmat <[email protected]>

Add MEM_SW_ATTR on the PSRAM memory in order to have it become accessible via the mem_attr_heap framework. On this device since we need to allocate a rather large framebuffer, we can do that on the PSRAM memory. Signed-off-by: Alain Volmat <[email protected]>

Video encoder requires non-cacheable memory. Signed-off-by: Hugues Fruchet <[email protected]>

Add YUV420 semi-planar pixel format (NV12). Signed-off-by: Hugues Fruchet <[email protected]>

Addition of description for the STM32 Video encoder (VENC). Signed-off-by: Hugues Fruchet <[email protected]>

The STM32 video encoder (VENC) peripheral is a hardware accelerator allowing to compress RGB/YUV frames into H264 video bitstream chunks. Signed-off-by: Hugues Fruchet <[email protected]>

Add node describing the venc in stm32n6.dtsi Signed-off-by: Hugues Fruchet <[email protected]>

Enable video encoder on stm32n6570_dk discovery board. Signed-off-by: Hugues Fruchet <[email protected]>

Sync with video capture sample. Signed-off-by: Hugues Fruchet <[email protected]>

Add video compression support to lowerize network bandwidth. To visualise camera content on host PC, use GStreamer command line: $> gst-launch-1.0 tcpclientsrc host=<board ip address> port=5000 ! decodebin ! autovideosink sync=false Signed-off-by: Hugues Fruchet <[email protected]>

Add YUV420 semi-planar support (NV12). This is the video encoder prefered pixel format. Signed-off-by: Hugues Fruchet <[email protected]>

Allow to configure the number of allocated capture frames. This allows to make tradeof between framerate versus memory usage. 2 buffers allows to capture while sending data (optimal framerate). 1 buffer allows to reduce memory usage but capture framerate is lower. Signed-off-by: Hugues Fruchet <[email protected]>

Add configuration files for the stm32n6570_dk board. This enables streaming over ethernet of the images captured by MB1854 camera module compressed in 1920x1088 H264 video bitstream. Signed-off-by: Hugues Fruchet <[email protected]>

Ethernet over USB support. Signed-off-by: Hugues Fruchet <[email protected]>

Disable ethernet phy support to use ethernet over USB. Signed-off-by: Hugues Fruchet <[email protected]>

github-actions · 2025-07-09T09:01:11Z

The following west manifest projects have changed revision in this Pull Request:

Name	Old Revision	New Revision	Diff
zephyr-isp 🆕	None (Added)	master	N/A

⛔ DNM label due to: 1 added project and 1 unreachable repo

Note: This message is automatically posted and updated by the Manifest GitHub Action.

sonarqubecloud · 2025-07-09T09:24:45Z

Quality Gate passed

Issues
19 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
8.7% Duplication on New Code

See analysis details on SonarQube Cloud

mathieuchopstm · 2025-07-09T12:44:15Z

drivers/video/Kconfig

@@ -58,6 +58,20 @@ config VIDEO_I2C_RETRY_NUM
 	  The default is to not retry. Board configuration files or user project can then
 	  use the number of retries that matches their situation.

+config VIDEO_BUFFER_USE_MEM_ATTR_HEAP
+	bool "Use mem attr api for video buffer"


Suggested change

bool "Use mem attr api for video buffer"

bool "Allocate video buffer from mem_attr heap"

+ maybe this should depends on MEM_ATTR_HEAP

mathieuchopstm · 2025-07-09T12:47:46Z

drivers/video/Kconfig

+	default n
+
+config VIDEO_BUFFER_MEM_SW_ATTRIBUTE
+	int "Mem SW attribute for video buffer"


Suggested change

int "Mem SW attribute for video buffer"

int "Heap allocations attribute"

mathieuchopstm · 2025-07-09T12:53:59Z

drivers/video/Kconfig

+	  Mem SW attribute for video buffer:
+	  1: ATTR_SW_ALLOC_CACHE
+	  2: ATTR_SW_ALLOC_NON_CACHE
+	  4: ATTR_SW_ALLOC_DMA


It would be best to avoid duplicating information from the header.
Proposal:

Suggested change

Mem SW attribute for video buffer:

1: ATTR_SW_ALLOC_CACHE

2: ATTR_SW_ALLOC_NON_CACHE

4: ATTR_SW_ALLOC_DMA

Attribute to request when performing allocations from mem_attr heap.

Refer to include/zephyr/dt-bindings/memory-attr/memory-attr-sw.h for

the list of allowed values (one of ATTR_SW_*).

For example, if you want to use ATTR_SW_ALLOC_CACHE defined as:

#define ATTR_SW_ALLOC_CACHE BIT(0)

This symbol should be set equal to 1 (equal to (1 << 0)).

mathieuchopstm · 2025-07-09T12:55:36Z

drivers/video/video_common.c

+#define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout)                                              \
+	mem_attr_heap_aligned_alloc((CONFIG_VIDEO_BUFFER_MEM_SW_ATTRIBUTE <<	\
+				    DT_MEM_SW_ATTR_SHIFT), align, size)


Suggested change

#define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout) \

mem_attr_heap_aligned_alloc((CONFIG_VIDEO_BUFFER_MEM_SW_ATTRIBUTE << \

DT_MEM_SW_ATTR_SHIFT), align, size)

#define VIDEO_COMMON_HEAP_ALLOC(align, size, timeout) \

mem_attr_heap_aligned_alloc(DT_MEM_SW(CONFIG_VIDEO_BUFFER_MEM_SW_ATTRIBUTE), \

align, size)

mathieuchopstm · 2025-07-09T12:57:44Z

drivers/video/video_common.c

+#if defined(CONFIG_VIDEO_BUFFER_USE_MEM_ATTR_HEAP)
+	mem_attr_heap_pool_init();
+#endif


Nit: should check that return value is 0 or -EALREADY

mathieuchopstm · 2025-07-09T13:25:57Z

drivers/video/video_stm32_venc.c

+{
+	/* VENC HW REset */
+	__HAL_RCC_VENC_FORCE_RESET();
+	k_sleep(K_MSEC(1));


Maybe k_busy_wait with a lower delay (if upper bound for tRESET is known and <<< 1ms).

Although polling a "is IP under reset" bit would be better. (Ideally, we would even call reset_line_toggle with info from DT, but driver doesn't perform polling...)

mathieuchopstm · 2025-07-09T13:29:02Z

drivers/video/video_stm32_venc.c

+		if (ret)
+			return -1;


Suggested change

if (ret)

return -1;

if (ret) {

return -1;

}

or even ret < 0 and propagate ret

mathieuchopstm · 2025-07-09T13:30:06Z

drivers/video/video_stm32_venc.c

+	}
+	LOG_DBG("output=%p\n", output);
+
+	if (!(data->frame_nb % 30) || data->resync) {


If this 30 is framerate dependent, it should come from a macro.

mathieuchopstm · 2025-07-09T13:31:08Z

drivers/video/video_stm32_venc.c

+static int video_stm32_venc_set_stream(const struct device *dev, bool enable,
+				       enum video_buf_type type)
+{
+	struct video_stm32_venc_data *data = dev->data;


Suggested change

struct video_stm32_venc_data *data = dev->data;

ARG_UNUSED(type);

struct video_stm32_venc_data *data = dev->data;

mathieuchopstm · 2025-07-09T13:35:08Z

drivers/video/video_stm32_venc.c

+ISR_DIRECT_DECLARE(stm32_venc_isr)
+{
+	VENC_EWL_TypeDef *enc = &ewl_instance;
+	u32 hw_handshake_status = READ_BIT(VENC_REG(BASE_HEncInstantInput >> 2U), (1U << 29U));


No HW define for (1 << 29)? (Nit: should be 1U << 29)

josuah

Thank you for preparing VENC!

This will really make video over network on Zephyr interesting.

It seems like Zephyr lacks a few APIs from Linux (no H.264 format definition, no planar API for NV12...), so if anything feels like missing, let us know so that the missing API can be provided in parallel.

josuah · 2025-07-19T22:00:18Z

drivers/video/video_stm32_venc.c

+	input = k_fifo_get(&data->fifo_input, K_NO_WAIT);
+	if (!input)
+		return 0;
+	LOG_DBG("input=%p\n", input);


In the meantime that a better buffering model is worked on, the way to video_buffer_free() a buffer is by giving it back to the application which video_dequeue() it and reuse it in the next cycle, for instance via data->fifo_input_out (in addition to a data->fifo_input_in).

If encountering errors where running out of buffers hopefully this can help.

josuah · 2025-07-19T22:13:45Z

drivers/video/video_stm32_dcmipp.c

+			ret = HAL_DCMIPP_CSI_PIPE_SemiPlanarStart(&dcmipp->hdcmipp,
+								  pipe->id, DCMIPP_VIRTUAL_CHANNEL0,
+								  &spaddr, DCMIPP_MODE_CONTINUOUS);
+		} else {
+			ret = HAL_DCMIPP_CSI_PIPE_Start(&dcmipp->hdcmipp, pipe->id,
+							DCMIPP_VIRTUAL_CHANNEL0,
+							addr, DCMIPP_MODE_CONTINUOUS);


Is this manageable without Linux-style planar API? Otherwise we can introduce planar APIs to help with this.

josuah · 2025-07-19T22:31:09Z

drivers/video/video_stm32_venc.c

+	}
+
+	data->pixelformat = fmt->pixelformat;
+	data->pitch = fmt->pitch;


Following these changes, when the application sets the format, the driver sets the ->pitch field.

But for H.264 the pitch might as well be 0 as a convention for variable data size...

zephyr/include/zephyr/drivers/video.h

Lines 1638 to 1647 in 9219c81

case VIDEO_PIX_FMT_ARGB32:

case VIDEO_PIX_FMT_ABGR32:

case VIDEO_PIX_FMT_RGBA32:

case VIDEO_PIX_FMT_BGRA32:

return 32;

default:

/* Variable number of bits per pixel or unknown format */

return 0;

}

}

josuah · 2025-07-19T22:35:17Z

drivers/video/video_stm32_venc.c

+  i32 irq_cnt;
+} VENC_EWL_TypeDef;
+
+#define ALIGNED_SIZE(x) ((((x) + ALIGNMENT_INCR - 1UL) / ALIGNMENT_INCR) * ALIGNMENT_INCR)


Should it be ROUND_DOWN to avoid overflowing of a few bytes due to the rounding?

josuah · 2025-07-19T23:06:07Z

drivers/video/video_stm32_venc.c

+
+	output = k_fifo_get(&data->fifo_output_in, K_NO_WAIT);
+	if (!output) {
+		return 0;


At this line, the input buffer is dequeued and dropped by the driver.

There would need some scheme to synchronize both queue to make sure to have both an input and an output buffer poped from the FIFO at the same time.

One method is to turn it into a thread with K_FOREVER instead of K_NO_WAIT, and therefre, never return 0 as they are guaranteed to return something with K_FOREVER.

One method is to 'peek' at the FIFO without popping the buffer. If there is a buffer ready on both FIFO, then go ahead and pop them on both in confidence that none of them would be NULL (as it was checked beforehand)...

Maybe this does not cause an issue in the way the tcpserversink uses this driver though!

erwango · 2025-07-22T12:49:07Z

west.yml

+    - name: zephyr-isp
+      url: [email protected]:AIS/mg-zephyr-isp.git
+      revision: master
+      path: Lib/zephyr-isp
+      submodules: true


Impacts the whole project, should be removed.

Dependency should be documented at board or sample level

avolmat-st · 2025-08-05T07:14:49Z

include/zephyr/drivers/video.h

@@ -1597,6 +1599,7 @@ static inline unsigned int video_bits_per_pixel(uint32_t pixfmt)
 	case VIDEO_PIX_FMT_SGRBG12P:
 	case VIDEO_PIX_FMT_SRGGB12P:
 	case VIDEO_PIX_FMT_Y12P:
+	case VIDEO_PIX_FMT_NV12:


I've added the whole multi buffer support of DCMIPP as well as description in video.h in the following PR which is still currently in draft so that I can fully test it:

#94081

avolmat-st · 2025-08-05T07:15:21Z

drivers/video/video_stm32_dcmipp.c

 	ret = HAL_DCMIPP_PIPE_SetMemoryAddress(&dcmipp->hdcmipp, Pipe, DCMIPP_MEMORY_ADDRESS_0,
 					       (uint32_t)pipe->next->buffer);
 	if (ret != HAL_OK) {
 		LOG_ERR("Failed to update memory address");
 		return;
 	}
+


I've added the whole multi buffer support of DCMIPP as well as description in video.h in the following PR which is still currently in draft so that I can fully test it:

#94081

No specific processing is necessary currently as part of the stm32_dcmipp_isp_start function hence remove the assert(0) currently preventing stop of the isp. Signed-off-by: Alain Volmat <[email protected]>

Allow to call HAL_DCMIPP_PIPE_SetConfig when the pipe state is READY. Currently it is only possible to use this function if the pipe state is RESET and ERROR states. Indeed, it is possible to reconfigure the PIPE as soon as the pipe is not currently being used, aka in the READY state. With that done, it becomes possible to change the PIPE configuration between two use-cases without having to DeInit / Init the HAL_DCMIPP or use the unitary pixel packer functions. Signed-off-by: Alain Volmat <[email protected]>

hfruchet-st and others added 17 commits July 9, 2025 10:58

west: add zephyr-isp package

57454b2

boards: st: stm32n6570_dk: switch psram area to non-cacheable

51ca23b

Video encoder requires non-cacheable memory. Signed-off-by: Hugues Fruchet <[email protected]>

video: add nv12 pixel format support

4cea4fa

Add YUV420 semi-planar pixel format (NV12). Signed-off-by: Hugues Fruchet <[email protected]>

drivers: video: dcmipp: add nv12 pixel format support

4a1cde0

Add YUV420 semi-planar pixel format (NV12). Signed-off-by: Hugues Fruchet <[email protected]>

dts-bindings: video: addition of stm32 venc description

20d30f5

Addition of description for the STM32 Video encoder (VENC). Signed-off-by: Hugues Fruchet <[email protected]>

drivers: video: introduction of the stm32 venc driver

25ecf94

The STM32 video encoder (VENC) peripheral is a hardware accelerator allowing to compress RGB/YUV frames into H264 video bitstream chunks. Signed-off-by: Hugues Fruchet <[email protected]>

dts: arm: st: n6: add venc node

4419a21

Add node describing the venc in stm32n6.dtsi Signed-off-by: Hugues Fruchet <[email protected]>

boards: stm32: stm32n6570_dk: enable video encoder

ef412ab

Enable video encoder on stm32n6570_dk discovery board. Signed-off-by: Hugues Fruchet <[email protected]>

samples: video: tcpserversink: sync with capture sample

58daa7c

Sync with video capture sample. Signed-off-by: Hugues Fruchet <[email protected]>

samples: video: tcpserversink: nv12 support

75284f7

Add YUV420 semi-planar support (NV12). This is the video encoder prefered pixel format. Signed-off-by: Hugues Fruchet <[email protected]>

samples: video: tcpserversink: ethernet over usb

2878753

Ethernet over USB support. Signed-off-by: Hugues Fruchet <[email protected]>

boards: stm32: stm32n6570_dk: disable ethernet phy support

a32b1da

Disable ethernet phy support to use ethernet over USB. Signed-off-by: Hugues Fruchet <[email protected]>

github-actions bot added the manifest label Jul 9, 2025

avolmat-st self-requested a review July 9, 2025 11:12

mathieuchopstm reviewed Jul 9, 2025

View reviewed changes

josuah added the area: Video Video subsystem label Jul 19, 2025

josuah reviewed Jul 19, 2025

View reviewed changes

josuah added the platform: STM32 ST Micro STM32 label Jul 19, 2025

erwango reviewed Jul 24, 2025

View reviewed changes

avolmat-st reviewed Aug 5, 2025

View reviewed changes

hfruchet-st mentioned this pull request Aug 13, 2025

video: stm32: dcmipp: addition of (semi)planar format support #94081

Open

ngphibang self-requested a review August 14, 2025 06:29

	bool "Use mem attr api for video buffer"
	bool "Allocate video buffer from mem_attr heap"

	int "Mem SW attribute for video buffer"
	int "Heap allocations attribute"

-	  Mem SW attribute for video buffer:
-: ATTR_SW_ALLOC_CACHE
-: ATTR_SW_ALLOC_NON_CACHE
-: ATTR_SW_ALLOC_DMA
+	  Attribute to request when performing allocations from mem_attr heap.
+	  Refer to include/zephyr/dt-bindings/memory-attr/memory-attr-sw.h for
+	  the list of allowed values (one of ATTR_SW_*).
+	  For example, if you want to use ATTR_SW_ALLOC_CACHE defined as:
+	      #define ATTR_SW_ALLOC_CACHE    BIT(0)
+	  This symbol should be set equal to 1 (equal to (1 << 0)).

	case VIDEO_PIX_FMT_ARGB32:
	case VIDEO_PIX_FMT_ABGR32:
	case VIDEO_PIX_FMT_RGBA32:
	case VIDEO_PIX_FMT_BGRA32:
	return 32;
	default:
	/* Variable number of bits per pixel or unknown format */
	return 0;
	}
	}

video: introduction of STM32 VENC driver for H264 hardware video compression #92884

Are you sure you want to change the base?

video: introduction of STM32 VENC driver for H264 hardware video compression #92884

Conversation

hfruchet-st commented Jul 9, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

sonarqubecloud bot commented Jul 9, 2025

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josuah left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josuah Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josuah Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josuah Jul 19, 2025 •

edited

Loading

josuah Jul 19, 2025 •

edited

Loading