ESP GMF Video is a suite of video processing elements designed for a variety of tasks such as video encoding, and applying video effects. These elements can be combined to form a complete video processing pipeline.
The following table lists the currently supported video processing modules and their detailed information:
| Name | TAG | Function | Method | Input Port | Output Port | Hardware Acceleration | Dependent on Video Information |
|---|---|---|---|---|---|---|---|
| VIDEO_ENC | vid_enc | Video encoder: H264, MJPEG | set_bitrateset_dst_codecget_src_fmtspresetget_frame_size |
Single | Single | Yes | Yes |
| VIDEO_DEC | vid_dec | Video decoder: H264, MJPEG | set_dst_fmtset_src_codecget_dst_fmts |
Single | Single | No | No |
| VIDEO_PPA | vid_ppa | Pixel processing accelerator: Color conversion, Scaling, Cropping, Rotation |
set_dst_formatset_dst_resolutionset_rotationset_cropped_rgn |
Single | Single | Yes | Yes |
| FPS_CVT | vid_fps_cvt | Frame rate conversion | set_fps |
Single | Single | No | Yes |
| OVERLAY_MIXER | vid_overlay | Video overlay mixer | overlay_enableset_rgnset_portset_alpha |
Multiple | Single | No | Yes |
| COLOR_CVT | vid_color_cvt | Software color conversion | set_dst_fmt |
Single | Single | No | Yes |
| CROP | vid_crop | Software video cropping | set_crop_rgn |
Single | Single | No | Yes |
| SCALE | vid_scale | Software video scaling | set_dst_res |
Single | Single | No | Yes |
| ROTATE | vid_rotate | Software video rotation | set_angle |
Single | Single | No | Yes |
The Video Encoder element uses the esp_video_codec to convert raw video frames into compressed video streams. It currently supports two codecs:
- H264
- MJPEG
The video decoder element use the esp_video_codec to convert compressed video streams into raw video frames. It currently supports two codecs：
- H264
- MJPEG
The Video PPA element is a compact element, support multiple functions. It is currently available on the ESP32P4 board and includes following functionalities:
- Color Conversion:
The ESP32P4 supports color conversion through two hardware modules:
- 2D-DMA: Automatically selected for better efficiency if supported.
- PPA: Used as a fallback when 2D-DMA not supported
- Resizing: Resizing functionality is provided by the PPA module.
- Cropping: Cropping functionality is provided by the PPA module.
- Rotation: Supports rotations at 0°, 90°, 180°, and 270°.
This module adjusts the frame rate of the video. It decreases the input frame rate to a specified output rate, using the Presentation Time Stamp (PTS) embedded in the input data to accurately schedule frames.
The Video Overlay Mixer module allows users to overlay additional graphics onto a video frame. By receiving overlay data via a user-defined port, it can blend elements such as timestamps, watermarks, or other images into a designated region of the original video frame.
Following elements are wrapped for Video Pixel Processor which implemented software video processing.
Element to do software color conversion for video image
Element to do software video cropper for video image
Element to do software video scaler for video image
Element to do software video rotator for video image
The following table summarizes the support for ESP-GMF-Video elements across Espressif SoCs in the current release. A check mark (✔) indicates that the element is supported, while a cross mark (✖) indicates it is not supported.
| Element | ESP32 | ESP32-S2 | ESP32-S3 | ESP32-P4 |
|---|---|---|---|---|
| Video PPA | ✖ | ✖ | ✖ | ✔ |
| FPS Converter | ✔ | ✔ | ✔ | ✔ |
| Overlay Mixer | ✔ | ✔ | ✔ | ✔ |
| Video Decoder | MJPEG only | MJPEG only | ✔ | ✔ |
| Video Encoder | MJPEG only | MJPEG only | ✔ | ✔ |
| Video Pixel Processor | ✔ | ✔ | ✔ | ✔ |
- Video PPA is currently supported only on the ESP32-P4.
- FPS Converter and Overlay Mixer are supported on all SoCs.
- Video Decoder supports MJPEG decoding on ESP32 and ESP32-S2
- Video Encoder supports MJPEG encoding on ESP32 and ESP32-S2
ESP GMF Video modules are often used together to build a complete video processing pipeline. For example, you might first convert its colors or size, adjust the frame rate, and overlay and finally output through video encoder. For a practical implementation, please refer to the example in test_app.