Video Module

Supported ICs: [RTL8735C]

Overview

The video_module is the AP-side interface to the video pipeline. Camera, ISP, and encoder control are handled by the VP firmware; applications configure streams through video_params_t and then connect the output to AP-side MMF modules.

../../_images/multimedia_pro_mmf_video_pipeline_ap_vp.drawio.svg — AP / VP video pipeline and runtime control path.

video_params_t video_params = {
    .stream_id = V1_STREAM_ID,
    .format    = VIDEO_H264,
    .width     = 1920,
    .height    = 1080,
    .fps       = 30,
    .gop       = 30,
    .bps       = 2000000,
    .rc_mode   = ENC_VBR,
    .min_qp    = 26,
    .max_qp    = 48,
    .qlevel    = 5,
    .ainr_en   = 0,
};

mm_context_t *video_ctx = mm_module_open(&video_module);
mm_module_ctrl(video_ctx, CMD_VIDEO_SET_PARAMS, (int)&video_params);
mm_module_ctrl(video_ctx, MM_CMD_SET_QUEUE_LEN, 60);
mm_module_ctrl(video_ctx, MM_CMD_INIT_QUEUE_ITEMS, MMQI_FLAG_DYNAMIC);
mm_module_ctrl(video_ctx, CMD_VIDEO_APPLY, 0);

stream_id

The ISP supports up to 5 simultaneous output channels. Each video_module instance occupies one channel.

Constant	Value	Typical use
`V1_STREAM_ID`	0	High-res encoded stream (recording / streaming)
`V2_STREAM_ID`	1	Mid-res encoded stream (live streaming)
`V3_STREAM_ID`	2	Low-fps encoded stream (JPEG snapshot)
`V4_STREAM_ID`	3	Raw NV12 output (motion detection)
`V5_STREAM_ID`	4	Raw RGB output (NN inference input)

Note

The hardware encoder supports only the first 3 channels (V1-V3). V4 and V5 output raw (unencoded) ISP frames; gop, bps, rc_mode, min_qp, and max_qp have no effect on these channels.

format

Selects the output format of the channel.

Constant	Value	Description
`VIDEO_HEVC`	1	H.265 / HEVC encoded bitstream
`VIDEO_H264`	2	H.264 / AVC encoded bitstream
`VIDEO_JPEG`	4	JPEG encoded frame
`VIDEO_NV12`	8	Raw YUV 4:2:0 semi-planar (NV12)
`VIDEO_RGB`	16	Raw RGB888 (used for NN input on V5)
`VIDEO_HEVC_JPEG`	5	Dual output: HEVC + JPEG simultaneously
`VIDEO_H264_JPEG`	6	Dual output: H.264 + JPEG simultaneously

fps / gop / bps

fps: Target frame rate of the channel output.
gop: I-frame interval in frames. A common setting is to match fps (one I-frame per second).
bps: Target bitrate in bits per second for encoded streams. Has no effect for raw output (NV12, RGB).

rc_mode

Rate control mode for the encoder.

Constant	Value	Description
`ENC_CVBR`	0	Constrained VBR - bitrate bounded by `bps`
`ENC_CBR`	1	Constant bitrate
`ENC_VBR`	2	Variable bitrate - quality-first, `bps` is a guide
`ENC_ABR`	3	Average bitrate
`ENC_CRF`	4	Constant rate factor - fixed perceptual quality
`ENC_CQP`	5	Constant QP - fixed quantisation parameter

min_qp / max_qp

Quantisation parameter bounds used by VBR/CVBR rate control to limit quality variation. Lower QP = higher quality, larger frame size. Typical values: min_qp = 26, max_qp = 48.

qlevel

JPEG quality level (0-9). Only applies when format includes VIDEO_JPEG. Higher value = better quality.

ainr_en

Enables AI Noise Reduction (AINR). Set to 1 to enable, 0 to disable. AINR is a processing module inserted between the image sensor and the ISP, so it operates on the raw sensor data before any channel split. Enabling it on any one channel activates AINR globally for all active channels. AINR improves image quality in low-light conditions but increases processing load.

video_params.ainr_en = 1;  /* enable AI noise reduction (applies to all channels) */

Encoder Parameter Configuration Timing

For AP applications, before encoder open means before CMD_VIDEO_APPLY is called. The usual sequence is:

Fill video_params_t and any other init-time stream parameters.
Send them with CMD_VIDEO_SET_PARAMS.
Configure the MMF queue.
Call CMD_VIDEO_APPLY. This asks VP firmware to create the stream and open the encoder when the channel uses H.264, H.265, or JPEG.

After CMD_VIDEO_APPLY and stream start, keep runtime changes to controls that the AP/VP path and encoder driver can queue safely. If a running stream must change codec, resolution, profile, VUI/header, or HRD behavior, stop the stream and open it again with the new init-time configuration.

Common customer requests

Request	Supported timing	What to do
Change codec / output format	Init-time only	Set `format` before `CMD_VIDEO_APPLY`. If the stream is already running, stop and reopen it.
Change width / height	Init-time only	Set `width` and `height` before `CMD_VIDEO_APPLY` because these values decide buffer geometry and encoder instance setup.
Change initial frame rate	Init-time configuration	Set `fps` before `CMD_VIDEO_APPLY` as the stream baseline.
Tune bitrate while streaming	Runtime supported	Use the runtime bitrate control path. In the joint test example, use `VAPP bps <ch> <bps>`.
Tune QP range while streaming	Runtime supported	Use the runtime QP control path. In the joint test example, use `VAPP qp <ch> <min_qp> <max_qp>`.
Change GOP while streaming	Runtime supported	Use the runtime GOP control path. In the joint test example, use `VAPP gop <ch> <gop>`.
Request an I-frame	Runtime supported	Use `CMD_VIDEO_FORCE_IFRAME` from AP code or `VAPP i <ch>` in the joint test example.
Change profile / level / VUI / SEI / HRD	Init-time only	Configure before stream start. Changing these mid-stream can confuse the decoder, cause playback artifacts, or require the client to reconnect.
Tune slice / deblock / CIR / ROI	Runtime supported by the encoder driver	Use only AP helpers that are explicitly provided by the SDK for these controls. They map to the queued coding-control path.
Enable encoder rotation / mirror	Not supported	Encoder preprocess rotation/mirror is not enabled in RTL configuration, so no AP API is provided.

CMD_VIDEO_* Parameter Reference

The tables below map every encoder-related parameter to its configuration timing and the exact AP-side command or helper to use.

Init-time only — configure in video_params_t (or via video_params_set_* helpers) before CMD_VIDEO_APPLY. These enter the encoder open sequence (VCEncInit(), SPS/PPS/VUI header, profile/level checks, or HRD model) and require a stream stop-and-reopen to change.

Parameters in this group are set in three different ways:

Direct field assignment — for basic fields like stream_id, format, width, height. Assign directly in the video_params_t struct initializer.
Helper functions (video_params_set_*) — for profile/level/tier, CABAC, SAR, and driver SEI. These set the internal flags automatically so you do not need to touch rc_ext.flags or coding.flags yourself.
rc_ext / coding flags — for advanced options not covered by a helper. You must set the flag bit and the value together. The flag tells VP that the field has been intentionally set — a value of 0 is still meaningful (e.g., pictureRc = 0 explicitly disables picture RC), so VP ignores any field whose flag is not set.

video_params_t video_params = {
    .stream_id = V1_STREAM_ID,
    .format    = VIDEO_H264,
    .width     = 1920,
    .height    = 1080,
    .fps       = 30,
    .gop       = 30,
    .bps       = 2000000,
    .rc_mode   = ENC_VBR,
};

/* Helper: sets profile, level, and internal flags automatically */
video_params_set_profile_level(&video_params, ENC_H264_MAIN, 41);
video_params_set_cabac(&video_params, 1);        /* enable CABAC */
video_params_set_driver_sei(&video_params, 1);   /* picture timing SEI */

/* rc_ext flags: must set flag AND value — VP ignores fields without flag */
video_params.rc_ext.flags |= VPAP_ENC_RC_EXT_HRD;
video_params.rc_ext.hrd = 1;

/* coding flags: same pattern */
video_params.coding.flags |= VPAP_ENC_CODING_FULL_RANGE;
video_params.coding.vuiVideoFullRange = 1;

mm_module_ctrl(video_ctx, CMD_VIDEO_SET_PARAMS, (int)&video_params);
mm_module_ctrl(video_ctx, CMD_VIDEO_APPLY, 0);

Parameter	`video_params_t` field / helper	Description
Stream channel	`stream_id`	V1–V5 channel. Determines the ISP output port and encoder instance.
Output format / codec	`format`	`VIDEO_H264` / `VIDEO_HEVC` / `VIDEO_JPEG` / `VIDEO_NV12` / `VIDEO_RGB`. Raw formats bypass the encoder.
Resolution	`width` / `height`	Sets ISP crop and encoder buffer geometry. Requires reopen to change.
JPEG quality level	`qlevel`	JPEG encoding quality (0–9). No effect on H.264/H.265.
Profile / Level	`video_params_set_profile_level()`	H.264/H.265 profile and level. Affects SPS signaling and decoder compatibility.
Tier	`video_params_set_tier()`	H.265 Main / High tier. Affects maximum bitrate limits in level definitions.
HRD conformance	`rc_ext.hrd` (flag: `VPAP_ENC_RC_EXT_HRD`)	Enable HRD bitstream timing model. Conflicts with `vbr=1`.
CPB buffer / max rate	`rc_ext.hrdCpbSize` / `cpbMaxRate` (flag: `VPAP_ENC_RC_EXT_CPB`)	CPB buffer size in bits and max CPB bitrate. `0` uses the level default.
Filler data	`rc_ext.fillerData` (flag: `VPAP_ENC_RC_EXT_FILLER_DATA`)	Insert filler NAL units to pad to target bitrate. Requires both `hrd` and `pictureRc` enabled.
VUI full range	`coding.vuiVideoFullRange` (flag: `VPAP_ENC_CODING_FULL_RANGE`)	Full-range or limited-range signal in VUI. Affects decoder luma/chroma interpretation.
HEVC SAO	`coding.enableSao` (flag: `VPAP_ENC_CODING_SAO`)	Enable HEVC Sample Adaptive Offset filter. Requires HW SAO support.
H.264 CABAC	`video_params_set_cabac()`	Enable CABAC entropy coding. H.264 Baseline profile does not support CABAC.
Sample aspect ratio	`video_params_set_sar()`	Non-square pixel SAR in VUI. Omit for standard square-pixel content.
Driver SEI	`video_params_set_driver_sei()`	Enable driver-generated picture timing and buffering period SEI messages.

Init + Runtime — may be set in video_params_t before CMD_VIDEO_APPLY as the initial value, and updated after stream start using the listed CMD_VIDEO_* command via mm_module_ctrl().

For runtime updates, scalar parameters pass the new value directly as the third argument. Parameters that take a struct pass a pointer cast to int:

/* Scalar: pass value directly */
mm_module_ctrl(video_ctx, CMD_VIDEO_BITRATE, 1000000);
mm_module_ctrl(video_ctx, CMD_VIDEO_GOP,     60);
mm_module_ctrl(video_ctx, CMD_VIDEO_FPS,     15);
mm_module_ctrl(video_ctx, CMD_VIDEO_RCMODE,  ENC_CBR);

/* Struct: fill a local struct and pass its address cast to int */
video_qp_range_t qp = { .min_qp = 28, .max_qp = 45 };
mm_module_ctrl(video_ctx, CMD_VIDEO_SET_QP, (int)&qp);

video_moving_bitrate_t moving = { .tolerance = 200, .frames = 30 };
mm_module_ctrl(video_ctx, CMD_VIDEO_MOVING_BITRATE, (int)&moving);

video_deblock_t deblock = { .disable = 0, .tc_offset = 0, .beta_offset = 0 };
mm_module_ctrl(video_ctx, CMD_VIDEO_DEBLOCK, (int)&deblock);

Init field	Runtime CMD	API function	Description
`fps`	`CMD_VIDEO_FPS`	`video_set_fps()`	Output frame rate.
`gop`	`CMD_VIDEO_GOP`	`video_set_gop()`	I-frame interval in frames.
`bps`	`CMD_VIDEO_BITRATE`	`video_set_bitrate()`	Target bitrate in bps.
`rc_mode`	`CMD_VIDEO_RCMODE`	`video_set_rcmode()`	Rate control mode (CVBR / CBR / VBR / ABR / CRF / CQP).
`min_qp`	`CMD_VIDEO_MINQP`	`video_set_minqp()`	Global QP lower bound.
`max_qp`	`CMD_VIDEO_MAXQP`	`video_set_maxqp()`	Global QP upper bound.
`min_qp` + `max_qp`	`CMD_VIDEO_SET_QP`	`video_set_qp()`	Set both global min and max QP in one call.
`ainr_en`	`CMD_VIDEO_AINR`	`video_ainr_enable()`	AI Noise Reduction. Configured on V1; applies globally to all channels.
`rc_ext.qpHdr`	`CMD_VIDEO_QP_HDR`	`video_set_qp_hdr()`	Initial picture QP (`-1` lets RC choose automatically).
`rc_ext.minIQp` / `maxIQp`	`CMD_VIDEO_SET_I_QP`	`video_set_i_qp()`	I-frame QP range. Overrides global min/max QP for intra frames.
`rc_ext.intraQpDelta`	`CMD_VIDEO_INTRA_QP_DELTA`	`video_set_intra_qp_delta()`	QP offset applied to intra frames relative to inter (−51 to +51).
`rc_ext.fixedIntraQp`	`CMD_VIDEO_FIXED_INTRA_QP`	`video_set_fixed_intra_qp()`	Pin intra QP to a fixed value. `0` disables fixed intra QP.
`rc_ext.pictureRc`	`CMD_VIDEO_PIC_RC`	`video_set_pic_rc()`	Enable picture-level rate control.
`rc_ext.ctbRc`	`CMD_VIDEO_CTB_RC`	`video_set_ctb_rc()`	CTB-level RC mode (0=off / 1=subjective / 2=precise / 3=mixed).
`rc_ext.blockRCSize`	`CMD_VIDEO_BLOCK_RC_SIZE`	`video_set_block_rc_size()`	CTB RC granularity (0=64×64 / 1=32×32 / 2=16×16).
`rc_ext.pictureSkip`	`CMD_VIDEO_PICTURE_SKIP`	`video_set_picture_skip()`	Allow RC to skip frames to stay within bitrate target.
`rc_ext.bitrateWindow`	`CMD_VIDEO_BITRATE_WINDOW`	`video_set_bitrate_window()`	RC averaging window (1–300 frames).
`rc_ext.tolMovingBitRate` / `monitorFrames`	`CMD_VIDEO_MOVING_BITRATE`	`video_set_moving_bitrate()`	Moving bitrate tolerance (0–2000 %) and monitoring window (10–120 frames).
`rc_ext.crf`	`CMD_VIDEO_CRF`	`video_set_crf()`	Constant Rate Factor (`-1` disables CRF).
`rc_ext.vbr`	`CMD_VIDEO_VBR`	`video_set_vbr()`	VBR-style quality control. Cannot be enabled together with `hrd=1`.
`coding.sliceSize`	`CMD_VIDEO_SLICE_SIZE`	`video_set_slice_size()`	Slice height in CTB rows. `0` disables slice splitting.
`coding.seiUserData`	`CMD_VIDEO_SEI_USER_DATA`	`video_set_sei_user_data()`	Insert user_data_unregistered SEI before each encoded frame.
`coding.disableDeblockingFilter` + offsets	`CMD_VIDEO_DEBLOCK`	`video_set_deblock()`	Deblocking filter on/off and tc/beta offsets (−6 to +6).

Runtime only — available only after CMD_VIDEO_APPLY. These have no init-time equivalent in the open command.

CMD	API function	Description
`CMD_VIDEO_FORCE_IFRAME`	`video_force_iframe()`	Force the next encoded frame to be an I-frame.
`CMD_VIDEO_CIR`	`video_set_cir()`	Cyclic Intra Refresh: continuously refresh a rotating band of CTBs instead of a full I-frame.
`CMD_VIDEO_ROI`	`video_set_roi()`	Region of Interest: per-region QP delta for up to `VPAP_ENC_MAX_ROI` regions.
`CMD_VIDEO_SET_OSD_BITMAP`	`video_set_osd_bitmap()`	Configure an OSD overlay (position, size, alpha, YUV buffer pointer).
`CMD_VIDEO_DISABLE_OSD`	`video_disable_osd()`	Disable an OSD overlay by hardware index.
`CMD_VIDEO_STREAM_STOP`	`video_close()`	Stop and close the encoder stream.
`CMD_VIDEO_SET_RC_EXT` / `CMD_VIDEO_GET_RC_EXT`	`video_set_rc_ext()` / `video_get_rc_ext()`	Bulk update or read rate-control extension fields. Set `vpap_shared_enc_rc_ext_t.flags` to select which fields apply.
`CMD_VIDEO_SET_CODING` / `CMD_VIDEO_GET_CODING`	`video_set_coding()` / `video_get_coding()`	Bulk update or read coding-control fields. Set `vpap_shared_enc_coding_t.flags` to select which fields apply.

Terms used in this section

RC: Rate control. Encoder logic that controls bitrate and quality.
QP: Quantisation parameter. Lower QP usually means higher quality and larger frame size.
GOP: Group of pictures. In this document it mainly refers to the I-frame interval.
SPS / PPS / VUI: Bitstream header/signaling data used by decoders to interpret H.264/H.265 streams.
HRD / CPB: Decoder buffering and bitstream timing model.
CIR / ROI: Coding-control features that can be updated through the encoder coding-control path when the SDK exposes a runtime helper for them.

For media example usage, see Media Example .