V4L2 and media controller

Posted by Marcus Folkesson on Sunday, March 29, 2020

V4L2 and media controller

The media infrastructure in the kernel is a giant beast handling many different types of devices involving different busses and electrical interfaces. Providing an interface to handle the complexity of the hardware is not an easy task. Most devices have multiple ICs with different communication protocols... so the device drivers tends to be very complex as well.

Video For Linux 2 (V4L2) is the interface for such media devices. V4L2 is the second version of V4L and is not really compatible with V4L, even if there is a compatibility mode but the support is more than often incomplete. The name Video4Linux is a counterpart to Video4Windows, but is not technically related to it at all.

Here is an example of what a system may look like (borrowed from the Linux kernel documentation)

/media/typical_media_device.svg

Media controller

System-on-Chips (SoC) devices often provides wide range of hardware blocks that can be interconnected in a variety of ways to obtain the desired functionality. To configure these hardware blocks, the kernel provides the Media Controller kernel API which expose detailed information about the media device and let them to be interconnected in a dynamic and complex way at runtime, all from userspace.

Each hardware block, called entity, in the media controller framework has one or more source and sink pads. The API let the user link source to sink pads and set the format of pads.

Here is a the topology exported from my sabresd with an imx219 (camera module) connected:

/media/media-ctl-graph.png

Let's go through the entities in the picture. All these entities is of course specific for the iMX6 SoC. (Partly taken from the kernel documentation)

imx219 1-0010

This is the camera sensor. The sensor is controlled with I2C commands and the data stream is over the MIPI CSI-2 interface. The name tells us that the sensor is connected to I2C bus 1. The device has the address 0x10.

The entity has one source pad.

imx6-mipi-csi2

This is the MIPI CSI-2 receiver entity. It has one sink pad to receive the MIPI CSI-2 stream (usually from a MIPI CSI-2 camera sensor). It has four source pads, corresponding to the four MIPI CSI-2 demuxed virtual channel outputs. Multiple source pads can be enabled to independently stream from multiple virtual channels.

ipuX_csiY_mux

This is the video multiplexers. They have two or more sink pads to select from either camera sensors with a parallel interface, or from MIPI CSI-2 virtual channels from imx6-mipi-csi2 entity. They have a single source pad that routes to a CSI (ipuX_csiY entities).

ipuX_csiY

These are the CSI entities. They have a single sink pad receiving from either a video mux or from a MIPI CSI-2 virtual channel as described above.

ipuX_vdic

The VDIC carries out motion compensated de-interlacing, with three motion compensation modes: low, medium, and high motion. The mode is specified with the menu control V4L2_CID_DEINTERLACING_MODE. The VDIC has two sink pads and a single source pad.

ipuX_ic_prp

This is the IC pre-processing entity. It acts as a router, routing data from its sink pad to one or both of its source pads.

The direct sink pad receives from an ipuX_csiY direct pad. With this link the VDIC can only operate in high motion mode.

ipuX_ic_prpenc

This is the IC pre-processing encode entity. It has a single sink pad from ipuX_ic_prp, and a single source pad. The source pad is routed to a capture device node, with a node name of the format "ipuX_ic_prpenc capture".

This entity performs the IC pre-process encode task operations: color-space conversion, resizing (downscaling and upscaling), horizontal and vertical flip, and 90/270 degree rotation. Flip and rotation are provided via standard V4L2 controls.

Like the ipuX_csiY IDMAC source, this entity also supports simple de-interlace without motion compensation, and pixel reordering.

ipuX_ic_prpvf

This is the IC pre-processing viewfinder entity. It has a single sink pad from ipuX_ic_prp, and a single source pad. The source pad is routed to a capture device node, with a node name of the format "ipuX_ic_prpvf capture".

This entity is identical in operation to ipuX_ic_prpenc, with the same resizing and CSC operations and flip/rotation controls. It will receive and process de-interlaced frames from the ipuX_vdic if ipuX_ic_prp is receiving from ipuX_vdic.

Capture video stream from sensor

In order to capture a video stream from the sensor we need to:

  1. Create links between the needed entities
  2. Configure pads to hold the correct image format

To do this, we use the media-ctl [1] tool.

Configure pads

We also need to configure each pad to the right format. This image sensor is ouput in raw bayer format (SRGGB8).

export fmt=SRGGB8_1X8/640x480
media-ctl --set-v4l2 "'imx219 1-0010':0[fmt:$fmt field:none]"
media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:$fmt field:none]"
media-ctl --set-v4l2 "'ipu1_csi0_mux':5[fmt:$fmt field:none]"
media-ctl --set-v4l2 "'ipu1_csi0':1[fmt:$fmt field:none]"

Stream to framebuffer

Now a full pipe is created from imx219 to the video0 device.

GStreamer is a handy multimedia framework that we can use to test the full chain

gst-launch-1.0 -vvv v4l2src device=/dev/video0 io-mode=dmabuf blocksize=76800 ! "video/x-bayer,format=rggb,width=640,height=480,framerate=30/1" ! queue ! bayer2rgbneon ! videoconvert ! fbdevsink sync=false
/media/imx219.jpg