V4L2 and media controller
The media infrastructure in the kernel is a giant beast handling many different types of devices involving different busses and electrical interfaces. Providing an interface to handle the complexity of the hardware is not an easy task. Most devices have multiple ICs with different communication protocols... so the device drivers tends to be very complex as well.
Video For Linux 2 (V4L2) is the interface for such media devices. V4L2 is the second version of V4L and is not really compatible with V4L, even if there is a compatibility mode but the support is more than often incomplete. The name Video4Linux is a counterpart to Video4Windows, but is not technically related to it at all.
Here is an example of what a system may look like (borrowed from the Linux kernel documentation)
Media controller
System-on-Chips (SoC) devices often provides wide range of hardware blocks that can be interconnected in a variety of ways to obtain the desired functionality. To configure these hardware blocks, the kernel provides the Media Controller kernel API which expose detailed information about the media device and let them to be interconnected in a dynamic and complex way at runtime, all from userspace.
Each hardware block, called entity, in the media controller framework has one or more source and sink pads. The API let the user link source to sink pads and set the format of pads.
Here is a the topology exported from my sabresd with an imx219 (camera module) connected:
Let's go through the entities in the picture. All these entities is of course specific for the iMX6 SoC. (Partly taken from the kernel documentation)
imx219 1-0010
This is the camera sensor. The sensor is controlled with I2C commands and the data stream is over the MIPI CSI-2 interface. The name tells us that the sensor is connected to I2C bus 1. The device has the address 0x10.
The entity has one source pad.
imx6-mipi-csi2
This is the MIPI CSI-2 receiver entity. It has one sink pad to receive the MIPI CSI-2 stream (usually from a MIPI CSI-2 camera sensor). It has four source pads, corresponding to the four MIPI CSI-2 demuxed virtual channel outputs. Multiple source pads can be enabled to independently stream from multiple virtual channels.
ipuX_csiY_mux
This is the video multiplexers. They have two or more sink pads to select from either camera sensors with a parallel interface, or from MIPI CSI-2 virtual channels from imx6-mipi-csi2 entity. They have a single source pad that routes to a CSI (ipuX_csiY entities).
ipuX_csiY
These are the CSI entities. They have a single sink pad receiving from either a video mux or from a MIPI CSI-2 virtual channel as described above.
ipuX_vdic
The VDIC carries out motion compensated de-interlacing, with three motion compensation modes: low, medium, and high motion. The mode is specified with the menu control V4L2_CID_DEINTERLACING_MODE. The VDIC has two sink pads and a single source pad.
ipuX_ic_prp
This is the IC pre-processing entity. It acts as a router, routing data from its sink pad to one or both of its source pads.
The direct sink pad receives from an ipuX_csiY direct pad. With this link the VDIC can only operate in high motion mode.
ipuX_ic_prpenc
This is the IC pre-processing encode entity. It has a single sink pad from ipuX_ic_prp, and a single source pad. The source pad is routed to a capture device node, with a node name of the format "ipuX_ic_prpenc capture".
This entity performs the IC pre-process encode task operations: color-space conversion, resizing (downscaling and upscaling), horizontal and vertical flip, and 90/270 degree rotation. Flip and rotation are provided via standard V4L2 controls.
Like the ipuX_csiY IDMAC source, this entity also supports simple de-interlace without motion compensation, and pixel reordering.
ipuX_ic_prpvf
This is the IC pre-processing viewfinder entity. It has a single sink pad from ipuX_ic_prp, and a single source pad. The source pad is routed to a capture device node, with a node name of the format "ipuX_ic_prpvf capture".
This entity is identical in operation to ipuX_ic_prpenc, with the same resizing and CSC operations and flip/rotation controls. It will receive and process de-interlaced frames from the ipuX_vdic if ipuX_ic_prp is receiving from ipuX_vdic.
Capture video stream from sensor
In order to capture a video stream from the sensor we need to:
- Create links between the needed entities
- Configure pads to hold the correct image format
To do this, we use the media-ctl [1] tool.
Create links
In the most straight forward example, we want to make the following links:
imx219 -> imx6_mipi_csi1 -> ipu1_csi0_mux -> ipu_csi0 -> ipu_csi0 capture
All links is between pads. For example
ipu1_csi0_mux':5->'ipu1_csi0':0[1]
Connects pad 5 of ipu1_csi0_mux to pad 0 of ipu1_csi0.
A full example of the pipeline above
media-ctl --links "'imx219 1-0010':0->'imx6-mipi-csi2':0[1]" media-ctl --links "'imx6-mipi-csi2':1->'ipu1_csi0_mux':0[1]" media-ctl --links "'ipu1_csi0_mux':5->'ipu1_csi0':0[1]" media-ctl --links "'ipu1_csi0':2->'ipu1_csi0 capture':0[1]"
Configure pads
We also need to configure each pad to the right format. This image sensor is ouput in raw bayer format (SRGGB8).
export fmt=SRGGB8_1X8/640x480 media-ctl --set-v4l2 "'imx219 1-0010':0[fmt:$fmt field:none]" media-ctl --set-v4l2 "'imx6-mipi-csi2':1[fmt:$fmt field:none]" media-ctl --set-v4l2 "'ipu1_csi0_mux':5[fmt:$fmt field:none]" media-ctl --set-v4l2 "'ipu1_csi0':1[fmt:$fmt field:none]"
Stream to framebuffer
Now a full pipe is created from imx219 to the video0 device.
GStreamer is a handy multimedia framework that we can use to test the full chain
gst-launch-1.0 -vvv v4l2src device=/dev/video0 io-mode=dmabuf blocksize=76800 ! "video/x-bayer,format=rggb,width=640,height=480,framerate=30/1" ! queue ! bayer2rgbneon ! videoconvert ! fbdevsink sync=false
