linux

Remoteproc, rpmsg and SYSBIOS

Posted by Marcus Folkesson on Thursday, December 5, 2024

Remoteproc, rpmsg and SYSBIOS

I'm currently working with a couple of SoCs (OMAPL-138, DRA7xx) from Texas Instruments where both has an C6xx DSP core along with the ARM core. This is no unusual configuration, many modern SoCs today have heterogeneous remote processor devices in asymetric multiprocessing (AMP) configurations like these two.

In this case, the DSP-cores where supposed to run SYS/BIOS with the TI IPC MessageQ stack to communicate with the ARM core. However, TI has not implemented support for everything needed in the 6.1 Linux kernel, which we were aiming to use.

This post will first briefly describe what remoteproc and rpmsg is, how it is used for communication beween the DSP/CPU cores followed by the steps I took to get it up and running on a 6.1 kernel.

/media/cores-talking.png

Remoteproc

First, lets have a look at the Remote Processor Framework.

Remoteproc is a framework that allows the ARM host processor to load firmware, power on and power off remote processors in an abstracted way.

Some SoCs may need some tricks&tweaks to start all cores. Such tweaks may be to set specific cmdline options to enable clocks ( e.g. imx8mm.mcore_booted [1] ), but most SoCs does not need any special treatment other than enable support for the remoteproc driver. In my case I have to enable CONFIG_DA8XX_REMOTEPROC or CONFIG_OMAP_REMOTEPROC for OMAPL138 and DRA7xx respectively.

Using Remoteproc from sysfs

All rproc functionalty is exposed via sysfs:

1root@mfoc:~# ls /sys/class/remoteproc/remoteproc0/
2coredump             firmware             recovery             subsystem
3dev                  name                 rproc-virtio.0.auto  uevent
4device               power                state

The first step is to load a firmware on the remote processor. Most of the remoteproc drivers is looking for a firmware file specified by the firmware-name property in the devicetree. This is what the node for omap4-dsp may look like:

 1    ocp {
 2        dsp: dsp {
 3            compatible = "ti,omap4-dsp";
 4            ti,bootreg = <&scm_conf 0x304 0>;
 5            iommus = <&mmu_dsp>;
 6            mboxes = <&mailbox &mbox_dsp>;
 7            memory-region = <&dsp_memory_region>;
 8            ti,timers = <&timer5>;
 9            ti,watchdog-timers = <&timer6>;
10            clocks = <&tesla_clkctrl OMAP4_DSP_CLKCTRL 0>;
11            resets = <&prm_tesla 0>, <&prm_tesla 1>;
12            firmware-name = "omap4-dsp-fw.xe64T";
13        };
14    };

In this case, the system will search "/lib/firmware/* " for the omap4-dsp-fw.xe64T firmware during boot. It is also possible to build firmware into the kernel binary by setting CONFIG_EXTRA_FIRMWARE, but be sure to use GPL on all your work to avoid any licence violations. Combining GPL and non-GPL work in the same binary is something that simply should not be done.

There are a few default paths [2] that the firmware_class searches for firmware, these are:

  • fw_path_para - module parameter - default is empty so this is ignored
  • /lib/firmware/updates/UTS_RELEASE/
  • /lib/firmware/updates/
  • /lib/firmware/UTS_RELEASE/
  • /lib/firmware/

The fw_path_para could be set to add an alterative search path if the firmware is not stored in any default location. Either add firmware_class.path=YOUR_PATH as a kernel parameter or, at runtime, write the path to /sys/module/firmware_class/parameters/path. E.g:

1root@mfoc# echo -n /path/to/firmware > /sys/module/firmware_class/parameters/path

Be aware that newline characters will be taken into account and may not produce the intended effects. Make sure to not have a trailing newline character in the path.

Lets assume I have stored my DSP firmware in /lib/firmware/dsp-firmare, I can then point it out and power up the DSP by write to firmware and state respectively:

 1root@mfoc# echo dsp-firmware > /sys/class/remoteproc/remoteproc0/firmware
 2root@mfoc# echo start > /sys/class/remoteproc/remoteproc0/state
 3
 4remoteproc remoteproc0: powering up dsp
 5remoteproc remoteproc0: Booting fw image dsp, size 4263692
 6rproc-virtio rproc-virtio.0.auto: assigned reserved memory node dsp-memory@c3000000
 7virtio_rpmsg_bus virtio0: rpmsg host is online
 8virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
 9rproc-virtio rproc-virtio.0.auto: registered virtio0 (type 7)
10remoteproc remoteproc0: remote processor dsp is now up

In the very same way, I can stop the execution on the DSP by write stop to the state entry: 

1root@mfoc# echo stop > /sys/class/remoteproc/remoteproc0/state

The state of the remote processor could be either running or offline.

rpmsg

Remote Processor Messaging, RPMSG, is a virtio-based messaging bus that allows kernel drivers to communicate with remote processors. It is basically the framework for sending frames between heterogenos processors. A higher level driver is required to expose the rpmsg functionality to userspace in a useful way, which you usually find in the SoC vendor tree.

There are a few mainlined rpmsg drivers, most of them to support SoCs from Qualcomm.

System components

First I will try to describe the components that we have to deal with.

OMAPL138 and DRA7XX are AMP SoCs that both contains ARM-core(s) and a DSP-core. Remoteproc is used to load and start firmware on the subcores and is implemented by the da8xx_remoteproc and omap_remoteproc driver respectively.

The SDK [3] (OMAPL138) contains everything you need to build applications for the DSP and it also contains a few examples to send messages between the cores using rpmsg on the ARM side and SYSBIOS with the TI IPC MessageQ stack on the DSP side.

/media/sysbios-messageq.png

All proceses in Linux communicats with the LAD daemon via AF_RPMSG sockets.

Linux

TI has their own BSP Yocto meta-layer [4] which implements recipes for the SDK including the examples (see ti-ipc and ti-ipc-examples-linux).

The SDK nor the examples are adapted to a recent (Scarthgap) Yocto version, which is quite surprising as TI_PREFERRED_BSP for the OMAPL138 is version 6.11 and the latest supported kernel version for for the SDK is v5.15. In other words, the remoteproc/rpmsg will not work with the latest BSP from Texas Instruments.

SYSBIOS

SYSBIOS is the operating system that executes on the DSP. It has its own IPC stack called MessageQ which is used to send and receive messages to/from the ARM CPU.

MessageQ

MessageQ is the IPC framework used by SYSBIOS to communicate with the ARM core.

AF_RPMSG

AF_RPMSG is a TI specific way to use for communication with the LAD service.

My workflow

I'm quite familiar with remoteproc and rpmsg, but I had no detailed knowledge on how it works on these SoCs. It was hard to find any good documentation (hence this post) that described the purpose of each component involved. So I did spend most of my time reading source code to figure out how it is supposed to work in order to fix what didn't work.

Firmware loading

When I first tried to startup a firmware on the subcore I faced this error:

1root@mfoc:/sys/class/remoteproc/remoteproc0# echo start > state
2remoteproc remoteproc0: powering up dsp
3remoteproc remoteproc0: Booting fw image dsp, size 4263692
4rproc-virtio rproc-virtio.1.auto: assigned reserved memory node dsp-memory@c3000000
5virtio_rpmsg_bus virtio0: rpmsg host is online
6virtio_rpmsg_bus virtio0: malformed ns msg (72)
7rproc-virtio rproc-virtio.1.auto: registered virtio0 (type 7)
8remoteproc remoteproc0: remote processor dsp is now up

virtio_rpmsg_bus virtio0: malformed ns msg (72)

The error comes from rpmsg_ns.c [5]

 1static int rpmsg_ns_cb(struct rpmsg_device * rpdev, void * data, int len,
 2		       void * priv, u32 src)
 3{
 4
 5 [...]
 6
 7	if (len != sizeof(* msg)) {
 8		dev_err(dev, "malformed ns msg (%d)\n", len);
 9		return -EINVAL;
10	}

Okay, so the size of the messages recieved from the rpmsg framework did not match the expected size.

The struct rpmsg_ns_msg in the kernel looks like this [6] :

1struct rpmsg_ns_msg {
2	char name[RPMSG_NAME_SIZE];
3	__rpmsg32 addr;
4	__rpmsg32 flags;
5} __packed;

But the corresponding definiton in the TI SDK looks a little different:

1struct rpmsg_ns_msg {
2	char name[RPMSG_NAME_SIZE];
3	char desc[RPMSG_NAME_SIZE];
4	__rpmsg32 addr;
5	__rpmsg32 flags;
6} __packed;

The structure is used by both SYSBIOS and the rpmsg framework and has of course to look the same on both ends. The struct is only used to transport data and the rpmsg framework does only cares about the size to match, so it should be safe to change.

After the change the firmware loads and start properly:

1[  141.562011] remoteproc remoteproc2: Booting fw image dsp1, size 4679980
2[  141.568908] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
3[  141.574829] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
4[  141.610839] rproc-virtio rproc-virtio.0.auto: assigned reserved memory node dsp1-memory@99000000
5[  141.631744] virtio_rpmsg_bus virtio0: rpmsg host is online
6[  141.637298] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x3d
7[  141.656341] rproc-virtio rproc-virtio.0.auto: registered virtio0 (type 7)
8[  141.676483] remoteproc remoteproc2: remote processor 40800000.dsp is now up

Porting RPMSG_PROTO to v6.1

To support a new addess family (AF_RPMSG) in the application layer, we will need kernel support. I found the vendor tree [9] that had it implemented and started to port it to Linux v6.1.

CONFIG_RPMSG_PROTO is the kernel configuraion option used by this implementation.

The porting was quite straight forward, not really much to say about it. You will find the patches in the summary section of this post.

Test it

After changing the rpmsg_ns_msg structure and porting CONFIG_RPMSG_PROTO to the v6.1 kernel, it is finally time for some more test runs:

1root@mfoc:~# /usr/bin/ipc/examples/ex02_messageq/debug/app_host -l
2--> main:
3Processor List
4    procId=0, procName=HOST
5    procId=1, procName=IPU2
6    procId=2, procName=IPU1
7    procId=3, procName=DSP2
8    procId=4, procName=DSP1

Nice! Communication with the LAD service is working.

What about the IPC communication with SYSBIOS?

 1root@mfoc:~# /usr/bin/ipc/examples/ex02_messageq/debug/app_host DSP1
 2--> main:
 3--> Main_main:
 4--> App_create:
 5App_create: Host is ready
 6<-- App_create:
 7--> App_exec:
 8App_exec: sending message 1
 9App_exec: sending message 2
10App_exec: sending message 3
11App_exec: message received, sending message 4
12App_exec: message received, sending message 5
13App_exec: message received, sending message 6
14App_exec: message received, sending message 7
15App_exec: message received, sending message 8
16App_exec: message received, sending message 9
17App_exec: message received, sending message 10
18App_exec: message received, sending message 11
19App_exec: message received, sending message 12
20App_exec: message received, sending message 13
21App_exec: message received, sending message 14
22App_exec: message received, sending message 15
23App_exec: message received
24App_exec: message received
25App_exec: message received
26<-- App_exec: 0
27--> App_delete:
28<-- App_delete:
29<-- Main_main:
30<-- main:

Works like a charm.

Summary

There were a few missing parts to get everything to work. Even if the meta-ti layer is compatible with the Scarthgap version of Yocto, it misses some essential parts. This probably applies to all of the OMAP4/OMAP5 & DRA7xx/AM57xx; Keystone K2 & Davinci OMAP-L138 SoCs.

You find my patches here:

Apply them to your v6.1 kernel tree and you are ready to go.

CATALOG
    FEATURED TAGS
    audio beer birds blacksmith booting buildroot camera clockwork cultivation debugging device drivers dnd docker git go ham home control hugo knives leatherwork libostree linux linux kernel network programming qemu radio swupdate systemd til tools v4l2 vim wireless yocto zephyr