Interrupts, and how to divide them between cores
Symetric MultiProcessing (SMP) are becoming increasingly common in embedded systems. Many chip manufacturers such as Texas Instruments and Freescale offers ARM multicores, even FPGA manufacturers like Xilinx and Altera has System-on-Chip with multiple ARM cores. One benefit with SoC is that it's even possible to add soft cores in the programmable logic if it's necessary.
The trend is clear, multiple cores is here and it's not likely to be fewer of them.
But how do we share resources between the cores? One way is to use cgroups (Control Groups) that was merged into the kernel version 2.6.24. Cgroups let you divide resources like cores and memory to specific task groups. More about that in another post.
It's also possible to tell which processor core that is allowed to handle a specific interrupt. This is done by procfs. There are a few entries in procfs related to interrupt-handling. The first interesting entry is /proc/interrupts that is used to record the number of interrupts per CPU.
Beside the the counts of hard IRQ numbers, it also includes interrupts internal to the system that is not associated with a device. Examples on these are NMI (nonmaskable interrupts) and LOC (Local timer interrupt). It also gives us the name of the registred IRQ handler.
Example output:
1# cat /proc/interrupts
2 CPU0 CPU1
334: 0 0 GIC sdma
435: 0 0 GIC VPU_JPG_IRQ
537: 0 0 GIC imx-ipuv3
638: 0 0 GIC imx-ipuv3
739: 0 0 GIC imx-ipuv3
8400: 0 0 GPIO pfuze
9461: 0 0 GPIO btn volume-up
10150: 19293 0 GIC enet
11IPI0: 0 6028 Timer broadcast interrupts
12IPI1: 1953 31365 Rescheduling interrupts
13IPI2: 0 0 Function call interrupts
14IPI3: 1825 1792 Single function call interrupts
15IPI4: 0 0 CPU stop interrupts
16LOC: 25302 130 Local timer interruptsSMP affinity is controlled by manipulating files in the /proc/irq directory.
In /proc/irq is the file default_smp_affinity that specifies the default affinity mask that applies to all non-active IRQs. Once the IRQ is allocated/activated its affinity bitmask will be set to the default mask. The default mask is set so all available cpu cores is allowed to handle the interrupt.
In /proc/irq is also directories that correspond to the IRQ present in the system. These directories has a few entries where two of them are of interest; smp_affinity_list and smp_affinity.
Here is the entries for IRQ 150, which correspond to the ethernet controller (look at the output from /proc/interrupts):
1# ls -1 /proc/irq/150
2affinity_hint
3effective_affinity
4effective_affinity_list
5node
6smp_affinity
7smp_affinity_list
8spurioussmp_affinity_list contains the available processor cores that are able to handle the interrupt, and the smp_affinity is the bitmask of processor cores that is allowed to handle it.
The current values are:
1# cat /proc/irq/150/smp_affinity_list
20-1
3# cat /proc/irq/150/smp_affinity
43This tells us that processor core 0 and 1 is able to handle the interrupt and processor 0 and 1 (bit 0 and 1 gives us 3 dec) is allowed to handle it.
If we look at the output from /proc/interrupts, we see that all interrupts (19293) has been handled by the CPU0. If we just want CPU1 to handle these interrupts, set smp_affinty to 2:
1# echo 2 > /proc/irq/150/smp_affinityThen look at /proc/interrupts again:
1# cat /proc/interrupts
2 CPU0 CPU1
3150: 20967 12 GIC enetThe CPU1 interrupt counter is ticking and the interrupt is now only served by CPU1.
Resources
Documentation/IRQ-affinity.txt