Crosscompile libcamera for RPi
Goal
The goal is to cross-compile libcamera [1] and libcamera-apps [2] for Raspberry Pi using the latest Raspbian [3] (Bullseye) release. Usually you setup the root filesystem with Buildroot [4] or Yocto [5] and generate a SDK that you can use to compile your application. The Raspbian distribution does not come with a SDK so we have to setup our own.
We will use a Raspberry Pi 3b for this.
What is libcamera?
You can read about libcamera in a previous post [6].
Prepare the SD-card
We will boot the Raspberry Pi from a SD-card, so we first have to prepare it.
Download Raspbian Bullseye
You could either download the image [3] yourself or use rpi-imager. Only make sure that you chose the 64-bit version as that is what the toolchain we are going to use is built for.
To download and flash it yourself
wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-09-26/2022-09-22-raspios-bullseye-arm64-lite.img.xz tar xf 2022-09-22-raspios-bullseye-arm64-lite.img.xz sudo dd if=2022-09-22-raspios-bullseye-arm64-lite.img.xz of=/dev/mmcblk0 conv=sync
/dev/mmcblk0 will be overwritten, please double check that this is your SD-card dedicated for your Raspberry and not your wedding pictures.
I will stick to rpi-imager as it let you configure WLAN, enable SSH and set passwords in the configuration menu. I find it smooth.
As I'm using Arch Linux, rpi-imager is available as an AUR package
git clone https://aur.archlinux.org/rpi-imager.git cd rpi-imager makepkg -i
Other Debian-based distribution could just use apt-get
apt-get install rpi-imager
Enable UART
Once the image is written to the SD-card we can enable the UART. This is not necessary but I strongly prefer to have a serial port connected to the device I'm working with to see the boot log and get a getty.
Mount the boot partition and enable UART by write "enable_uart=1" to config.txt
sudo mount /dev/mmcblk0p1 mnt/disk/ echo enable_uart=1 | sudo tee -a mnt/disk/config.txt sudo umount mnt/disk
Permit root login on SSH
Now it's time to power up the Raspberry Pi. Put the SD-card in the slot, power it up and login as the pi user, either via UART or SSH.
Permit root login in order to mount root filesystem via SSH
echo PermitRootLogin yes | sudo tee -a /etc/ssh/sshd_config
Restart the SSH service
sudo systemctl restart sshd.service
Note that it's bad practice to let the root user login via SSH (especially with password). Either use SSH-keys or disable it later.
If you not allready have figured out which IP address the RPi has, grab it with ip addr
pi@raspberrypi:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:91:e6:2a brd ff:ff:ff:ff:ff:ff
inet 192.168.1.111/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
valid_lft 409sec preferred_lft 334sec
inet6 fe80::cfe6:1f35:c5b6:aa1a/64 scope link
valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:c4:b3:7f brd ff:ff:ff:ff:ff:ff
inet 192.168.1.56/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan0
valid_lft 408sec preferred_lft 333sec
inet6 fe80::61e6:aeb:2c0e:f31b/64 scope link
valid_lft forever preferred_lft forever
My board has the IP 192.168.1.111 on the eth0 interface.
Set root password
The root user does not have a password by default, so set it
sudo passwd root
Install dependencies
Especially libcamera-apps have a lot of dependencies, install those using apt-get
sudo apt-get install libboost-program-options-dev libboost-dev libexif-dev libjpeg-dev libtiff-dev libpng-dev libdrm-dev libavcodec-dev libavdevice-dev
Prepare the host
Now we have everything on target in place so can switch back to the host system.
From now on, we will use environment variables to setup paths to all directories we will refer to. First, create those directories
mkdir rootfs staging tools
rootfs will be used for our network mounted root filesystem. staging will be our sysroot we compile against tools will contain our cross toolchain
Export the environment variables
export RPI_BASE=`pwd` export RPI_ROOTFS=$RPI_BASE/rootfs export RPI_STAGING=$RPI_BASE/staging export RPI_TOOLS=$RPI_BASE/tools/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/
Install cross toolchain
Download cross toolchain with the same GCC version (10.2) as Raspian.
wget http://sources.buildroot.net/toolchain-external-arm-aarch64/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz tar -xf gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu.tar.xz -C tools/
Add the toolchain to your $PATH
export PATH=$RPI_TOOLS:$PATH
Mount root filesystem
SSHFS (Secure SHell File System) [7] is a handy tool based in libfuse that let you mount a filesystem over SSH.
Mount the root filesystem
sshfs root@192.168.1.111:/ $RPI_ROOTFS
Prepare the staging directory
Here we will copy files from our root filesystem
mkdir -p $RPI_STAGING/usr/ cp -r $RPI_ROOTFS/usr/lib $RPI_STAGING/usr/ cp -r $RPI_ROOTFS/usr/include/ $RPI_STAGING/usr/
Now you may use ./rootfs as your sysroot. However, as the Rasbian image has C-library for both musl and glibc, the search pathes becomes hard to handle. We will look on how we could handle this later, but that is simply not worth it in my opinion. Instead, copy the glibc libraries to /usr/lib
cp $RPI_STAGING/usr/lib/aarch64-linux-gnu/* $RPI_STAGING/usr/lib/
We also need to create a symlink to /lib as the toolchain is looking for the linux loader in that directory
ln -s usr/lib/ $RPI_STAGING/lib
The the libpthread.so.6 is pointing to an aboslut path (will point to our host system). The result will be that the linker does not find the library and will fallback on the static linked library instead. That will not fall out good as the glibc is still dynamically linked... so create a new symlink
ln -sf libpthread.so.0 $RPI_STAGING/lib/libpthread.so
Cross compile libcamera
Cross compile libcamera is quite straight forward as it does use Meson as build system and do not have any external dependencies.
Clone the repository
git clone https://git.libcamera.org/libcamera/libcamera.git cd libcamera
We need to create a toolchain file to instruct the Meson build system which toolchain it should use. Create aarch64.txt which contains
[binaries] c = 'aarch64-none-linux-gnu-gcc' cpp = 'aarch64-none-linux-gnu-g++' ar = 'aarch64-none-linux-gnu-ar' strip = 'aarch64-none-linux-gnu-strip' [host_machine] system = 'linux' cpu_family = 'x86_64' cpu = 'x86_64' endian = 'little'
Now we could build
meson build -Dprefix=/usr/ --cross-file ./aarch64.txt cd build ninja
Install on both root filesystem and staging directory
DESTDIR=$RPI_ROOTFS ninja install DESTDIR=$RPI_STAGING ninja install
Cross compile libcamera-apps
libcamera-apps on the other hand, have a lot of dependencies but as we have collected all of those into the staging directory, even this is pretty straight forward.
Just for your information, if we did not copy the glibc-libraries from /usr/lib/aarch-linux-gnu to /usr/lib, this would not be a straight trip at all.
Clone the repository
git clone https://github.com/raspberrypi/libcamera-apps.git cd libcamera-apps
I'm not interresting in the preview application as it needs Qt5, so I will remove it
sed -i "/add_subdirectory(preview)/d" CMakeLists.txt
I also did remove libcamera-raw and libcamera-vid from apps/CMakeLists.txt as those have dependencies that I do not want on my target.
libcamera-apps does use CMake as build system, and we need to create a toolchain file for this as well. Create the file aarch64.cmake which contains
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)
set(CMAKE_SYSROOT $ENV{RPI_STAGING})
#We need to point to the glibc-headers
set(CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES $ENV{RPI_STAGING}/usr/include/aarch64-linux-gnu/)
set(CMAKE_C_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER $ENV{RPI_TOOLS}/aarch64-none-linux-gnu-g++)
#Let pkg-config look in our root
SET(ENV{PKG_CONFIG_LIBDIR} ${CMAKE_FIND_ROOT_PATH}/lib/pkgconfig/)
set(CMAKE_FIND_ROOT_PATH "${CMAKE_SYSROOT}")
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
Now we are ready to build
mkdir build cd build cmake -DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake ../ make
And install
make DESTDIR=$RPI_ROOTFS install
We do only need to install to our root filessytem as it should run on target.
Test on target
We can now run the applications on target and verify that it's our newly built app:
pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg --version libcamera-apps build: 80f17befef34-intree-dirty 26-11-2022 (15:55:47) libcamera build: v0.0.2+34-b35f04b3
For the good sake, we could also take an image
pi@raspberrypi:/usr/local/bin$ LD_LIBRARY_PATH=/usr/local/lib ./libcamera-jpeg -o /tmp/test1.jpg
Now you can disable root login for SSH
sed -i "/PermitRootLogin/d" /etc/ssh/sshd_config sudo systemctl restart sshd.service
What if...
...we should not copy /usr/lib/aarch64-linux-gnu to /usr/lib but keep it as it's? It would be nice to get rid of the staging directory and only use the root filesystem for cross compiling.
That was my first intention, but it turned out to be really troublesome, even if I got it to work at last.
In short what is needed to be done is:
- Export PKG_CONFIG_SYSROOT_DIR to $RPI_ROOTFS
- Export PKG_CONFDIG_PATH to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu/pkgconfig to make it find pkgconfig files
- Export BOOST_ROOT to $RPI_ROOTFS to give FindBoost.cmake a hint of where to look
- Export BOOST_LIBRARYDIR to $RPI_ROOTFS/usr/lib/aarch64-linux-gnu
- Set CMAKE_LIBRARY_PATH to look into $RPI_ROOTFS/usr/lib/aarch64_linux-gnu
- Set CMAKE_CXX_FLAGS_INIT, CMAKE_C_FLAGS_INIT and CMAKE_EXE_LINKER_FLAGS to search $RPI_ROOTFS/usr/lib/aarch64-linux-gnu for libraries
There are simply too much special and I think it's not worth it.
Conclusion
This sounds like a simple thing, but it actually took quite a while to get it working. Mostly because of the that the Raspian distribution supports both and glibc/musl so that libraries, pkgconfig files and headers end up in non-standard search paths.
I find it quite strange that there are no SDK available for the Raspian images, it would help development against the platform a lot.
References
| [1] | https://git.libcamera.org/libcamera/libcamera.git |
| [2] | https://github.com/raspberrypi/libcamera-apps.git |
| [3] | (1, 2) https://downloads.raspberrypi.org/raspios_lite_armhf/images/ |
| [4] | https://buildroot.org/ |
| [5] | https://www.yoctoproject.org/ |
| [6] | https://www.marcusfolkesson.se/blog/libcamera/ |
| [7] | https://github.com/libfuse/sshfs |