Fetching the kernel using HTTP on the RPI3¶
To make a long story short when working with the Raspberry Pi 3 constantly removing and inserting the SD card is not practical
A good solution is network booting, hosting the root filesystem and kernel on the development host and retrieving them over the network - the SD card will only contain a bootloader. This provides an effective way to switch between different kernels and also work with various user space environments: GNU, busybox/buildroot, alpine/busybox, without touching the SD card.
Another benefit is the improvement of the speed of the ‘build, deploy, test, fix’ cycle.
It’s not the first time that I am configuring such a setup on a RaspberryPi 3, This time, I am documenting it.. :)
This article is more of a development journal - and will be split into multiple parts.
The first part explains how to configure U-Boot to fetch Linux over HTTP, subsequent parts will focus on how to configure the kernel to mount a remote root filesystem. Both the kernel and the rootfs will be hosted on my development host - which is my work laptop running Ubuntu Jammy 22.04
U-Boot to rescue¶
By default, the Raspberry Pi 3 uses a custom bootloader and it is perfectly sufficient for most use cases. However, this use case requires a more ‘feature rich’ bootloader because the kernel needs to be fetched over the network.
U-Boot is a popular bootloader that supports many architectures, it can fetch a kernel image over the network. Currently only HTTP and TFTP protocols are supported, HTTP support being the most recent addition. It was added back in 2022 and there has been many fixes since then.
On the Rpi 3 u-boot acts as a third-stage bootloader, the first stage, which is provided by the vendor.
It lives in the bootcode.bin file on the SD Card, the bootloader then runs start.elf which reads config.txt and the device tree
and only control is given to U-Boot.
Setting up the project directory¶
mkdir -p ~/rpi3/{dependencies,etc,volume,out}
cd ~/rpi3/
git init
git submodule add --depth 1 https://github.com/raspberrypi/firmware dependencies/rpi_firmware/src
git submodule add --depth 1 https://github.com/u-boot/u-boot dependencies/u-boot/src
mkdir creates the file hierarchy for this project.
git downloads the pre-compiled binaries of the current Raspberry Pi Linux kernel,
kernel modules, compiled device trees, user space libraries, and the bootloader/GPU firmware and also U-Boot.
Submodules can be used as a practical way to manage dependencies
U-boot¶
Nowadays, configuring U-Boot for the Raspberry Pi 3 B plus is easy.
Be sure to install the required build dependencies, by following the official documentation https://docs.u-boot.org/en/latest/build/gcc.html#configuration
Copy-paste the apt-get commands from the documentation,
they will install the GCC compiler for the aarch64 architecture and the necessary libraries required to compile U-Boot.
The next step is configuring U-Boot
cd rpi3/dependencies/uboot
make rpi_3_b_plus_defconfig
This generates a .config file based on the default configuration for the Raspberry Pi 3 B+
If you’re using another model, use du -a configs | grep rpi_ to list the available default
configurations for the various models of the Raspberry Pi
Fetching the kernel over HTTP requires the use of the wget monitor command, it can be enabled
by setting the CMD_WGET Kconfig option which requires PROT_TCP which enables the TCP stack
This can be done with menuconfig, a ncurses configuration tool, by typing make menuconfig
and selecting the correct options, the .config file will be modified accordingly.
Save the generated .config somewhere out of tree, to avoid deleting it when running make mrproper
cp .config ~/rpi3/etc/uboot_rpi_3_b_plus.config
Now that U-Boot is configured, simply run make to build it.
CROSS_COMPILE=aarch64-linux-gnu- make all
The compiler generates a uboot.bin file in the current directory
Preparing the SD-Card¶
Most SD cards are pre-formatted with a single FAT-32 partition, the Raspberry Pi 3
uses a two partition setup, one 512 Mb FAT32 boot partition and a EXT4 partition for the rootfs.
Adjusting the partition layout can be achieved with the fdisk(8) utility
sudo fdisk -l
This will output the list of available disk drives, ignore the /dev/loopXX devices of which
there are many on Ubuntu…
USB card readers usually use /dev/sdX so one can filter the output
sudo fdisk -l | grep sd.:
If your system has SATA disks, locate the SD card by looking at the size.
sudo fdisk /dev/sdd will start fdisk(8) in interactive mode - to manipulate the partition table of the disk.
d 1
n 1
p
1
2048
+512M
t
0b
w
It removes the first partition, creates another primary partition of 512 Mb, sets the type to FAT32 and writes the changes to the disk and exists
Note that sfdisk(8) can be used instead to achieve the above result from a script or build system
Check if the partition table is correct by running fdisk -l /dev/sdd sometimes it’s necessary
to remove and insert the card.
Next format the partition by running sudo mkfs.fat /dev/sdd1 and mount the drive, with sudo mount /dev/sdd1 volume
Copy the necessary files to the SD card
sudo cp -r dependencies/rpi_firmware/src/boot/{bootcode.bin,start.elf,fixup.dat,overlays,bcm2710-rpi-3-b-plus.dtb} volume
sudo cp -r dependencies/u-boot/src/u-boot.bin volume
Create the config.txt which is read and parsed by start.elf secondary stage bootloader
cat > etc/bootloader_config.txt <<heredoc
kernel=u-boot.bin
arm64_bit=1
core_freq=250
device_tree=bcm2710-rpi-3-b-plus.dtb
heredoc
Copy the configuration to the SD Card sudo cp etc/bootloader_config.txt volume/config.txt
I found these notes useful, to understand the startup sequence of the Raspberry Pi and the content of config.txt
https://github.com/mhomran/u-boot-rpi3-b-plus
The next step is to create a script for U-Boot to avoid typing monitor commands on the keyboard on each boot
cat > etc/uboot_script.txt <<heredoc
setenv autoload 0
dhcp
setenv serverip 172.22.22.57
wget \${kernel_addr_r} /kernel/linux5.10_rpi
booti \${kernel_addr_r} - \${fdt_addr}
heredoc
This script interrupts the normal loading sequence, acquires an address for the Ethernet card
using the DHCP client and performs a HTTP GET request to fetch the Linux image and store at the address
specified by kernel_addr_r. The request URL will look like this: http://172.22.22.57/linux5.10_rpi
Compile the script and copy it to the SD card
./dependencies/u-boot/src/tools/mkimage -T script -d etc/uboot_script boot.scr
sudo cp boot.scr volume/
Finally, unmount the volume sudo umount volume
Plug in the Ethernet cable to the Raspberry Pi and insert the card - it will attempt to fetch the kernel and fail.
Server setup¶
At this stage, we have a working U-boot setup that makes use of the wget monitor command to fetch
the kernel and then boot it with the booti command great! But a HTTP server needs to serve the kernel when
the bootloader makes a HTTP GET request.
Servers are supposed to have a static IP address, it would be wise to make a DHCP reservation. Setting a static IP on the Ethernet interface used by the web server is another option, It must be an IP address that is outside of the DHCP range of the router you’re using.
Both solutions have a major disadvantage, the U-Boot script will have to be modified when working in a different location. Because the network configuration will most certainly be different from one location to the next.
A solution that doesn’t require patching U-Boot, is to use a dedicated USB network adapter with a DHCP server listening on it.
On Ubuntu, this requires a deep dive into systemd and systemd-networkd and then
choosing the right DHCP server and configuring it…
Another solution could be passing the IP address as a DHCP option, is it possible without patching U-Boot ? - I Need to investigate this by checking the source code/documentation, so it will be the focus of a subsequent article.
For this guide - I will use NGINX, a famous HTTP server that I am familiar with.
It can be installed like so sudo apt-get install nginx
I like to disable the systemd unit file for NGINX sudo systemctl disable nginx
because I tend to run NGINX on-demand, by hand or from a script. It is also possible to run multiple instances of NGINX
but this requires to specify a custom HTTP port in the configuration
Create the NGINX config file
cat > etc/nginx_rpi.conf <<heredoc
user etag;
worker_processes auto;
pid /run/nginx.pid;
events {
worker_connections 768;
}
http {
sendfile on;
tcp_nopush on;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/rpi_access.log;
error_log /var/log/nginx/rpi_error.log;
server {
listen 80 default_server;
root /home/etag/rpi3/out/kernel;
server_name _;
location /kernel {
try_files \$uri \$uri/ =404;
}
}
}
heredoc
A good idea is to create the webroot in the project file hierarchy since the kernel is fetched over HTTP
make sure the HTTP server serves the content of the out/kernel directory.
mkdir -p out/kernel
Start NGINX: sudo nginx -p . -c etc/nginx_config.conf
It will spin up NGINX with the custom configuration - this requires setting the prefix with -p option.
Reboot the Pi, while running tail -f /var/log/nginx/rpi_access.log, you should see a GET /kernel/linux5.10_rpi request being logged.
Conclusion¶
To integrate this into an existing build system, scripts will need to be written but before writting them, it is necessary to make it work on other networks without modifying the U-Boot boot script each time..
The next article will focus on configuring and compiling the Raspberry Pi stable kernel, to make it mount the rootfs over the network - so stay tuned!