Raspberry Pi development environment

This guide will cover how I've built a Raspberry Pi kernel development setup based around a Linux laptop.

SD cards are highly unreliable, and you'll be uncleanly powering off the Pi if you're doing kernel work, so avoiding SD cards is key. As of Raspberry Pi 3, the device will come up and attempt to DHCP and TFTP down all its bootloader and the kernel, so we're going to build around that.

I also want the setup to be portable, since I often work from coffee shops, and I want the Pi to be separated from whatever network the laptop is on but with access to the Internet. Thus, I'll have my network be:

+--------------+                +----------------------------+                +----------+
| Raspberry Pi | <--ethernet--> | amd64 Debian Linux desktop | <--wireless--> | Internet |
+--------------+                +----------------------------+                +----------+

Setting up the chroot

I haven't done this for a bit, so these steps will be a bit incomplete. First, build an armhf chroot (mine is /home/anholt/rpi2) using debootstrap.

Set its /etc/fstab to:

/dev/nfs        /               nfs4    defaults,rw      0 0
debugfs         /debug          debugfs mode=0755        0 0

Also run:

sudo mkdir /home/anholt/rpi2/debugfs
git clone https://github.com/raspberrypi/firmware raspberrypi-firmware
sudo cp -R raspberrypi-firmware/boot/ /home/anholt/rpi2/  

Edit /home/anholt/rpi2/boot/cmdline.txt:

# This setup is for downstream kernels:
dwc_otg.lpm_enable=0 console=ttyS0,115200 root=/dev/nfs rootfstype=nfs ip=dhcp nfsroot=10.42.0.1:/home/anholt/rpi2,tcp rootwait

# This setup is for upstream kernels:
# console=ttyS0,115200 root=/dev/nfs rootfstype=nfs ip=dhcp nfsroot=10.42.0.1:/home/anholt/rpi2,tcp rootwait cma=256M@256M

# Add this for early boot console output on Pi3.
# earlycon=uart8250,mmio32,0x3f215040 earlyprintk

# Add this for DRM debug information if you're submitting vc4 bug reports
# drm.debug=0x1e

# If you're using a Pi2 that's been updated for netbooting, change ttyS0 to ttyAMA0.

Set /home/anholt/rpi2/boot/config.txt to:

[pi3]
# We want serial console output!
enable_uart=1

Setting up the network

Use nm-connection-editor to define a new ethernet connection for the Pi hosting. In IPv4 Settings, set Method to "Shared to other computers".

Create a new file /etc/NetworkManager/dnsmasq-shared.d/tftp.conf:

# Skip DNS hosting, we're just trying to serve DHCP here.
port=0
# It's really nice to be able to debug what the firmware is doing
log-dhcp
# dnsmasq will be our tftp server
enable-tftp
tftp-root=/home/anholt/rpi2/boot
pxe-service=0,"Raspberry Pi Boot"

Now run:

sudo /etc/init.d/network-manager restart

Building the kernel

Get your appropriate linux tree, either upstream or downstream. Set up your defconfig as follows:

• Downstream BCM2835 32-bit: ARCH=arm make bcmrpi_defconfig
• Downstream BCM2836/BCM2837 32-bit: ARCH=arm make bcm2709_defconfig
• Upstream BCM2835 32-bit: ARCH=arm make bcm2835_defconfig
• Upstream BCM2836/BCM2837 32-bit: ARCH=arm make multi_v7_defconfig
• Upstream BCM2837 64-bit: ARCH=arm64 make defconfig

Once you have the .config in place, you can build and install the kernel so that the bootloader will find it using the following script:

set -e

if ! test -e .config; then
	echo "Needs make ARCH=arm multi_v7_defconfig or ARCH=arm64 defconfig"
	exit 1
fi

if grep -q CONFIG_ARM64=y .config; then
	export ARCH=arm64
	export CROSS_COMPILE="ccache /usr/bin/aarch64-linux-gnu-"
else
	export ARCH=arm
	export CROSS_COMPILE="ccache /usr/bin/arm-linux-gnueabihf-"
fi

export INITRD=No
export INSTALL_PATH=/home/anholt/rpi2/boot
export INSTALL_MOD_PATH=/home/anholt/rpi2

# Fix up arm64 defconfig to not fill up my disk so badly.
./scripts/config -d CONFIG_LOCALVERSION_AUTO

# Fix up arm64 defconfig for netbooting.
./scripts/config -e CONFIG_USB_USBNET
./scripts/config -e CONFIG_USB_NET_SMSC95XX

make

VERSION=`make -f Makefile kernelrelease 2> /dev/null`

sudo -E make zinstall dtbs_install

if grep 'CONFIG_MODULES=y' .config > /dev/null; then
	sudo -E make modules_install
fi

sudo perl -pi -e "s|^dtoverlay=vc4-kms-v3d.*|#dtoverlay=vc4-kms-v3d|" \
	   $INSTALL_PATH/config.txt

if test $ARCH = arm64; then
	# Upstream 64-bit kernel
	sudo ln -f $INSTALL_PATH/vmlinuz-$VERSION $INSTALL_PATH/kernel8.img
	sudo ln -f $INSTALL_PATH/dtbs/$VERSION/broadcom/bcm2837-rpi-3-b.dtb \
		   $INSTALL_PATH/bcm2710-rpi-3-b.dtb
else
	if test -e arch/arm/boot/dts/bcm2709.dtsi; then
		# Downstream 32-bit kernel
		sudo ln -f $INSTALL_PATH/dtbs/$VERSION/bcm2709-rpi-2-b.dtb \
			   $INSTALL_PATH/bcm2709-rpi-2-b.dtb
		sudo ln -f $INSTALL_PATH/dtbs/$VERSION/bcm2710-rpi-3-b.dtb \
			   $INSTALL_PATH/bcm2710-rpi-3-b.dtb
		sudo perl -pi -e "s|^#dtoverlay=vc4-kms-v3d.*|dtoverlay=vc4-kms-v3d|" \
			   $INSTALL_PATH/config.txt
	else
		# Upstream 32-bit kernel
		sudo ln -f $INSTALL_PATH/dtbs/$VERSION/bcm2836-rpi-2-b.dtb \
			   $INSTALL_PATH/bcm2709-rpi-2-b.dtb
		sudo ln -f $INSTALL_PATH/dtbs/$VERSION/bcm2837-rpi-3-b.dtb \
			   $INSTALL_PATH/bcm2710-rpi-3-b.dtb
	fi
	sudo ln -f $INSTALL_PATH/vmlinuz-$VERSION $INSTALL_PATH/kernel7.img
	# Remove the 64-bit kernel's link, so we default to 32-bit boot.
	sudo rm -f $INSTALL_PATH/kernel8.img
fi

echo "Installed $VERSION"

Testing

Start sudo tail -f /var/log/daemon.log in a terminal, connect the Raspberry Pi over ethernet, and power on the Pi. Once the power comes up, your network should be selected by NetworkManager (check that now, select the right one and restart the Pi if not). After several seconds, the Pi should start hitting DHCP, and the daemon.log will show:

Jan 27 13:49:05 eliezer dnsmasq-dhcp[11630]: 653460281 DHCPDISCOVER(eth6) b8:27:eb:61:75:0b
Jan 27 13:49:05 eliezer dnsmasq-dhcp[11630]: 653460281 DHCPOFFER(eth6) 10.42.0.70 b8:27:eb:61:75:0b
...
[... wait a bit longer...]
...
Jan 27 13:49:05 eliezer dnsmasq-tftp[11630]: file /home/anholt/rpi2/boot/bootsig.bin not found
Jan 27 13:49:05 eliezer dnsmasq-tftp[11630]: sent /home/anholt/rpi2/boot/bootcode.bin to 10.42.0.70

Expect to see it probe for several files that don't exist (like the arm stubs). Also, it will report an early terminate on the kernel, then load it fully a bit later -- this is it probing for what files the server has.