Raspberry Pi/Minimal musl+busybox cross building

Raspberry Pi 1 (A/B/A+/B+/zero/zero w) musl/busybox environment.

Build your own minimal system using libressl, musl-libc and busybox.

Note
This page is a work in progress and has some formatting errors and a few bits of missing information. Please contribute corrections to it if necessary. UPDATE-2018-10-20: An example Raspberry Pi 3B project based on these instructions can be found here: https://github.com/armtux/gentoo-rpi3b-radio/releases/tag/0.1-beta1

This entire process was done within an official gentoo-hardened stage3 chroot with USE="libressl", updated to ACCEPT_KEYWORDS="~amd64" (if you don't want to do this, set ~amd64 keyword individually where needed to obtain the same results from instructions below). Follow the Gentoo Handbook if you haven't already. Please read everything twice before trying - that's recommended when first installing Gentoo because it really helps! If you find mistakes below, ask for help in the #gentoo-arm or #gentoo-hardened channels on Freenode irc.

Make sure you have the following tools installed in your host system.

root #emerge -av bc layman squashfs-tools

Install crossdev.

root #emerge -av crossdev

Build the cross-toolchain for a Raspberry Pi 1 target.

Note
As of 2017-12-16 when building the cross-toolchain it'll complain that it can't autoconfigure CHOST for the target and errors out because of it. Temporary solution:

root #export CHOST=armv6j-hardfloat-linux-musleabi

Note
However, before running emerge for the host, you must unset the CHOST environment variable. for the duration of the cross-compiling process, just leave it set...

root #crossdev -t armv6j-hardfloat-linux-musleabi

Note
Toolchain versions used:

gcc-7.2.0
linux-headers-4.13
musl-1.1.18
binutils-2.29.1

Note
As of 2017-12-16, cross-musl fails, complaining about -march=native which is clearly wrong... Temporary solution: set desired target CFLAGS in the hosts's make.conf and revert to the host's cflags once crossdev completes successfully.

root #nano -w /etc/portage/make.conf

Important
CFLAGS="-O2 -mfpu=vfp -mfloat-abi=hard -march=armv6zk -mtune=arm1176jzf-s -pipe"

Note
As of 2017-12-16, cross-gcc-stage2 fails due to execinfo.h missing.

Temporary workaround: USE="-vtv" in make.conf temporarily, because /etc/portage/package.use/cross-armv6j-hardfloat-linux-musleabi gets

overwritten every time you run the crossdev command. Then, it errors out because of an error in sanitizer functions. So USE="-sanitize" as well, but these are both temporary solutions...

root #nano -w /etc/portage/make.conf

Re-run the crossdev command until it completes successfully...

When done, re-edit /etc/portage/make.conf and revert temporary changes.

root #nano -w /etc/portage/make.conf

Important
Replace CFLAGS with proper host CFLAGS. Also, remove -vtv and -sanitize from USE variables.

Add -vtv and -sanitize to cross-gcc portage configuration; now that crossdev is finished, a world update should not overwrite the portage configuration files.

root #nano -w /etc/portage/package.use/cross-armv6j-hardfloat-linux-musleabi

Add the Gentoo musl overlay with layman.

root #layman -a musl

Disable the musl overlay in host portage configuration.

root #layman -D musl

Add the musl and your local overlays to target portage configuration.

root #

echo 'PORTDIR_OVERLAY="/var/lib/layman/musl /usr/local/portage"' >> /usr/armv6j-hardfloat-linux-musleabi/etc/portage/make.conf

Edit the cross make.conf to your liking.

root #nano -w /usr/armv6j-hardfloat-linux-musleabi/etc/portage/make.conf

Important
Useful settings:

CHOST=armv6j-hardfloat-linux-musleabi
CFLAGS="-O2 -mfpu=vfp -mfloat-abi=hard -march=armv6zk -mtune=arm1176jzf-s -pipe"
USE="${ARCH} -pam libressl"
CURL_SSL="libressl"

Set the correct cross-environment make.profile

Replace the cross-environment's make.profile with the directory right before the hardened/musl armv7a profile.

root #rm /usr/armv6j-hardfloat-linux-musleabi/etc/portage/make.profile

root #ln -s /usr/portage/profiles/default/linux/musl/arm /usr/armv6j-hardfloat-linux-musleabi/etc/portage/make.profile

Replace linux-headers with raspberry pi kernel headers.

root #mkdir -p /usr/armv6j-hardfloat-linux-musleabi/usr/src

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src

root #git clone --depth=1 --branch=rpi-4.14.y https://github.com/raspberrypi/linux

Backup the downloaded kernel sources.

root #cp -r linux rpi-4.14.y

root #cd linux

root #

ARCH=arm CROSS_COMPILE=armv6j-hardfloat-linux-musleabi- make headers_install INSTALL_HDR_PATH=/usr/armv6j-hardfloat-linux-musleabi/usr

Time for the package salad buffet, but remember: this system is on a diet!

Check what the system would usually install if running emerge world, put it in a text file and remove the packages you don't want/need/are replacing with busybox, leaving only a very minimal, but functional list comprising of the customized stage3 install.

root #armv6j-hardfloat-linux-musleabi-emerge -ep @world > /root/armv6j-world

root #nano -w /root/armv6j-world

root #nano -w /usr/armv6j-hardfloat-linux-musleabi/etc/portage/profile/package.provided

Here is an example of a slimmed down list of packages that were kept. It was formatted for an upcoming emerge command which causes 33 packages to install.

FILE /root/armv6j-worldExample package list.

=virtual/libintl-0-r2
=sys-apps/gentoo-functions-0.12
=virtual/libiconv-0-r2
=app-misc/mime-types-9
=virtual/shadow-0
=net-firewall/iptables-1.6.1-r2:0/12
=sys-apps/sysvinit-2.88-r9
=sys-apps/install-xattr-0.5-r1
=virtual/pager-0
=sys-apps/busybox-1.27.2
=virtual/editor-0
=sys-apps/net-tools-1.60_p20161110235919
=sys-apps/baselayout-2.4.1-r2
=virtual/os-headers-0
=dev-libs/libressl-2.6.3-r2:0/44
=sys-fs/eudev-3.2.5
=virtual/dev-manager-0
=sys-apps/iproute2-4.14.1-r2
=sys-apps/kbd-2.0.4
=virtual/modutils-0
=net-misc/iputils-20171016_pre
=sys-apps/openrc-0.34.11
=net-misc/netifrc-0.6.0
=virtual/service-manager-0

FILE /usr/armv6j-hardfloat-linux-musleabi/etc/portage/profile/package.providedHere is an example package.provided file.

app-editors/nano-2.9.0
sys-apps/less-529
sys-apps/shadow-4.5
sys-process/psmisc-23.1
sys-apps/debianutils-4.8.3
sys-apps/opentmpfiles-0.1.3
sys-apps/util-linux-2.31
sys-kernel/linux-headers-4.13

Now, set use flags for packages you are going to be installing. Remember, it is meant to be tiny, and it'll all be running in the Raspberry Pi's RAM, so USE="-static make-symlinks savedconfig syslog" for busybox.

root #nano -w /usr/armv6j-hardfloat-linux-musleabi/etc/portage/package.use/world

FILE /usr/armv6j-hardfloat-linux-musleabi/etc/portage/package.use/worldHere is an example of minimizing use flags.

sys-apps/openrc -ncurses
sys-apps/busybox -static make-symlinks savedconfig syslog
sys-apps/file -zlib
sys-apps/kmod -zlib
sys-fs/eudev -hwdb

Configure busybox and choose the tools wanted/needed in order to have a fully bootable/usable system. Provided at the end of this document will be known working busybox configuration examples to base oneself on.

root #emerge --fetchonly busybox

root #mkdir -p /root/work/busybox

root #cd /root/work/busybox

root #tar xjf /usr/portage/distfiles/busybox-1.27.2.tar.bz2

root #cd busybox-1.27.2

root #make menuconfig

Save the busybox .config and place it in target portage configuration.

root #mkdir -p /usr/armv6j-hardfloat-linux-musleabi/etc/portage/savedconfig/sys-apps

root #cp .config /usr/armv6j-hardfloat-linux-musleabi/etc/portage/savedconfig/sys-apps/busybox

Start by rebuilding musl-libc first, then build the base system.

{{Note|As of 2017-12-17, the musl ebuild has a hardcoded file collision test which, in this process, is to be commented out in your local overlay, which may be wrong. Possible alternative: --buildpkgonly Look inside the if conditional for cross-* stuff and comment out [[ -e "${D}"/lib/ld-musl-${arch}.so.1 ]] || die in the src_install section of the ebuild.}}

root #mkdir -p /usr/local/portage/sys-libs

root #cp -a /usr/portage/eclass /usr/local/portage/eclass

root #cp -a /usr/portage/sys-libs/musl /usr/local/portage/sys-libs/musl

root #nano -w /usr/local/portage/sys-libs/musl/musl-1.1.18.ebuild

root #ebuild /usr/local/portage/sys-libs/musl/musl-1.1.18.ebuild digest

root #armv6j-hardfloat-linux-musleabi-emerge -av sys-libs/musl

Now, emerge the packages from the minimal list. If you find errors, please make bug reports and/or see if you can find patches for your issues and submit them in your bug reports.

Note
As of 2017-12-17, libressl fails because it can't link to musl. Solution: env/package.env stuff.

root #

mkdir -p /usr/armv6j-hardfloat-linux-musleabi/etc/portage/env /usr/armv6j-hardfloat-linux-musleabi/etc/portage/package.env

root #

echo 'LDFLAGS="${LDFLAGS} -L/usr/armv6j-hardfloat-linux-musleabi/usr/lib"' > /usr/armv6j-hardfloat-linux-musleabi/etc/portage/env/muslpath.conf

root #echo 'dev-libs/libressl muslpath.conf' > /usr/armv6j-hardfloat-linux-musleabi/etc/portage/package.env/libressl

root #armv6j-hardfloat-linux-musleabi-emerge -av `cat /root/armv6j-world`

Now wait. musl-libc is built twice for ease of use, once with crossdev and once with cross-emerge, as usually one would not be built, but once compiling has completed the binary package that this generates will be useful, along with all of the other binary packages.

Start building your squashfs root tree

Now that most of the system is built, time to emerge the extra sotware that your Raspberry Pi project will require. For the sake of simplifying these instructions, no extra packages will be installed and the instructions will skip to the next step: creating a slimmed down version of your cross-built environment using the binary packages that cross-emerge generated.

root #mkdir /usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs

root #

armv6j-hardfloat-linux-musleabi-emerge --root=/usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs --config-root=/usr/armv6j-hardfloat-linux-musleabi -Kav `cat /root/armv6j-world`

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs

root #rm -rf usr/include var/db/pkg

Install missing gcc libraries.

root #mkdir usr/lib/gcc

root #cp -a /usr/lib/gcc/armv6j-hardfloat-linux-musleabi usr/lib/gcc/

root #rm -rf usr/lib/gcc/*/*/include* usr/lib/gcc/*/*/plugin/include usr/lib/gcc/*/*/*.a usr/lib/gcc/*/*/*.la

root #cd lib

root #ln -s ../usr/lib/gcc/*/*/*.so ./

Set/unset services manually in /etc/runlevels/*

Note
Make sure you remove hwclock from boot and add swclock. The busybox-syslogd service is also useful. So is iptables, but writing rules manually is out of scope for this document.

Here is a runlevel example.

root #cd ../etc/runlevels

root #ln -s /etc/init.d/swclock boot/

root #rm boot/hwclock

Check what services/init scripts are available.

root #ls ../init.d

Important
As of 2017-12-17, busybox sysctl doesn't support an option used by Gentoo's sysctl init script. Solution: remove the option from the script.

root #cd ../init.d

root #cp -a sysctl sysctl.old

root #nano -w sysctl

Important
Find the sysctl --system command to read /etc/sysctl.conf and replace it with sysctl ${quiet} -p /etc/sysctl.conf

Backup the new sysctl init script just in case your change is overwritten during updates.

root #cp -a sysctl sysctl.bak

Note
Here is an example of which init scripts can be in which runlevels.

boot: bootmisc hostname iptables keymaps localmount loopback modules mtab procfs root swclock sysctl termencoding urandom
default: busybox-syslogd local
nonetwork: local
shutdown: killprocs mount-ro savecache
sysinit: devfs dmesg kmod-static-nodes sysfs udev udev-trigger

Configure everything relevant as you would a normal Gentoo system.

You can leave /etc/fstab empty though, for now. Read the gentoo handbook and see what configuration is needed before first boot if necessary. The other Gentoo Wiki Raspberry Pi pages are useful too.

At least set a root password so you can login.

root #cp -a /etc/shadow /etc/shadow.bak

root #passwd

root #grep root /etc/shadow >> /usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs etc/shadow

root #mv /etc/shadow.bak /etc/shadow

Replace the first root line with the bottom root line.

root #nano -w /usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs/etc/shadow

Complete the root skeleton of needed but missing files/directories.

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs

root #mkdir dev home media mnt opt proc root sys

root #chmod 700 root

root #cp -a /dev/null /dev/console /dev/tty /dev/loop0 /dev/random /dev/urandom dev/

root #mknod -m 660 dev/ttyAMA0 c 204 64

Configure and build the kernel.

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src/linux

root #unset CHOST

root #export KERNEL=kernel

root #ARCH=arm CROSS_COMPILE=armv6j-hardfloat-linux-musleabi- make bcmrpi_defconfig

root #ARCH=arm CROSS_COMPILE=armv6j-hardfloat-linux-musleabi- make menuconfig

You probably already know what to enable/disable, and you can probably mostly leave everything as-is, but make sure that you enable tmpfs, squashfs with xz support and overlayfs for this guide, and set the default initramfs location to /usr/armv6j-hardfloat-linux-musleabi/usr/src/initramfs

Important
Make sure you don't select xz compression for kernel or initramfs or else the kernel xz decompressor will think the xz data is corrupt and will fail to boot the kernel and halt... These instructions use gz compression instead.

root #mkdir /usr/armv6j-hardfloat-linux-musleabi/usr/src/initramfs

We have a dependency loop here. The kernel needs an initramfs, and the initramfs needs to contain kernel modules. So, first we build the kernel and install its modules into our new slimmed down/configured gentoo root.

root #ARCH=arm CROSS_COMPILE=armv6j-hardfloat-linux-musleabi- make -j4

Note
These instructions were tested with a dual-core processor with hyperthreading. Set -jN to the correct value for your processor.

root #

ARCH=arm CROSS_COMPILE=armv6j-hardfloat-linux-musleabi- INSTALL_MOD_PATH=/usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs make modules_install

Note
Building out-of-tree drivers is out of scope for these instructions.

Build the initramfs

Take a break from the kernel to create the initramfs with a newly compiled ultra-minimal static busybox binary, the /init script that uses busybox, rudimentary device nodes and a compressed squashfs image of the /usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs directory.

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src/initramfs

root #mkdir bin dev mnt proc sys

root #cp -a /dev/null /dev/console /dev/tty /dev/loop0 /dev/random /dev/urandom dev/

root #mknod -m 660 dev/ttyAMA0 c 204 64

Rebuild minimal reconfigured busybox with USE="static -syslog -make-symlinks".

root #cp /usr/armv6j-hardfloat-linux-musleabi/etc/portage/savedconfig/sys-apps/busybox /root/busybox.config

root #cd /root/work/busybox/busybox-1.27.2

root #make menuconfig

Note
It is a good idea to use the minimal busybox configuration provided below as a base.

root #cp .config ../../../busybox.mini.config

root #cp .config /usr/armv6j-hardfloat-linux-musleabi/etc/portage/savedconfig/sys-apps/busybox

root #export CHOST=armv6j-hardfloat-linux-musleabi

root #USE="static -syslog -make-symlinks" armv6j-hardfloat-linux-musleabi-emerge -1Bav busybox

Install minimal busybox in your initramfs.

root #

armv6j-hardfloat-linux-musleabi-emerge --root=/usr/armv6j-hardfloat-linux-musleabi/usr/src/initramfs --config-root=/usr/armv6j-hardfloat-linux-musleabi -1Kav busybox

Delete unneeded files from initramfs.

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src/initramfs

root #rm -rf etc var usr tmp

Make sure you statically linked busybox.

root #file /usr/armv6j-hardfloat-linux-musleabi/usr/src/initramfs/bin/busybox

Create a squashfs image of your minimal root for inclusion in initramfs.

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src/squashfs

root #mksquashfs . ../initramfs/squash -comp xz -b 1048576 -Xbcj arm -Xdict-size 1048576

Your squashfs image should be somewhere around 5 to 10 megabytes if you didn't install anything other than the list of packages ealier in these instructions.

Write your initramfs init script and make it executable.

root #cd ../initramfs

root #nano -w init

root #chmod 700 init

Note
A small example init script is provided below.

Regenerate an initramfs built into the kernel, including the entire OS.

root #cd ../linux

root #rm usr/initramfs_data.cpio.gz

root #unset CHOST

root #ARCH=arm CROSS_COMPILE=armv6j-hardfloat-linux-musleabi- make -j4

Complete the installation.

Your cross-gentoo build is complete, all in only one small file! One last thing that needs doing is putting it on a FAT32 SD card partition along with the Raspberry Pi firmware.

root #mkdir /root/work/bootloader

root #cd /root/work/bootloader

root #git clone --depth=1 --branch=next https://github.com/raspberrypi/firmware

root #mount /dev/SDCARD /media/sdcard

root #cd /usr/armv6j-hardfloat-linux-musleabi/usr/src/linux

root #cp arch/arm/boot/zImage /media/sdcard/kernel.img

root #mkdir /media/sdcard/overlays

root #cp arch/arm/boot/dts/overlays/*.dtbo /media/sdcard/overlays/

root #cp arch/arm/boot/dts/*.dtb /media/sdcard/

root #cd /root/work/bootloader/firmware/boot

root #cp -r boot* fixup* start* /media/sdcard/

root #umount /media/sdcard

Once you see that the system is booting, it is safe to remove the SD card from the Raspberry Pi. This setup allows for SD card swapping, because you already loaded the entire operating system into the Raspberry Pi's RAM.

Troubleshooting

System busybox config: https://gist.github.com/anonymous/d9a10a8d380a313e54f4435be15c21c1

Minimal busybox config: https://gist.github.com/anonymous/749fd5f60d0c837cb42bfd921048ac71

Example init script: https://gist.github.com/anonymous/8787dd24b6fc7e061b6637d9f61c2f8f

This article is issued from Gentoo. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.