Building an Ubuntu image for ARM "from scratch"

When development is just starting, it is often still not clear which packages will go into the target rootfs.

In other words, it's too early to grab onto LFS, buildroot or yocto (or something else), but you already need to start. For the rich (I have 4GB eMMC on pilot samples), there is a way to distribute a distribution kit to developers that will allow them to quickly deliver something that is missing at the moment, and then we can always collect package lists and form a list for the target rootfs.

This article is not new and is a simple copy-paste instruction.

The purpose of the article is to build Ubuntu rootfs for ARM boards (in my case based on Colibri imx7d).

Building an image

We collect the target rootfs for replication.

Unpack Ubuntu Base

We choose the release ourselves based on the need and our own preferences. Here I have given 20.

$ mkdir ubuntu20
$ cd ubuntu20
$ mkdir rootfs
$ wget http://cdimage.ubuntu.com/ubuntu-base/releases/20.04/release/ubuntu-base-20.04-base-armhf.tar.gz
$ tar xf ubuntu-base-20.04-base-armhf.tar.gz -C rootfs

Verifying BINFMT support in the kernel

If you have a common distribution, then there is BINFMT_MISC support and everything is configured, if not, then I'm sure you know how to enable BINFMT support in the kernel.

Make sure BINFMT_MISC is enabled in the kernel:

$ zcat /proc/config.gz | grep BINFMT
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_BINFMT_SCRIPT=y
CONFIG_BINFMT_MISC=y

Now we need to check the settings:

$ ls /proc/sys/fs/binfmt_misc
qemu-arm  register  status
$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm
flags: OC
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

You can register manually using, for example, this instruction.

Setting up qemu static arm

Now we need a statically built qemu instance.

!!! ATTENTION!!!
If you're planning on using a container to build something, check out:
https://sourceware.org/bugzilla/show_bug.cgi?id=23960
https://bugs.launchpad.net/qemu/+bug/1805913
Then for x86_64 host and arm guest you need to use the i386 version of qemu:
http://ftp.ru.debian.org/debian/pool/main/q/qemu/qemu-user-static_5.0-13_i386.deb

$ wget http://ftp.debian.org/debian/pool/main/q/qemu/qemu-user-static_5.0-13_amd64.deb
$ alient -t qemu-user-static_5.0-13_amd64.deb
# ΠΏΡƒΡ‚ΡŒ Π² rootfs ΠΈ имя исполняСмого Ρ„Π°ΠΉΠ»Π° Π΄ΠΎΠ»ΠΆΠ½ΠΎ ΡΠΎΠ²ΠΏΠ°Π΄Π°Ρ‚ΡŒ с /proc/sys/fs/binfmt_misc/qemu-arm
$ mkdir qemu
$ tar xf qemu-user-static-5.0.tgz -C qemu
$ file qemu/usr/bin/qemu-arm-static
qemu/usr/bin/qemu-arm-static: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=be45f9a321cccc5c139cc1991a4042907f9673b6, for GNU/Linux 3.2.0, stripped
$ cp qemu/usr/bin/qemu-arm-static rootfs/usr/bin/qemu-arm
$ file rootfs/usr/bin/qemu-arm
rootfs/usr/bin/qemu-arm: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=be45f9a321cccc5c139cc1991a4042907f9673b6, for GNU/Linux 3.2.0, stripped

Chroot

Simple script:

ch-mount.sh

#!/bin/bash

function mnt() {
    echo "MOUNTING"
    sudo mount -t proc /proc proc
    sudo mount --rbind /sys sys
    sudo mount --make-rslave sys
    sudo mount --rbind /dev dev
    sudo mount --make-rslave dev
    sudo mount -o bind /dev/pts dev/pts
    sudo chroot 
}

function umnt() {
    echo "UNMOUNTING"
    sudo umount proc
    sudo umount sys
    sudo umount dev/pts
    sudo umount dev

}

if [ "$1" == "-m" ] && [ -n "$2" ] ;
then
    mnt $1 $2
elif [ "$1" == "-u" ] && [ -n "$2" ];
then
    umnt $1 $2
else
    echo ""
    echo "Either 1'st, 2'nd or both parameters were missing"
    echo ""
    echo "1'st parameter can be one of these: -m(mount) OR -u(umount)"
    echo "2'nd parameter is the full path of rootfs directory(with trailing '/')"
    echo ""
    echo "For example: ch-mount -m /media/sdcard/"
    echo ""
    echo 1st parameter : 
    echo 2nd parameter : 
fi

We love the result:

$ ./ch-mount.sh -m rootfs/
# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
# uname -a
Linux NShubin 5.5.9-gentoo-x86_64 #1 SMP PREEMPT Mon Mar 16 14:34:52 MSK 2020 armv7l armv7l armv7l GNU/Linux

For the sake of interest, let's measure the size before and after installing the minimum (for me) set of packages:

# du -d 0 -h / 2>/dev/null
63M     /

Let's update:

# apt update
# apt upgrade --yes

Install the packages we are interested in:

# SYSTEMD_IGNORE_CHROOT=yes apt install --yes autoconf kmod socat ifupdown ethtool iputils-ping net-tools ssh g++ iproute2 dhcpcd5 incron ser2net udev systemd gcc minicom vim cmake make mtd-utils util-linux git strace gdb libiio-dev iiod

Kernel header files, modules, this is a separate conversation. Of course, we will not install the bootloader, kernel, modules, device tree through Ubuntu. They will come to us from outside, or we will assemble them ourselves, or they will be given to us by the board manufacturer, in any case, this is beyond the scope of this instruction.

To some extent, version discrepancies are acceptable, but it is better to take them from the kernel build.

# apt install --yes linux-headers-generic

Let's see what happened and a lot happened:

# apt clean
# du -d 0 -h / 2>/dev/null
770M    /

Don't forget to set a password.

Packing the image

$ sudo tar -C rootfs --transform "s|^./||" --numeric-owner --owner=0 --group=0 -c ./ | tar --delete ./ | gzip > rootfs.tar.gz

Additionally, we can install etckeeper with the autopush setting

Well, let's say we distributed our assembly, the work went on, how best to assemble then various versions of our system.

etckeeper can come to our aid.

Safety is everyone's business:

  • you can protect certain branches
  • generate a unique key for each device
  • disable force push
  • etc. ...
# ssh-keygen
# apt install etckeeper
# etckeeper init
# cd /etc
# git remote add origin ...

Set up autopush

Of course, we can create branches on the device in advance (let's make a script or a service that will work on the first start).

# cat /etc/etckeeper/etckeeper.conf
PUSH_REMOTE="origin"

And we can do smarter...

lazy way

Let's have some unique identifier, let's say the serial number of the processor (well, or MAC - serious companies buy the range):

cat / proc / cpuinfo

# cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 60.36
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 1
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 60.36
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : Freescale i.MX7 Dual (Device Tree)
Revision        : 0000
Serial          : 06372509

Then we can use it for the name of the branch we will be pushing to:

# cat /proc/cpuinfo | grep Serial | cut -d':' -f 2 | tr -d [:blank:]
06372509

Let's create a simple script:

# cat /etc/etckeeper/commit.d/40myown-push
#!/bin/sh
set -e

if [ "$VCS" = git ] && [ -d .git ]; then
  branch=$(cat /proc/cpuinfo | grep Serial | cut -d':' -f 2 | tr -d [:blank:])
  cd /etc/
  git push origin master:${branch}
fi

And that's it - after a while we can see the changes and generate a list of packages for the target firmware.

Recommended materials

BINFMT_MISC
Kernel Support for miscellaneous Binary Formats (binfmt_misc)
Compiling with qemu user chroot
Building Ubuntu rootfs for ARM
How to create a custom Ubuntu live from scratch
crossdev qemu-static-user-chroot
etckeeper

getdents64 problem

readdir() returns NULL (errno=EOVERFLOW) for 32-bit user-static qemu on 64-bit host
Ext4 64 bit hash breaks 32 bit glibc 2.28+
compiler_id_detection fails for armhf when using QEMU user-mode emulation
CMake doesn't work properly under qemu-arm

Source: habr.com