Unofficial Poudriere technical resources

BSDRP use a heavy customized nanobsd script that include package generation. This code need to be adapted each time the port build infrastructure change.

Why not using the new shinny “poudriere image” feature that will avoid to re-invent the wheel ?

Here are a list of docs and tutorials about Poudriere:

• Poudriere wiki: Official Wiki
• Building Packages with Poudriere : Presentation page on FreeBSD Handbook
• poudriere man page : The man page

It's a shell script used for build package in a clean (jail) environment. But once you get a clean jail in one side, and a list of fresh generated package in other side, why not mix them together in a “nanobsd like” firmware ?

This is the “poudriere image” feature includes in poudriere-git-3.3.99.20220713 that is presented here.

Poudriere can generate multiple “image” types (default is iso+zmfs):

• iso: ISO 9660 format image
  • iso+mfs: ISO 9660 variant where the root filesystem is MFS mounted
  • iso+zmfs (default): ISO 9660 variant where the root filesystem is LZ77 compressed and is MFS mounted.
• usb: GPT-layout prepared UFS2 image containing a UEFI boot loader.
  • usb+mfs : variant where the root filesystem is MFS mounted
  • usb+zmfs: variant where the root filesystem is LZ77 compressed and is MFS mounted.
• rawdisk: raw UFS2, softupdates-enabled, disk image
• zrawdisk: raw ZFS disk image
• tar: XZ-compressed tarball
• firmware: NanoBSD style image with a GPT partitions and a UEFI boot loader
• rawfirmware: raw disk image (Update image that includes only one system partition of the firmware?)
• embedded: u-boot ready embedded image
• zsnapshot: zfs snapshot full and incremental to be used in a jail

Using poudriere from a ZFS is not mandatory but strongly advised.

1. Install poudriere and configure it:

   sudo pkg install poudriere-devel
   echo "ZPOOL="`zpool list -H | cut -f1` | sudo tee -a /usr/local/etc/poudriere.conf

2. Create a poudriere jail WITH a GENERIC kernel (by default kernel is not build & installed), here named “router”:

   sudo poudriere jail -c -j router -v 14.2-RELEASE -K GENERIC

3. Create a port-tree using “poudriere ports”:

   sudo poudriere ports -c -p router_ports

4. Generate list of ports to be build & added into the firmware image:

   cat > ~/router-pkglist <<EOF
   sysutils/tmux
   net/frr10
   net/bird2
   net/mpd5
   EOF

5. Build the ports (ie: generate binary packages) from the corresponding jail using “poudriere bulk”:

   sudo poudriere bulk -j router -p router_ports -f ~/router-pkglist

6. Generate your disk image (4Gb total, because 2 systems partitions of 2Gb) using “poudriere image”:

   sudo poudriere image -t firmware -j router -s 4g -p router_ports -h router -n router -f ~/router-pkglist
   (...)
   [00:00:15] Creating ESP image
   [00:00:15] ESP Image created
   [00:00:21] Image available at: /usr/local/poudriere/data/images/router.img

Start by checking poudriere firmware file size:

# ls -alh /usr/local/poudriere/data/images/router.img
-rw-r--r--  1 root  wheel   3.8G Jul 22 19:24 /usr/local/poudriere/data/images/router.img

We've obtained a 3.8GiB disk image file… which fit into a marketed-size 4GB flash disk.

The resulting images partition layout will have the same behavior than a nanobsd:

• 10M GPT partition with EFI bootloader (nanobsd uses a MBR scheme with BIOS bootloader)
• first 991M (calculated from user input) system partition called gpt/${IMAGENAME}1
• second system partition called gpt/${IMAGENAME}2
• configuration configuration partition (hard-coded to 32M) called gpt/cfg
• data partition (hard-coded to 32M) called gpt/data

# mdconfig -a -t vnode -f /usr/local/poudriere/data/images/router.img
md0
# gpart show -l md0
=>      4  7995515  md0  GPT  (3.8G)
        4    20480    1  (null)  (10M)
    20484      123    2  (null)  (62K)
    20607  3921920    3  router1  (1.9G)
  3942527  3921920    4  router2  (1.9G)
  7864447    65536    5  cfg  (32M)
  7929983    65536    6  data  (32M)
  
# mount /dev/gpt/router1 /mnt/
# df -h /mnt
Filesystem          Size    Used   Avail Capacity  Mounted on
/dev/gpt/router1    1.8G    1.5G    103M    94%    /mnt

Poudriere image is correctly compliant to a nanobsd /etc & /var ramdisk:

root@router:~ # mount
/dev/gpt/router1 on / (ufs, local, read-only)
devfs on /dev (devfs, local, multilabel)
/dev/md0 on /etc (ufs, local)
/dev/md1 on /var (ufs, local)

Comparing to a standard nanobsd:

[root@nanobsd]~# mount
/dev/ufs/NANOs1a on / (ufs, local, read-only)
devfs on /dev (devfs, local)
/dev/md0 on /etc (ufs, local)
/dev/md1 on /var (ufs, local)

poudriere's fstab is using gpt labels, while nanobsd one using ufs labels:

root@router:~ # cat /etc/fstab
/dev/gpt/router1 / ufs ro 1 1
/dev/gpt/cfg  /cfg  ufs rw,noatime,noauto        2 2
/dev/gpt/data /data ufs rw,noatime,noauto,failok 2 2

[root@nanobsd]~# cat /etc/fstab
/dev/ufs/NANOs2a / ufs ro 1 1
/dev/ufs/NANOs3 /cfg ufs rw,noatime,noauto 2 2
/dev/ufs/NANOs4 /data ufs rw,noauto,failok 2 2

nanobsd configuration file is a shell script,and BSDRP use a highly nanobsd customized configuration file generated by a wrapper (make.sh).

The challenge of migrating the customization done on nanobsd to poudriere image can be resumed here:

List of mandatory patches for poudriere in Pull-request review:

• Overlay permission bug on the final image

Merged patches:

• Optimize disk space for space
• Need to replace loader.efi by gptboot.efi to use GPT attribute bootonce and bootme
• Need to generate the upgrade image too
• Fix default permissions to cfg and data partitions
• Add bytes to bibytes conversion regarding image size
• Generate /etc/os-release
• Fix excludelist path

How to configure poudriere to generate a BSDRP firmware image.

To be able to reproduce the highly customized BSDRP firmware image, we need multiples configuration files.

Poudriere will requiere jail and port so prepending j and p to avoid confusion:

• BSDRPj for the jail name
• BSDRPp for the port tree name

We need to start creating a set of configuration files, named prefixed with the name BSDRP-:

• poudriere.d/BSDRPj-src.conf : Include all src.conf parameters used for reference jail buildworld/installworld
• poudriere.d/BSDRPj-make.conf : Include all common ports parameters (and ports build options)
• poudriere.d/image-BSDRPj-src.conf : parameters added for installworld (followed by a delete-old)

Then need other configuration files:

• BSDRP-pkglist : List of packages to be build and included in the final image
• A kernel configuration file: I'm using the BSDRP amd64 configuration
• excluded.files: A list of file we want to be exculed during the installworld
• overlaydir/usr/local/etc/pkg.conf: with a FILES_IGNORE_GLOB list that will exclude files from being extracted during packages installation

The previous section of the NanoBSD configuration files found in variables CONF_BUILD and CONF_WORLD in file BSDRP/BSDRP.nano should be copied in this file.

Notice this jail will be used to build the port, so compiler should be kept here.

The BSDRPj-src.conf is on github.

Allow to ADD WITHOUT_ knob that will be removed during installworld into the final image.

This is where we remove compiler and other no-more used part.

The image-BSDRPj-src.conf is on github.

This file contains build parameters for the ports.

The BSDRPj-make.conf is on github.

This file includes the list of package to be builded and added to the final image.

The BSDRP-pkglist is on github.

List of files/directory that WITHOUT_ wasn't able to prevent to be on the final image.

The excluded.files is on github.

When customizing ports options, some could not be disabled but we could configure pkg to not install some files from packages while extracting them.

The customized pkg.conf is on github.

The simplest solution is to re-use already existing BSDRP patched source tree: specific kernel configuration files can be installed into these source tree and use after.

Start by only patching BSDRP sources (sources and ports) using the BSDRP makefile:

make patch-sources

There are now 2 sources ready patched:

• obj/FreeBSD (including BSDRP specific kernel configuration file)
• obj/ports

poudriere jail -e poudriere.etc -c -j BSDRPj -b -m src=obj/FreeBSD -K amd64

Command line details:

• -b: Build from source
• -c: create a jail
• -j: SHORT name for the jail (I can't use BSDRP-amd64-10.3R here because later it will generate a long directory name and long name aren't well supported)
• -e: Load all configurations files from ./poudriere.etc
• -m src=: Path to the patched source branch we want to use
• -K: The kernel configuration file (was copied here during patching BSDRP code tree)

Now we need to create a port-tree using the patched existing port tree:

poudriere ports -e poudriere.etc -c -p BSDRPp -m null -M obj/ports

poudriere native role, we just give the jail name and port-tree name to use then the list of packages.

poudriere bulk -e poudriere.etc -j BSDRPj -p BSDRPp -f /usr/local/etc/poudriere.d/BSDRP-pkglist.common

Here I'm instructing to build a 2GB image using the previous sets, jail, port-tree.

poudriere image -t firmware -s 2g \
    -j BSDRPj -p BSDRPp -n BSDRP -h router.bsdrp.net \
    -c BSDRP/Files/ \
    -f poudriere.etc/poudriere.d/BSDRP-pkglist \
    -X poudriere.etc/poudriere.d/excluded.files \
    -A poudriere.etc/poudriere.d/post-script.sh

Command line explanation:

• -s: Size of full image size (same as the flash media)
• -j: The jail we just generated
• -p: The poudriere port tree, we just generated its packages
• -n: Image name, will be use as the partition name too
• -h: Hostname configured on the image
• -c: Directory tree to be copied on the image (that should include a pkg.conf with the FILES_IGNORE_GLOB)
• -f: List of package to be installed on the image
• -X: List of file to be excluded from the installworld
• -A: The post-script, executed at the end, to do the last image tuning (like an mtree or other)