Unofficial Poudriere technical resources

BSDRP use a heavy customized nanobsd script that include package generation. This code need to be adapted each time the port build infrastructure change.

Why not using the new shinny “poudriere image” feature that will avoid to re-invent the wheel ?

Here are a list of docs and tutorials about Poudriere:

• Poudriere wiki: Official Wiki
• Building Packages with Poudriere : Presentation page on FreeBSD Handbook
• poudriere man page : The man page

It's a shell script used for build package in a clean (jail) environment. But once you get a clean jail in one side, and a list of fresh generated package in other side, why not mix them together in a “nanobsd like” firmware ?

This is the “poudriere image” feature includes in poudriere-devel-3.2.99.20180601 that is presented here.

Poudriere can generate multiple “image” types (default is iso+zmfs):

• iso: An ISO 9660 format image
  • iso+mfs: An ISO 9660 variant where the root filesystem is MFS mounted
  • iso+zmfs (default): An ISO 9660 variant where the root filesystem is LZ77 compressed and is MFS mounted.
• usb: A GPT-layout prepared UFS2 image containing a UEFI boot loader.
  • usb+mfs : A variant where the root filesystem is MFS mounted
  • usb+zmfs: A variant where the root filesystem is LZ77 compressed and is MFS mounted.
• firmware: A NanoBSD style image with a GPT partitions and a UEFI boot loader
  • rawfirmware: A raw disk image (Update image that includes only one system partition of the firmware?)
• rawdisk: A raw UFS2, softupdates-enabled, disk image
• zrawdisk: A raw ZFS disk image
• tar: An XZ-compressed tarball
• zsnapshot: Create a zfs snapshot full and incremental to be used in a jail

Using poudriere from a ZFS is not mandatory but strongly advised.

1. Install poudriere and configure it:

   pkg install poudriere-devel
   echo "ZPOOL="`zpool list -H | cut -f1` >> /usr/local/etc/poudriere.conf

2. Create a poudriere jail WITH a GENERIC kernel (by default kernel is not build & installed):

   poudriere jail -c -j router -v 12.0-RELEASE -K GENERIC

3. Create a port-tree using “poudriere ports”:

   poudriere ports -c -p router-ports

4. Generate list of ports to be build & added into the firmware image:

   cat > ~/router-pkglist <<EOF
   sysutils/tmux
   net/frr6
   net/bird
   net/mpd5
   EOF

5. Build them using “poudriere bulk”:

   poudriere bulk -j router -p router-ports -f ~/router-pkglist

6. Generate your disk image (4Gb total, because 2 systems partitions of 2Gb) using “poudriere image”:

   poudriere image -t firmware -j router -s 4g -p router-ports -h router -n router -f ~/router-pkglist
   (...)
   [00:00:40] Image available at: /usr/local/poudriere/data/images//router.img

Start by checking poudriere firmware file size:

root@lame4:~ # # ls -alh /usr/local/poudriere/data/images/router.img
-rw-r--r--  1 root  wheel   3.6G Feb 19 23:48 /usr/local/poudriere/data/images/router.img

We've obtained a 3.6GiB disk image file… which fit into a marketed-size 4GB flash disk.

The resulting images partition layout will have the same behavior than a nanobsd:

• GPT partition with EFI bootloader (nanobsd uses a MBR scheme with BIOS bootloader)
• first 991M (calculated from user input) system partition called gpt/${IMAGENAME}1
• second system partition called gpt/${IMAGENAME}2
• configuration configuration partition (hard-coded to 32M) called gpt/cfg
• data partition (hard-coded to 32M) called gpt/data

root@router:~ # gpart show -l
=>      4  4191926  vtbd0  GPT  (2.0G)
        4     1600      1  (null)  (800K)
     1604      118      2  (null)  (59K)
     1722  2029568      3  router1  (991M)
  2031290  2029568      4  router2  (991M)
  4060858    65536      5  cfg  (32M)
  4126394    65536      6  data  (32M)

=>      4  4191926  diskid/DISK-BHYVE-59FF-34DA-1F19  GPT  (2.0G)
        4     1600                                 1  (null)  (800K)
     1604      118                                 2  (null)  (59K)
     1722  2029568                                 3  router1  (991M)
  2031290  2029568                                 4  router2  (991M)
  4060858    65536                                 5  cfg  (32M)
  4126394    65536                                 6  data  (32M)

root@router:~ # df -h
Filesystem          Size    Used   Avail Capacity  Mounted on
/dev/gpt/router1    1.8G    1.2G    424M    75%    /
devfs               1.0K    1.0K      0B   100%    /dev
tmpfs                32M    3.2M     29M    10%    /etc
tmpfs                32M    2.7M     29M     8%    /var

Poudriere image is correctly compliant to a nanobsd /etc & /var ramdisk:

root@router:~ # mount
/dev/gpt/router1 on / (ufs, local, read-only)
devfs on /dev (devfs, local, multilabel)
/dev/md0 on /etc (ufs, local)
/dev/md1 on /var (ufs, local)

Comparing to a standard nanobsd:

[root@nanobsd]~# mount
/dev/ufs/NANOs1a on / (ufs, local, read-only)
devfs on /dev (devfs, local)
/dev/md0 on /etc (ufs, local)
/dev/md1 on /var (ufs, local)

fstab is compliant to nanobsd too:

root@router:~ # cat /etc/fstab
/dev/gpt/router1 / ufs ro 1 1
/dev/gpt/cfg  /cfg  ufs rw,noatime,noauto        2 2
/dev/gpt/data /data ufs rw,noatime,noauto,failok 2 2

[root@nanobsd]~# cat /etc/fstab
/dev/ufs/NANOs2a / ufs ro 1 1
/dev/ufs/NANOs3 /cfg ufs rw,noatime,noauto 2 2
/dev/ufs/NANOs4 /data ufs rw,noauto,failok 2 2

nanobsd configuration file is a shell script,and BSDRP use a highly nanobsd customized configuration file generated by a wrapper (make.sh).

The challenge of migrating the customization done on nanobsd to poudriere image can be resumed here:

BSDRP customized nanobsd usage	poudriere image	Purpose
Support differents src.conf: One for buildworld and another for installword	src.conf used for buildworld and installworld, image-JAILNAME[-SET]-src.conf added for installworld/delete-old. Allow to use the -X exclude_list too	Using WITHOUT_INCLUDES or WITHOUT_DEBUG just for installworld but not for buildworld
Downloading a specific source revision of a stable/releng/head	use svn url@rev (or custom source tree with '-b -m src=/usr/src')	Allow reproducing the same build on different systems
Build a specific kernel file	Just had to install custom kernel int the custom source tree	Customized kernel
Build only list of kernel modules	How to do that?	Customized module list
Advanced scripting for building non-ports softwares	Not available, no idea of how to cleanly add this feature	Need to build some of /usr/src/tools
Advanced tunning of final image	Need to add a hook	Generating a mtree (for host-IDS), creating specific users
Building ports using own (outdated) embedded script	NATIVE role of poudriere	Adding ports to image
System upgrade by changing MBR active mode on system partition	System upgrade by setting “bootonce” attribute on system partition (and need to be removed from older?)	Upgrading system partition (new/old)

How to manually (without a “wrapper” script) configure poudriere for generating a BSDRP firmware image.

We need to start creating a set of configuration files, named BSDRP:

• poudriere.d/BSDRP-src.conf : Include all src.conf parameters used for reference jail buildworld/installworld
• poudriere.d/BSDRP-make.conf : Include all common ports parameters
• poudriere.d/BSDRP-options/$CATEGORY_$PORTNAME/options : Include specific options for $CATEGORY/$PORTNAME
• poudriere.d/image-$JAILNAME[-$SETNAME]-src.conf : parameters added for installworld (followed by a delete-old)

Then need other configuration files:

• BSDRP-pkglist : List of packages to be build and included in the final image

The previous section of the NanoBSD configuration files found in variables CONF_BUILD and CONF_WORLD in file BSDRP/BSDRP.nano should be copied in this file.

As for the BSDRP example, this give this file:

MALLOC_PRODUCTION=
BOOT_BOOT0_COMCONSOLE_SPEED=0
WITHOUT_ACCT=
WITHOUT_AMD=
WITHOUT_APM=
WITHOUT_ASSERT_DEBUG=
WITHOUT_ATF=
WITHOUT_ATM=
WITHOUT_AUDIT=
WITHOUT_AUTHPF=
WITHOUT_AUTOFS=
WITHOUT_BHYVE=
WITHOUT_BLUETOOTH=
WITHOUT_BOOTPARAMD=
WITHOUT_BOOTPD=
WITHOUT_BSDINSTALL=
WITHOUT_CALENDAR=
WITHOUT_CCD=
WITHOUT_CTM=
WITHOUT_CVS=
WITHOUT_DICT=
WITHOUT_EE=
WITHOUT_EXAMPLES=
WITHOUT_FINGER=
WITHOUT_FLOPPY=
WITHOUT_FREEBSD_UPDATE=
WITHOUT_GAMES=
WITHOUT_GCOV=
WITHOUT_GNUCXX=
WITHOUT_GPIB=
WITHOUT_GPIO=
WITHOUT_GROFF=
WITHOUT_HAST=
WITHOUT_HTML=
WITHOUT_ICONV=
WITHOUT_INFO=
WITHOUT_IPX=
WITHOUT_IPX_SUPPORT=
WITHOUT_ISCSI=
WITHOUT_LIB32=
WITHOUT_LINT=
WITHOUT_LOADER_FIREWIRE=
WITHOUT_LOADER_GELI=
WITHOUT_LOCATE=
WITHOUT_LPR=
WITHOUT_NCP=
WITHOUT_NDIS=
WITHOUT_NETCAT=
WITHOUT_NIS=
WITHOUT_NLS=
WITHOUT_NLS_CATALOGS=
WITHOUT_NS_CACHING=
WITHOUT_PC_SYSINSTALL=
WITHOUT_PORTSNAP=
WITHOUT_PROFILE=
WITHOUT_QUOTAS=
WITHOUT_RBOOTD=
WITHOUT_RCMDS=
WITHOUT_RCS=
WITHOUT_RESCUE=
WITHOUT_ROUTED=
WITHOUT_SENDMAIL=
WITHOUT_SERVICESDB=
WITHOUT_SHAREDOCS=
WITHOUT_SVNLITE=
WITHOUT_SYSCONS=
WITHOUT_SYSINSTALL=
WITHOUT_TALK=
WITHOUT_TESTS=
WITHOUT_TESTS_SUPPORT=
WITHOUT_TFTP=
WITHOUT_TIMED=
WITHOUT_UNBOUND=
WITHOUT_USB_GADGET_EXAMPLES=
WITHOUT_WIRELESS=
WITHOUT_WPA_SUPPLICANT_EAPOL=
WITHOUT_ZFS=
WITH_IDEA=
WITH_OFED=
WITH_DEBUG_FILES=
WITH_REPRODUCIBLE_BUILD=
WITH_DIRDEPS_BUILD=
WITH_RETPOLINE=

Allow to ADD WITHOUT that will be removed during installworld

cat <<EOF > /usr/local/etc/poudriere.d/image-BSDRP-src.conf
WITHOUT_DEBUG_FILES=
WITHOUT_TOOLCHAIN=
WITHOUT_INCLUDES=
EOF

This file contains configuration parameters common for all ports.

For BSDRP, it is simple:

cat <<EOF > /usr/local/etc/poudriere.d/BSDRP-make.conf
OPTIONS_UNSET+= DOCS NLS X11 EXAMPLES
EOF

If the ports builded need to use specific options, they need to be declared.

Using the command line “poudriere option -z BSDRP CATEGORY/PORTNAME” is the way of doing this easly.

Or you can do this:

rm -rf /var/db/ports/*
cd /usr/ports/net/bird
make config
cd ../frr
make config
cd ../../security/ipsec-tools/
make config
cd ../strongswan
make config
cd ../../sysutils/flashrom/
make config
cp -r /var/db/ports/* poudriere.d/BSDRP-options/

Now the list of package to be builded and added to the final image:

security/ca_root_nss                                                                                                               [15/9237]
net-mgmt/bsnmp-regex
net-mgmt/bsnmp-ucd
lang/python36
lang/python3
lang/python
devel/py-setuptools
net-mgmt/rtrlib
net/frr6
net/bird2
net/freevrrpd
net/openldap24-client
security/ipsec-tools
security/strongswan
net-mgmt/pmacct
net/ucarp
net/arping
net/smcroute
net/pim6-tools
net/pim6dd
net/pim6sd
net/mrouted
net/pimdd
net/pimd
net/tayga
net/isc-dhcp44-server
net/dhcprelya
net/dhcp6
sysutils/fswatch-mon
sysutils/monit
sysutils/tmux
sysutils/ipmitool
security/sudo
net/mpd5
net/mlvpn
security/easy-rsa
security/openvpn
benchmarks/iperf3
net/exabgp
sysutils/flashrom
sysutils/x86info
sysutils/devcpu-data
sysutils/intel-pcm
net/ntraceroute
net-mgmt/bgpq3
net/ixl_unlock
net/graphpath
net/quagga-bgp-netgen

The simplest solution is to re-use already existing BSDRP patched source tree: specific kernel configuration files can be installed into these source tree and use after.

Start by only patching BSDRP sources (sources and ports) using the BSDRP make.sh script:

./make.sh -U
Update ONLY done

There is now 2 sources ready patched:

• /usr/local/BSDRP/BSDRP/FreeBSD/src (including BSDRP specific kernel configuration file)
• /usr/local/BSDRP/BSDRP/FreeBSD/ports

poudriere jail -c -j BSDRP -a amd64 -z BSDRP -b -m src=/usr/local/BSDRP/BSDRP/FreeBSD/src -K amd64

Command line details:

• -b: Build from source
• -c: create a jail
• -j: SHORT name for the jail (I can't use BSDRP-amd64-10.3R here because later it will generate a long directory name and long name aren't well supported)
• -m src=: Path to the patched source branch we want to use
• -z: Configuration set name, used for loading the BSDRP-src.conf
• -K: The kernel configuration file, was copied here during patching BSDRP code trees

Now we need to create a port-tree using the patched existing port tree:

poudriere ports -c -p BSDRP-ports -m null -M /usr/local/BSDRP/BSDRP/FreeBSD/ports

poudriere native role, we just give the jail name, sets name and port-tree name to use then the list of packages.

poudriere bulk -j BSDRP -z BSDRP -p BSDRP-ports -f /usr/local/etc/poudriere.d/BSDRP-pkglist

Here I'm instructing to build a 4GB image using the previous sets, jail, port-tree.

poudriere image -t firmware -s 4g -j BSDRP -p BSDRP-ports -z BSDRP -n BSDRP -h router.bsdrp.net -c /usr/local/BSDRP/BSDRP/Files/ -f /usr/local/etc/poudriere.d/BSDRP-pkglist

Command line explanation:

• -s: Size of full image size (same as the flash media)
• -n: Image name, will be use as the partition name too
• -h: Hostname configured on the image
• -f: List of package to be installed on the image
• -c: Directory tree to be copied on the image

: It needs a hook at the end of image generation for advanced task like generating an mtree (used for host-IDS) or specific user creation.