documentation:technical_docs:poudriere
Differences
This shows you the differences between two versions of the page.
Next revision | Previous revision | ||
documentation:technical_docs:poudriere [2019/03/02 14:33] – external edit 127.0.0.1 | documentation:technical_docs:poudriere [2024/04/04 12:19] (current) – [Features matrix comparison] olivier | ||
---|---|---|---|
Line 21: | Line 21: | ||
But once you get a clean jail in one side, and a list of fresh generated package in other side, why not mix them together in a " | But once you get a clean jail in one side, and a list of fresh generated package in other side, why not mix them together in a " | ||
- | This is the " | + | This is the " |
- | that is presented here. | + | |
===== Images generated by Poudriere ===== | ===== Images generated by Poudriere ===== | ||
Poudriere can generate multiple " | Poudriere can generate multiple " | ||
- | * iso: An ISO 9660 format image | + | * iso: ISO 9660 format image |
- | * iso+mfs: | + | * iso+mfs: ISO 9660 variant |
- | * iso+zmfs (default): | + | * iso+zmfs (default): |
- | * usb: A GPT-layout prepared UFS2 image containing a UEFI boot loader. | + | * usb: GPT-layout prepared UFS2 image containing a UEFI boot loader. |
- | * usb+mfs : A GPT-layout prepared UFS2 image containing a UEFI boot loader | + | * usb+mfs : variant |
- | * usb+zmfs: | + | * usb+zmfs: |
- | * firmware: | + | * rawdisk: raw UFS2, softupdates-enabled, |
- | * rawfirmware: | + | * zrawdisk: raw ZFS disk image |
- | * rawdisk: A raw UFS2, softupdates-enabled, disk image | + | * tar: XZ-compressed tarball |
- | * zrawdisk: A raw ZFS disk image | + | * firmware: NanoBSD style image with a GPT partitions and a UEFI boot loader |
- | * tar: An XZ-compressed tarball | + | * rawfirmware: |
- | * embedded: Create | + | * embedded: u-boot ready embedded |
+ | * zsnapshot: zfs snapshot full and incremental to be used in a jail | ||
===== The 6 minimum steps to build a poudriere firmware image ===== | ===== The 6 minimum steps to build a poudriere firmware image ===== | ||
Line 45: | Line 45: | ||
- Install poudriere and configure it: < | - Install poudriere and configure it: < | ||
echo " | echo " | ||
- | - Create a poudriere jail WITH a GENERIC kernel (by default kernel is not build & installed): | + | - Create a poudriere jail WITH a GENERIC kernel (by default kernel is not build & installed), here named " |
- Create a port-tree using " | - Create a port-tree using " | ||
- Generate list of ports to be build & added into the firmware image:< | - Generate list of ports to be build & added into the firmware image:< | ||
cat > ~/ | cat > ~/ | ||
sysutils/ | sysutils/ | ||
- | net/frr6 | + | net/frr9 |
- | net/bird | + | net/bird2 |
net/mpd5 | net/mpd5 | ||
EOF</ | EOF</ | ||
- | - Build them using " | + | - Build the ports (ie: generate binary packages) from the corresponding jail using " |
- Generate your disk image (4Gb total, because 2 systems partitions of 2Gb) using " | - Generate your disk image (4Gb total, because 2 systems partitions of 2Gb) using " | ||
poudriere image -t firmware -j router -s 4g -p router-ports -h router -n router -f ~/ | poudriere image -t firmware -j router -s 4g -p router-ports -h router -n router -f ~/ | ||
(...) | (...) | ||
- | [00:00:40] Image available at: / | + | [00:00:15] Creating ESP image |
+ | [00:00:15] ESP Image created | ||
+ | [00:00:21] Image available at: / | ||
===== Comparing poudriere firmware image and nanobsd image ===== | ===== Comparing poudriere firmware image and nanobsd image ===== | ||
Line 66: | Line 68: | ||
Start by checking poudriere firmware file size: | Start by checking poudriere firmware file size: | ||
< | < | ||
- | root@lame4: | + | # ls -alh / |
- | -rw-r--r-- | + | -rw-r--r-- |
</ | </ | ||
- | We've obtained a 3.6GiB disk image file... which fit into a [[https:// | + | We've obtained a 3.8GiB disk image file... which fit into a [[https:// |
Line 77: | Line 78: | ||
The resulting images partition layout will have the same behavior than a nanobsd: | The resulting images partition layout will have the same behavior than a nanobsd: | ||
- | * GPT partition with EFI bootloader (nanobsd uses a MBR scheme with BIOS bootloader) | + | * 10M GPT partition with EFI bootloader (nanobsd uses a MBR scheme with BIOS bootloader) |
* first 991M (calculated from user input) system partition called gpt/ | * first 991M (calculated from user input) system partition called gpt/ | ||
* second system partition called gpt/ | * second system partition called gpt/ | ||
Line 84: | Line 85: | ||
< | < | ||
- | root@router:~ # gpart show -l | + | # mdconfig -a -t vnode -f / |
- | => 4 | + | md0 |
- | 4 1600 | + | # gpart show -l md0 |
- | 1604 118 | + | => 4 |
- | 1722 2029568 | + | 4 |
- | | + | |
- | | + | |
- | | + | |
- | + | | |
- | => 4 4191926 | + | |
- | 4 | + | |
- | | + | # mount /dev/gpt/ |
- | | + | # df -h /mnt |
- | | + | |
- | 4060858 | + | |
- | 4126394 | + | |
- | + | ||
- | root@router: | + | |
Filesystem | Filesystem | ||
- | / | + | / |
- | devfs | + | |
- | tmpfs 32M 3.2M | + | |
- | tmpfs 32M 2.7M | + | |
</ | </ | ||
==== /etc ram disk ==== | ==== /etc ram disk ==== | ||
Line 129: | Line 122: | ||
==== fstab ==== | ==== fstab ==== | ||
- | fstab is compliant to nanobsd | + | poudriere' |
< | < | ||
Line 155: | Line 148: | ||
^ BSDRP customized nanobsd usage ^ poudriere image ^ Purpose ^ | ^ BSDRP customized nanobsd usage ^ poudriere image ^ Purpose ^ | ||
| Support differents src.conf: One for buildworld and another for installword | src.conf used for buildworld and installworld, | | Support differents src.conf: One for buildworld and another for installword | src.conf used for buildworld and installworld, | ||
- | | Downloading a specific source revision of a stable/ | + | | Downloading a specific source revision of a stable/ |
| Build a specific kernel file | Just had to install custom kernel int the custom source tree | Customized kernel | | | Build a specific kernel file | Just had to install custom kernel int the custom source tree | Customized kernel | | ||
- | | Build only list of kernel modules | FIXME How to do that? | Customized module list | | + | | Build only list of kernel modules | Declaring MODULES_OVERRIDE in -src.conf |
- | | Advanced scripting for building non-ports softwares | FIXME Not available, no idea of how to cleanly add this feature | + | | Advanced scripting for building non-ports softwares | Solution should be to create a port | Need to build some of / |
- | | Advanced | + | | Advanced |
| Building ports using own (outdated) embedded script | NATIVE role of poudriere | Adding ports to image | | | Building ports using own (outdated) embedded script | NATIVE role of poudriere | Adding ports to image | | ||
| System upgrade by changing MBR active mode on system partition | System upgrade by setting " | | System upgrade by changing MBR active mode on system partition | System upgrade by setting " | ||
+ | |||
+ | ===== poudriere-image patches ===== | ||
+ | |||
+ | List of mandatory patches for poudriere in Pull-request review: | ||
+ | * [[https:// | ||
+ | * [[https:// | ||
+ | |||
+ | List of tuning patches in Pull-request review: | ||
+ | * [[https:// | ||
+ | |||
+ | Merged patches: | ||
+ | * [[https:// | ||
+ | * [[https:// | ||
+ | * [[https:// | ||
+ | * [[https:// | ||
====== Customized poudriere image for BSDRP ====== | ====== Customized poudriere image for BSDRP ====== | ||
- | How to manually (without a " | + | How to configure poudriere |
===== Set of configuration files ===== | ===== Set of configuration files ===== | ||
- | We need to start creating a set of configuration files, named BSDRP: | + | To be able to reproduce the highly customized BSDRP firmware image, we need multiples configuration files. |
- | * poudriere.d/ | + | |
- | * poudriere.d/ | + | Poudriere will requiere jail and port so prepending j and p to avoid confusion: |
- | * poudriere.d/ | + | * BSDRPj for the jail name |
- | * poudriere.d/ | + | * BSDRPp for the port tree name |
+ | |||
+ | We need to start creating a set of configuration files, named prefixed with the name BSDRP-: | ||
+ | * poudriere.d/ | ||
+ | * poudriere.d/ | ||
+ | * poudriere.d/ | ||
Then need other configuration files: | Then need other configuration files: | ||
* BSDRP-pkglist : List of packages to be build and included in the final image | * BSDRP-pkglist : List of packages to be build and included in the final image | ||
- | + | * A kernel configuration file: I'm using the [[https:// | |
- | ==== poudriere.d/ | + | * excluded.files: |
+ | * [[https:// | ||
+ | ==== poudriere.d/ | ||
The previous section of the NanoBSD configuration files found in variables CONF_BUILD and CONF_WORLD in file [[https:// | The previous section of the NanoBSD configuration files found in variables CONF_BUILD and CONF_WORLD in file [[https:// | ||
- | As for the BSDRP example, this give this file: | + | Notice this jail will be used to build the port, so compiler should be kept here. |
- | < | + | |
- | MALLOC_PRODUCTION= | + | |
- | BOOT_BOOT0_COMCONSOLE_SPEED=0 | + | |
- | WITHOUT_ACCT= | + | |
- | WITHOUT_AMD= | + | |
- | WITHOUT_APM= | + | |
- | WITHOUT_ASSERT_DEBUG= | + | |
- | WITHOUT_ATF= | + | |
- | WITHOUT_ATM= | + | |
- | WITHOUT_AUDIT= | + | |
- | WITHOUT_AUTHPF= | + | |
- | WITHOUT_AUTOFS= | + | |
- | WITHOUT_BHYVE= | + | |
- | WITHOUT_BLUETOOTH= | + | |
- | WITHOUT_BOOTPARAMD= | + | |
- | WITHOUT_BOOTPD= | + | |
- | WITHOUT_BSDINSTALL= | + | |
- | WITHOUT_CALENDAR= | + | |
- | WITHOUT_CCD= | + | |
- | WITHOUT_CTM= | + | |
- | WITHOUT_CVS= | + | |
- | WITHOUT_DICT= | + | |
- | WITHOUT_EE= | + | |
- | WITHOUT_EXAMPLES= | + | |
- | WITHOUT_FINGER= | + | |
- | WITHOUT_FLOPPY= | + | |
- | WITHOUT_FREEBSD_UPDATE= | + | |
- | WITHOUT_GAMES= | + | |
- | WITHOUT_GCOV= | + | |
- | WITHOUT_GNUCXX= | + | |
- | WITHOUT_GPIB= | + | |
- | WITHOUT_GPIO= | + | |
- | WITHOUT_GROFF= | + | |
- | WITHOUT_HAST= | + | |
- | WITHOUT_HTML= | + | |
- | WITHOUT_ICONV= | + | |
- | WITHOUT_INFO= | + | |
- | WITHOUT_IPX= | + | |
- | WITHOUT_IPX_SUPPORT= | + | |
- | WITHOUT_ISCSI= | + | |
- | WITHOUT_LIB32= | + | |
- | WITHOUT_LINT= | + | |
- | WITHOUT_LOADER_FIREWIRE= | + | |
- | WITHOUT_LOADER_GELI= | + | |
- | WITHOUT_LOCATE= | + | |
- | WITHOUT_LPR= | + | |
- | WITHOUT_NCP= | + | |
- | WITHOUT_NDIS= | + | |
- | WITHOUT_NETCAT= | + | |
- | WITHOUT_NIS= | + | |
- | WITHOUT_NLS= | + | |
- | WITHOUT_NLS_CATALOGS= | + | |
- | WITHOUT_NS_CACHING= | + | |
- | WITHOUT_PC_SYSINSTALL= | + | |
- | WITHOUT_PORTSNAP= | + | |
- | WITHOUT_PROFILE= | + | |
- | WITHOUT_QUOTAS= | + | |
- | WITHOUT_RBOOTD= | + | |
- | WITHOUT_RCMDS= | + | |
- | WITHOUT_RCS= | + | |
- | WITHOUT_RESCUE= | + | |
- | WITHOUT_ROUTED= | + | |
- | WITHOUT_SENDMAIL= | + | |
- | WITHOUT_SERVICESDB= | + | |
- | WITHOUT_SHAREDOCS= | + | |
- | WITHOUT_SVNLITE= | + | |
- | WITHOUT_SYSCONS= | + | |
- | WITHOUT_SYSINSTALL= | + | |
- | WITHOUT_TALK= | + | |
- | WITHOUT_TESTS= | + | |
- | WITHOUT_TESTS_SUPPORT= | + | |
- | WITHOUT_TFTP= | + | |
- | WITHOUT_TIMED= | + | |
- | WITHOUT_UNBOUND= | + | |
- | WITHOUT_USB_GADGET_EXAMPLES= | + | |
- | WITHOUT_WIRELESS= | + | |
- | WITHOUT_WPA_SUPPLICANT_EAPOL= | + | |
- | WITHOUT_ZFS= | + | |
- | WITH_IDEA= | + | |
- | WITH_OFED= | + | |
- | WITH_DEBUG_FILES= | + | |
- | WITH_REPRODUCIBLE_BUILD= | + | |
- | WITH_DIRDEPS_BUILD= | + | |
- | WITH_RETPOLINE= | + | |
- | </ | + | |
- | ==== poudriere.d/ | + | The [[https:// |
+ | ==== poudriere.d/ | ||
- | Allow to ADD WITHOUT | + | Allow to ADD WITHOUT_ knob that will be removed during installworld |
- | < | + | This is where we remove compiler and other no-more used part. |
- | cat <<EOF > / | + | |
- | WITHOUT_DEBUG_FILES= | + | |
- | WITHOUT_TOOLCHAIN= | + | |
- | WITHOUT_INCLUDES= | + | |
- | EOF | + | |
- | </ | + | |
- | ==== poudriere.d/ | + | |
- | This file contains configuration parameters common for all ports. | + | The [[https:// |
- | + | ==== poudriere.d/BSDRPj-make.conf ==== | |
- | For BSDRP, it is simple: | + | |
- | < | + | |
- | cat <<EOF > /usr/local/ | + | |
- | OPTIONS_UNSET+= DOCS NLS X11 EXAMPLES | + | |
- | EOF | + | |
- | </code> | + | |
- | ==== poudriere.d/ | + | This file contains build parameters for the ports. |
- | If the ports builded need to use specific options, they need to be declared. | + | The [[https:// |
- | Using the command line " | ||
- | |||
- | Or you can do this: | ||
- | < | ||
- | rm -rf / | ||
- | cd / | ||
- | make config | ||
- | cd ../frr | ||
- | make config | ||
- | cd ../ | ||
- | make config | ||
- | cd ../ | ||
- | make config | ||
- | cd ../ | ||
- | make config | ||
- | cp -r / | ||
- | </ | ||
==== BSDRP-pkglist ==== | ==== BSDRP-pkglist ==== | ||
- | Now the list of package to be builded and added to the final image: | + | This file includes |
- | < | + | The [[https://github.com/ocochard/BSDRP/blob/master/poudriere.etc/poudriere.d/BSDRP-pkglist|BSDRP-pkglist]] is on github. |
- | security/ | + | |
- | net-mgmt/bsnmp-regex | + | |
- | net-mgmt/bsnmp-ucd | + | |
- | lang/python36 | + | |
- | lang/python3 | + | ==== excluded.files ==== |
- | lang/python | + | |
- | devel/py-setuptools | + | List of files/directory that WITHOUT_ wasn't able to prevent to be on the final image. |
- | net-mgmt/rtrlib | + | |
- | net/frr6 | + | The [[https://github.com/ocochard/BSDRP/blob/master/poudriere.etc/poudriere.d/excluded.files|excluded.files ]] is on github. |
- | net/bird2 | + | |
- | net/ | + | |
- | net/ | + | |
- | security/ | + | |
- | security/ | + | |
- | net-mgmt/ | + | |
- | net/ucarp | + | |
- | net/arping | + | |
- | net/ | + | |
- | net/ | + | |
- | net/pim6dd | + | |
- | net/pim6sd | + | |
- | net/mrouted | + | |
- | net/pimdd | + | |
- | net/pimd | + | |
- | net/tayga | + | |
- | net/isc-dhcp44-server | + | |
- | net/dhcprelya | + | |
- | net/dhcp6 | + | |
- | sysutils/fswatch-mon | + | |
- | sysutils/monit | + | |
- | sysutils/ | + | |
- | sysutils/ | + | |
- | security/ | + | |
- | net/mpd5 | + | |
- | net/mlvpn | + | |
- | security/ | + | |
- | security/ | + | |
- | benchmarks/ | + | |
- | net/ | + | |
- | sysutils/ | + | |
- | sysutils/ | + | |
- | sysutils/ | + | |
- | sysutils/ | + | |
- | net/ | + | |
- | net-mgmt/ | + | |
- | net/ | + | |
- | net/ | + | |
- | net/ | + | |
- | </ | + | |
Line 373: | Line 234: | ||
The simplest solution is to re-use already existing BSDRP patched source tree: specific kernel configuration files can be installed into these source tree and use after. | The simplest solution is to re-use already existing BSDRP patched source tree: specific kernel configuration files can be installed into these source tree and use after. | ||
- | Start by only patching BSDRP sources (sources and ports) using the BSDRP make.sh script: | + | Start by only patching BSDRP sources (sources and ports) using the [[https:// |
< | < | ||
./make.sh -U | ./make.sh -U | ||
Line 385: | Line 246: | ||
< | < | ||
- | poudriere jail -c -j BSDRP -a amd64 -z BSDRP -b -m src=/ | + | poudriere jail -c -j BSDRPj |
</ | </ | ||
Command line details: | Command line details: | ||
Line 392: | Line 253: | ||
* -j: SHORT name for the jail (I can't use BSDRP-amd64-10.3R here because later it will generate a long directory name and long name aren't well supported) | * -j: SHORT name for the jail (I can't use BSDRP-amd64-10.3R here because later it will generate a long directory name and long name aren't well supported) | ||
* -m src=: Path to the patched source branch we want to use | * -m src=: Path to the patched source branch we want to use | ||
- | | + | * -K: The kernel configuration file (was copied here during patching BSDRP code tree) |
- | | + | |
===== Creating port tree ===== | ===== Creating port tree ===== | ||
Line 400: | Line 260: | ||
< | < | ||
- | poudriere ports -c -p BSDRP-ports | + | poudriere ports -c -p BSDRPp |
</ | </ | ||
===== Build packages ===== | ===== Build packages ===== | ||
- | poudriere native role, we just give the jail name, sets name and port-tree name to use then the list of packages. | + | poudriere native role, we just give the jail name and port-tree name to use then the list of packages. |
< | < | ||
- | poudriere bulk -j BSDRP -z BSDRP -p BSDRP-ports | + | poudriere bulk -j BSDRPj |
</ | </ | ||
===== Generating firmware image ===== | ===== Generating firmware image ===== | ||
- | Here I'm instructing to build a 4GB image using the previous sets, jail, port-tree. | + | Here I'm instructing to build a 2GB image using the previous sets, jail, port-tree. |
< | < | ||
- | poudriere image -t firmware -s 4g -j BSDRP -p BSDRP-ports -z BSDRP -n BSDRP -h router.bsdrp.net -c / | + | poudriere image -t firmware -s 2g \ |
+ | | ||
+ | | ||
+ | | ||
+ | -X poudriere.etc/poudriere.d/ | ||
+ | -A poudriere.etc/ | ||
</ | </ | ||
Command line explanation: | Command line explanation: | ||
* -s: Size of full image size (same as the flash media) | * -s: Size of full image size (same as the flash media) | ||
+ | * -j: The jail we just generated | ||
+ | * -p: The poudriere port tree, we just generated its packages | ||
* -n: Image name, will be use as the partition name too | * -n: Image name, will be use as the partition name too | ||
* -h: Hostname configured on the image | * -h: Hostname configured on the image | ||
+ | * -c: Directory tree to be copied on the image (that should include a pkg.conf with the FILES_IGNORE_GLOB) | ||
* -f: List of package to be installed on the image | * -f: List of package to be installed on the image | ||
- | * -c: Directory tree to be copied on the image | + | * -X: List of file to be excluded from the installworld |
- | + | * -A: The post-script, | |
- | FIXME: It needs a hook at the end of image generation for advanced task like generating | + |
documentation/technical_docs/poudriere.1551533603.txt.gz · Last modified: 2019/03/02 14:33 by 127.0.0.1