documentation:technical_docs:poudriere
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
| documentation:technical_docs:poudriere [2019/03/02 14:33] – external edit 127.0.0.1 | documentation:technical_docs:poudriere [2024/11/18 19:11] (current) – [Build packages] olivier | ||
|---|---|---|---|
| Line 21: | Line 21: | ||
| But once you get a clean jail in one side, and a list of fresh generated package in other side, why not mix them together in a " | But once you get a clean jail in one side, and a list of fresh generated package in other side, why not mix them together in a " | ||
| - | This is the " | + | This is the " |
| - | that is presented here. | + | |
| ===== Images generated by Poudriere ===== | ===== Images generated by Poudriere ===== | ||
| Poudriere can generate multiple " | Poudriere can generate multiple " | ||
| - | * iso: An ISO 9660 format image | + | * iso: ISO 9660 format image |
| - | * iso+mfs: | + | * iso+mfs: ISO 9660 variant |
| - | * iso+zmfs (default): | + | * iso+zmfs (default): |
| - | * usb: A GPT-layout prepared UFS2 image containing a UEFI boot loader. | + | * usb: GPT-layout prepared UFS2 image containing a UEFI boot loader. |
| - | * usb+mfs : A GPT-layout prepared UFS2 image containing a UEFI boot loader | + | * usb+mfs : variant |
| - | * usb+zmfs: | + | * usb+zmfs: |
| - | * firmware: | + | * rawdisk: raw UFS2, softupdates-enabled, |
| - | * rawfirmware: | + | * zrawdisk: raw ZFS disk image |
| - | * rawdisk: A raw UFS2, softupdates-enabled, disk image | + | * tar: XZ-compressed tarball |
| - | * zrawdisk: A raw ZFS disk image | + | * firmware: NanoBSD style image with a GPT partitions and a UEFI boot loader |
| - | * tar: An XZ-compressed tarball | + | * rawfirmware: |
| - | * embedded: Create | + | * embedded: u-boot ready embedded |
| + | * zsnapshot: zfs snapshot full and incremental to be used in a jail | ||
| ===== The 6 minimum steps to build a poudriere firmware image ===== | ===== The 6 minimum steps to build a poudriere firmware image ===== | ||
| Using poudriere from a ZFS is not mandatory but strongly advised. | Using poudriere from a ZFS is not mandatory but strongly advised. | ||
| - | - Install poudriere and configure it: < | + | - Install poudriere and configure it: < |
| - | echo " | + | echo " |
| - | - Create a poudriere jail WITH a GENERIC kernel (by default kernel is not build & installed): | + | - Create a poudriere jail WITH a GENERIC kernel (by default kernel is not build & installed), here named " |
| - | - Create a port-tree using " | + | - Create a port-tree using " |
| - Generate list of ports to be build & added into the firmware image:< | - Generate list of ports to be build & added into the firmware image:< | ||
| cat > ~/ | cat > ~/ | ||
| sysutils/ | sysutils/ | ||
| - | net/frr6 | + | net/frr10 |
| - | net/bird | + | net/bird2 |
| net/mpd5 | net/mpd5 | ||
| EOF</ | EOF</ | ||
| - | - Build them using " | + | - Build the ports (ie: generate binary packages) from the corresponding jail using " |
| - Generate your disk image (4Gb total, because 2 systems partitions of 2Gb) using " | - Generate your disk image (4Gb total, because 2 systems partitions of 2Gb) using " | ||
| - | poudriere image -t firmware -j router -s 4g -p router-ports | + | sudo poudriere image -t firmware -j router -s 4g -p router_ports |
| (...) | (...) | ||
| - | [00:00:40] Image available at: / | + | [00:00:15] Creating ESP image |
| + | [00:00:15] ESP Image created | ||
| + | [00:00:21] Image available at: / | ||
| ===== Comparing poudriere firmware image and nanobsd image ===== | ===== Comparing poudriere firmware image and nanobsd image ===== | ||
| Line 66: | Line 68: | ||
| Start by checking poudriere firmware file size: | Start by checking poudriere firmware file size: | ||
| < | < | ||
| - | root@lame4: | + | # ls -alh / |
| - | -rw-r--r-- | + | -rw-r--r-- |
| </ | </ | ||
| - | We've obtained a 3.6GiB disk image file... which fit into a [[https:// | + | We've obtained a 3.8GiB disk image file... which fit into a [[https:// |
| Line 77: | Line 78: | ||
| The resulting images partition layout will have the same behavior than a nanobsd: | The resulting images partition layout will have the same behavior than a nanobsd: | ||
| - | * GPT partition with EFI bootloader (nanobsd uses a MBR scheme with BIOS bootloader) | + | * 10M GPT partition with EFI bootloader (nanobsd uses a MBR scheme with BIOS bootloader) |
| * first 991M (calculated from user input) system partition called gpt/ | * first 991M (calculated from user input) system partition called gpt/ | ||
| * second system partition called gpt/ | * second system partition called gpt/ | ||
| Line 84: | Line 85: | ||
| < | < | ||
| - | root@router:~ # gpart show -l | + | # mdconfig -a -t vnode -f / |
| - | => 4 | + | md0 |
| - | 4 1600 | + | # gpart show -l md0 |
| - | 1604 118 | + | => 4 |
| - | 1722 2029568 | + | 4 |
| - | | + | |
| - | | + | |
| - | | + | |
| - | + | | |
| - | => 4 4191926 | + | |
| - | 4 | + | |
| - | | + | # mount /dev/gpt/ |
| - | | + | # df -h /mnt |
| - | | + | |
| - | 4060858 | + | |
| - | 4126394 | + | |
| - | + | ||
| - | root@router: | + | |
| Filesystem | Filesystem | ||
| - | / | + | / |
| - | devfs | + | |
| - | tmpfs 32M 3.2M | + | |
| - | tmpfs 32M 2.7M | + | |
| </ | </ | ||
| ==== /etc ram disk ==== | ==== /etc ram disk ==== | ||
| Line 129: | Line 122: | ||
| ==== fstab ==== | ==== fstab ==== | ||
| - | fstab is compliant to nanobsd | + | poudriere' |
| < | < | ||
| Line 155: | Line 148: | ||
| ^ BSDRP customized nanobsd usage ^ poudriere image ^ Purpose ^ | ^ BSDRP customized nanobsd usage ^ poudriere image ^ Purpose ^ | ||
| | Support differents src.conf: One for buildworld and another for installword | src.conf used for buildworld and installworld, | | Support differents src.conf: One for buildworld and another for installword | src.conf used for buildworld and installworld, | ||
| - | | Downloading a specific source revision of a stable/ | + | | Downloading a specific source revision of a stable/ |
| | Build a specific kernel file | Just had to install custom kernel int the custom source tree | Customized kernel | | | Build a specific kernel file | Just had to install custom kernel int the custom source tree | Customized kernel | | ||
| - | | Build only list of kernel modules | FIXME How to do that? | Customized module list | | + | | Build only list of kernel modules | Declaring MODULES_OVERRIDE in -src.conf |
| - | | Advanced scripting for building non-ports softwares | FIXME Not available, no idea of how to cleanly add this feature | + | | Advanced scripting for building non-ports softwares | Solution should be to create a port | Need to build some of / |
| - | | Advanced | + | | Advanced |
| | Building ports using own (outdated) embedded script | NATIVE role of poudriere | Adding ports to image | | | Building ports using own (outdated) embedded script | NATIVE role of poudriere | Adding ports to image | | ||
| | System upgrade by changing MBR active mode on system partition | System upgrade by setting " | | System upgrade by changing MBR active mode on system partition | System upgrade by setting " | ||
| + | |||
| + | ===== poudriere-image patches ===== | ||
| + | |||
| + | List of mandatory patches for poudriere in Pull-request review: | ||
| + | * [[https:// | ||
| + | |||
| + | Merged patches: | ||
| + | * [[https:// | ||
| + | * [[https:// | ||
| + | * [[https:// | ||
| + | * [[https:// | ||
| + | * [[https:// | ||
| + | * [[https:// | ||
| + | * [[https:// | ||
| ====== Customized poudriere image for BSDRP ====== | ====== Customized poudriere image for BSDRP ====== | ||
| - | How to manually (without a " | + | How to configure poudriere |
| ===== Set of configuration files ===== | ===== Set of configuration files ===== | ||
| - | We need to start creating a set of configuration files, named BSDRP: | + | To be able to reproduce the highly customized BSDRP firmware image, we need multiples configuration files. |
| - | * poudriere.d/ | + | |
| - | * poudriere.d/ | + | Poudriere will requiere jail and port so prepending j and p to avoid confusion: |
| - | * poudriere.d/ | + | * BSDRPj for the jail name |
| - | * poudriere.d/ | + | * BSDRPp for the port tree name |
| + | |||
| + | We need to start creating a set of configuration files, named prefixed with the name BSDRP-: | ||
| + | * poudriere.d/ | ||
| + | * poudriere.d/ | ||
| + | * poudriere.d/ | ||
| Then need other configuration files: | Then need other configuration files: | ||
| * BSDRP-pkglist : List of packages to be build and included in the final image | * BSDRP-pkglist : List of packages to be build and included in the final image | ||
| - | + | * A kernel configuration file: I'm using the [[https:// | |
| - | ==== poudriere.d/ | + | * excluded.files: |
| + | * [[https:// | ||
| + | ==== poudriere.d/ | ||
| The previous section of the NanoBSD configuration files found in variables CONF_BUILD and CONF_WORLD in file [[https:// | The previous section of the NanoBSD configuration files found in variables CONF_BUILD and CONF_WORLD in file [[https:// | ||
| - | As for the BSDRP example, this give this file: | + | Notice this jail will be used to build the port, so compiler should be kept here. |
| - | < | + | |
| - | MALLOC_PRODUCTION= | + | |
| - | BOOT_BOOT0_COMCONSOLE_SPEED=0 | + | |
| - | WITHOUT_ACCT= | + | |
| - | WITHOUT_AMD= | + | |
| - | WITHOUT_APM= | + | |
| - | WITHOUT_ASSERT_DEBUG= | + | |
| - | WITHOUT_ATF= | + | |
| - | WITHOUT_ATM= | + | |
| - | WITHOUT_AUDIT= | + | |
| - | WITHOUT_AUTHPF= | + | |
| - | WITHOUT_AUTOFS= | + | |
| - | WITHOUT_BHYVE= | + | |
| - | WITHOUT_BLUETOOTH= | + | |
| - | WITHOUT_BOOTPARAMD= | + | |
| - | WITHOUT_BOOTPD= | + | |
| - | WITHOUT_BSDINSTALL= | + | |
| - | WITHOUT_CALENDAR= | + | |
| - | WITHOUT_CCD= | + | |
| - | WITHOUT_CTM= | + | |
| - | WITHOUT_CVS= | + | |
| - | WITHOUT_DICT= | + | |
| - | WITHOUT_EE= | + | |
| - | WITHOUT_EXAMPLES= | + | |
| - | WITHOUT_FINGER= | + | |
| - | WITHOUT_FLOPPY= | + | |
| - | WITHOUT_FREEBSD_UPDATE= | + | |
| - | WITHOUT_GAMES= | + | |
| - | WITHOUT_GCOV= | + | |
| - | WITHOUT_GNUCXX= | + | |
| - | WITHOUT_GPIB= | + | |
| - | WITHOUT_GPIO= | + | |
| - | WITHOUT_GROFF= | + | |
| - | WITHOUT_HAST= | + | |
| - | WITHOUT_HTML= | + | |
| - | WITHOUT_ICONV= | + | |
| - | WITHOUT_INFO= | + | |
| - | WITHOUT_IPX= | + | |
| - | WITHOUT_IPX_SUPPORT= | + | |
| - | WITHOUT_ISCSI= | + | |
| - | WITHOUT_LIB32= | + | |
| - | WITHOUT_LINT= | + | |
| - | WITHOUT_LOADER_FIREWIRE= | + | |
| - | WITHOUT_LOADER_GELI= | + | |
| - | WITHOUT_LOCATE= | + | |
| - | WITHOUT_LPR= | + | |
| - | WITHOUT_NCP= | + | |
| - | WITHOUT_NDIS= | + | |
| - | WITHOUT_NETCAT= | + | |
| - | WITHOUT_NIS= | + | |
| - | WITHOUT_NLS= | + | |
| - | WITHOUT_NLS_CATALOGS= | + | |
| - | WITHOUT_NS_CACHING= | + | |
| - | WITHOUT_PC_SYSINSTALL= | + | |
| - | WITHOUT_PORTSNAP= | + | |
| - | WITHOUT_PROFILE= | + | |
| - | WITHOUT_QUOTAS= | + | |
| - | WITHOUT_RBOOTD= | + | |
| - | WITHOUT_RCMDS= | + | |
| - | WITHOUT_RCS= | + | |
| - | WITHOUT_RESCUE= | + | |
| - | WITHOUT_ROUTED= | + | |
| - | WITHOUT_SENDMAIL= | + | |
| - | WITHOUT_SERVICESDB= | + | |
| - | WITHOUT_SHAREDOCS= | + | |
| - | WITHOUT_SVNLITE= | + | |
| - | WITHOUT_SYSCONS= | + | |
| - | WITHOUT_SYSINSTALL= | + | |
| - | WITHOUT_TALK= | + | |
| - | WITHOUT_TESTS= | + | |
| - | WITHOUT_TESTS_SUPPORT= | + | |
| - | WITHOUT_TFTP= | + | |
| - | WITHOUT_TIMED= | + | |
| - | WITHOUT_UNBOUND= | + | |
| - | WITHOUT_USB_GADGET_EXAMPLES= | + | |
| - | WITHOUT_WIRELESS= | + | |
| - | WITHOUT_WPA_SUPPLICANT_EAPOL= | + | |
| - | WITHOUT_ZFS= | + | |
| - | WITH_IDEA= | + | |
| - | WITH_OFED= | + | |
| - | WITH_DEBUG_FILES= | + | |
| - | WITH_REPRODUCIBLE_BUILD= | + | |
| - | WITH_DIRDEPS_BUILD= | + | |
| - | WITH_RETPOLINE= | + | |
| - | </ | + | |
| - | ==== poudriere.d/ | + | The [[https:// |
| + | ==== poudriere.d/ | ||
| - | Allow to ADD WITHOUT | + | Allow to ADD WITHOUT_ knob that will be removed during installworld |
| - | < | + | This is where we remove compiler and other no-more used part. |
| - | cat <<EOF > / | + | |
| - | WITHOUT_DEBUG_FILES= | + | |
| - | WITHOUT_TOOLCHAIN= | + | |
| - | WITHOUT_INCLUDES= | + | |
| - | EOF | + | |
| - | </ | + | |
| - | ==== poudriere.d/ | + | |
| - | This file contains configuration parameters common for all ports. | + | The [[https:// |
| - | + | ==== poudriere.d/BSDRPj-make.conf ==== | |
| - | For BSDRP, it is simple: | + | |
| - | < | + | |
| - | cat <<EOF > /usr/local/ | + | |
| - | OPTIONS_UNSET+= DOCS NLS X11 EXAMPLES | + | |
| - | EOF | + | |
| - | </code> | + | |
| - | ==== poudriere.d/ | + | This file contains build parameters for the ports. |
| - | If the ports builded need to use specific options, they need to be declared. | + | The [[https:// |
| - | Using the command line " | ||
| - | |||
| - | Or you can do this: | ||
| - | < | ||
| - | rm -rf / | ||
| - | cd / | ||
| - | make config | ||
| - | cd ../frr | ||
| - | make config | ||
| - | cd ../ | ||
| - | make config | ||
| - | cd ../ | ||
| - | make config | ||
| - | cd ../ | ||
| - | make config | ||
| - | cp -r / | ||
| - | </ | ||
| ==== BSDRP-pkglist ==== | ==== BSDRP-pkglist ==== | ||
| - | Now the list of package to be builded and added to the final image: | + | This file includes |
| - | < | + | The [[https://github.com/ocochard/BSDRP/blob/master/poudriere.etc/poudriere.d/BSDRP-pkglist|BSDRP-pkglist]] is on github. |
| - | security/ | + | |
| - | net-mgmt/bsnmp-regex | + | |
| - | net-mgmt/bsnmp-ucd | + | |
| - | lang/python36 | + | |
| - | lang/python3 | + | ==== excluded.files ==== |
| - | lang/python | + | |
| - | devel/py-setuptools | + | List of files/directory that WITHOUT_ wasn't able to prevent to be on the final image. |
| - | net-mgmt/rtrlib | + | |
| - | net/frr6 | + | The [[https://github.com/ocochard/BSDRP/blob/master/poudriere.etc/poudriere.d/excluded.files|excluded.files ]] is on github. |
| - | net/bird2 | + | |
| - | net/ | + | ==== Packages excluded files ==== |
| - | net/ | + | |
| - | security/ | + | |
| - | security/ | + | |
| - | net-mgmt/ | + | |
| - | net/ucarp | + | |
| - | net/arping | + | |
| - | net/ | + | |
| - | net/ | + | |
| - | net/pim6dd | + | |
| - | net/pim6sd | + | |
| - | net/mrouted | + | |
| - | net/pimdd | + | |
| - | net/pimd | + | |
| - | net/tayga | + | |
| - | net/isc-dhcp44-server | + | |
| - | net/dhcprelya | + | |
| - | net/dhcp6 | + | |
| - | sysutils/fswatch-mon | + | |
| - | sysutils/monit | + | |
| - | sysutils/ | + | |
| - | sysutils/ | + | |
| - | security/ | + | |
| - | net/mpd5 | + | |
| - | net/mlvpn | + | |
| - | security/ | + | |
| - | security/ | + | |
| - | benchmarks/ | + | |
| - | net/ | + | |
| - | sysutils/ | + | |
| - | sysutils/ | + | |
| - | sysutils/ | + | |
| - | sysutils/ | + | |
| - | net/ | + | |
| - | net-mgmt/ | + | |
| - | net/ | + | |
| - | net/ | + | |
| - | net/ | + | |
| - | </ | + | |
| + | When customizing ports options, some could not be disabled but we could configure pkg to not install some files from packages while extracting them. | ||
| + | The [[https:// | ||
| ===== Building the jail ===== | ===== Building the jail ===== | ||
| Line 373: | Line 237: | ||
| The simplest solution is to re-use already existing BSDRP patched source tree: specific kernel configuration files can be installed into these source tree and use after. | The simplest solution is to re-use already existing BSDRP patched source tree: specific kernel configuration files can be installed into these source tree and use after. | ||
| - | Start by only patching BSDRP sources (sources and ports) using the BSDRP make.sh script: | + | Start by only patching BSDRP sources (sources and ports) using the [[https:// |
| < | < | ||
| - | ./make.sh -U | + | make patch-sources |
| - | Update ONLY done | + | |
| </ | </ | ||
| - | There is now 2 sources ready patched: | + | There are now 2 sources ready patched: |
| - | * / | + | * obj/FreeBSD (including BSDRP specific kernel configuration file) |
| - | * / | + | * obj/ports |
| < | < | ||
| - | poudriere jail -c -j BSDRP -a amd64 -z BSDRP -b -m src=/ | + | poudriere jail -e poudriere.etc |
| </ | </ | ||
| Command line details: | Command line details: | ||
| Line 391: | Line 253: | ||
| * -c: create a jail | * -c: create a jail | ||
| * -j: SHORT name for the jail (I can't use BSDRP-amd64-10.3R here because later it will generate a long directory name and long name aren't well supported) | * -j: SHORT name for the jail (I can't use BSDRP-amd64-10.3R here because later it will generate a long directory name and long name aren't well supported) | ||
| + | * -e: Load all configurations files from ./ | ||
| * -m src=: Path to the patched source branch we want to use | * -m src=: Path to the patched source branch we want to use | ||
| - | | + | * -K: The kernel configuration file (was copied here during patching BSDRP code tree) |
| - | | + | |
| ===== Creating port tree ===== | ===== Creating port tree ===== | ||
| Line 400: | Line 262: | ||
| < | < | ||
| - | poudriere ports -c -p BSDRP-ports | + | poudriere ports -e poudriere.etc |
| </ | </ | ||
| ===== Build packages ===== | ===== Build packages ===== | ||
| - | poudriere native role, we just give the jail name, sets name and port-tree name to use then the list of packages. | + | poudriere native role, we just give the jail name and port-tree name to use then the list of packages. |
| < | < | ||
| - | poudriere bulk -j BSDRP -z BSDRP -p BSDRP-ports | + | poudriere bulk -e poudriere.etc |
| </ | </ | ||
| ===== Generating firmware image ===== | ===== Generating firmware image ===== | ||
| - | Here I'm instructing to build a 4GB image using the previous sets, jail, port-tree. | + | Here I'm instructing to build a 2GB image using the previous sets, jail, port-tree. |
| < | < | ||
| - | poudriere image -t firmware -s 4g -j BSDRP -p BSDRP-ports -z BSDRP -n BSDRP -h router.bsdrp.net -c / | + | poudriere image -t firmware -s 2g \ |
| + | | ||
| + | | ||
| + | | ||
| + | -X poudriere.etc/poudriere.d/ | ||
| + | -A poudriere.etc/ | ||
| </ | </ | ||
| Command line explanation: | Command line explanation: | ||
| * -s: Size of full image size (same as the flash media) | * -s: Size of full image size (same as the flash media) | ||
| + | * -j: The jail we just generated | ||
| + | * -p: The poudriere port tree, we just generated its packages | ||
| * -n: Image name, will be use as the partition name too | * -n: Image name, will be use as the partition name too | ||
| * -h: Hostname configured on the image | * -h: Hostname configured on the image | ||
| + | * -c: Directory tree to be copied on the image (that should include a pkg.conf with the FILES_IGNORE_GLOB) | ||
| * -f: List of package to be installed on the image | * -f: List of package to be installed on the image | ||
| - | * -c: Directory tree to be copied on the image | + | * -X: List of file to be excluded from the installworld |
| - | + | * -A: The post-script, | |
| - | FIXME: It needs a hook at the end of image generation for advanced task like generating | + | |
documentation/technical_docs/poudriere.1551533603.txt.gz · Last modified: 2019/03/02 14:33 by 127.0.0.1