User Tools

Site Tools


documentation:examples:freebsd_performance_regression_lab

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

documentation:examples:freebsd_performance_regression_lab [2017/03/29 09:32]
documentation:examples:freebsd_performance_regression_lab [2017/03/29 09:32] (current)
Line 1: Line 1:
 +====== FreeBSD performance regression lab ======
 +{{description>​FreeBSD performance regression testing lab}}
  
 +===== Concepts =====
 +
 +This article presents the method used for spotting some performance regression problem during the evolution of FreeBSD code (regarding different svn numbers).
 +
 +Here is an example of a final bench graphic measuring the impact of forwarding performance:​
 +{{:​documentation:​examples:​freebsd-performance-regression-lab-example-graph-pps.png|}}
 +
 +This method follow these steps:
 +  - A lab of one or more servers is set-up. [[documentation/​examples/​forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_10-gigabit_intel_82599eb|On this example, 2 servers were used]]: one traffic generator/​receiver,​ one device under test (DUT).
 +  - A [[https://​github.com/​ocochard/​BSDRP/​tree/​master/​TESTING|BSDRP sub-project folder is created]]: This project configure a minimal (for rapid compilation) nanobsd image.
 +  - A first [[https://​github.com/​ocochard/​BSDRP/​blob/​master/​tools/​bisection-gen.sh|script generate lot's of nanobsd image regarding a list of FreeBSD code SVN revision number]]
 +  - [[https://​github.com/​ocochard/​netbenches/​tree/​master/​Xeon_L5630-4Cores-Intel_82599EB/​fastforwarding-pf-ipfw/​configs|Different configuration sets to be tested a created]]. The example given use 3 tests: One with forwarding, a second with a minimal ipfw configuration and a third with a minimal pf configuration
 +  - A second [[https://​github.com/​ocochard/​netbenches/​blob/​master/​scripts/​bench-lab.sh|script will run the test and collect data]] (by sending ssh commands to the lab servers). This order are followed:
 +    - Upgrading the server to the nanobsd release to be tested;
 +    - ..Uploading the configuration set to be tested (in this example: forwarding-only,​ ipfw or pf) & reboot the DUT;
 +    - ....Start the bench test and collecting the result;
 +    - ....Reboot the DUT;
 +    - ....Loop 5 times (for having usable [[http://​www.freebsd.org/​cgi/​man.cgi?​query=ministat|ministat]] datas);
 +    - ..Loop to next configuration set;
 +    - Loop to next nanobsd release
 +  - [[https://​github.com/​ocochard/​netbenches/​blob/​master/​scripts/​bench-lab-ministat.sh|A third script filter the raw output of each tests]] (heavy dependent of the tool used during the bench) and generate synthesis data files for each Revision-number/​configuration-sets pairs.
 +===== Physical lab =====
 +
 +The physical lab used by BSDRP targets forwarding network speed. And for this purpose it's composed of minimum 3 servers and 1 switches, and maximum 4 servers and 2 switchs.
 +Common to all these setups:
 +  * The controller server that runs the bench script;
 +  * The admin switch (send SSH commands)
 +
 +Specific:
 +  * One server with netmap compliant NIC for being used as packet generator/​receiver ​
 +  * One server as DUT
 +  * And for network crypto (IPSec, OpenVPN) bench, a referent server
 +  * A dedicated switch with ([[documentation/​examples/​setting_up_a_forwarding_performance_benchmark_lab#​switch_configuration_cisco_example|with specific configuration for not polluting packet counters]]). Useful for double-checking NIC drivers statistics against the switch statistics
 +
 +==== 2 nodes lab examples ====
 +
 +=== simplest lab ===
 +
 +This simple lab is great for benching 10Gigabit performance without the need of 10Gigabit switch:
 +
 +{{documentation:​examples:​bsdrp_simple_2_node_bench_lab.png|}}
 +
 +=== with switch lab ===
 +
 +This simple lab is usefull for cross-checking NIC statistics counters with switch statistics counters:
 +
 +{{documentation:​examples:​bsdrp_switched_2_node_bench_lab.png|}}
 +
 +==== 3 nodes lab example ====
 +
 +This is the setup used for a VPN benchmark:
 +
 +{{documentation:​examples:​bsdrp_vpn_3_node_bench_lab.png|}}
 +
 +===== Generating FreeBSD nanobsd images =====
 +
 +The objective is to generate BSDRP nanobsd images regarding a list of FreeBSD'​s code SVN revision number.
 +
 +You need to [[documentation:​technical_docs?&#​getting_bsdrp_source_code|download BSDRP source code]] on a FreeBSD machine.
 +
 +Then you have to create or customize a BSDRP project for obtaining a small image: Try to remove maximum [[http://​svnweb.freebsd.org/​base/​head/​tools/​build/​options/​|FreeBSD options]] and no ports for improving the image generation speed. The BSDRP'​s [[https://​github.com/​ocochard/​BSDRP/​tree/​master/​TESTING|TESTING project]] is a good example for this step.
 +
 +Take care of the kernel configuration file: the head branch can impose kernel configuration parameters incompatible between different code revisions. The best method for avoiding this incompatibility is to start the kernel configuration by "​include GENERIC"​ and adding or removing (nooptions XXX, nodevice XXX) unwanted parts. Refer the [[https://​github.com/​ocochard/​BSDRP/​blob/​master/​TESTING/​kernels/​amd64|kernel configuration file used for BSDRP regression bench lab]] as an example.
 +
 +Once your BSDRP bench-project ready, edit the script [[https://​github.com/​ocochard/​BSDRP/​blob/​master/​tools/​bisection-gen.sh|tools/​bisection-gen.sh ]]for adapting the project basic configuration (PROJECT name, CONSOLE type, ARCH) and filling all SVN revision numbers (SVN_REV_LIST),​ then start the image generation script (from the top level directory of BSDRP source code):
 +<​code>​
 +root@dev:/​usr/​local/​BSDRP # mkdir -p /​root/​benchs/​nanobsd
 +root@dev:/​usr/​local/​BSDRP # tools/​bisection-gen.sh /​root/​benchs/​nanobsd
 +Building image matching revision 257506...done
 +Building image matching revision 257657...done
 +Building image matching revision 257715...done
 +All images were put in /​root/​benchs/​nanobsd
 +</​code>​
 +
 +At the end, you will find all the nanobsd images in your selected directory.
 +
 +You can find an example of finals images generated here: [[http://​dev.bsdrp.net/​benchs/​265145-274745/​nanobsd.images/​|]]
 +
 +===== Configuration sets =====
 +
 +The creation of different configuration sets is pretty simple: just create a main folder and inside this folder, creating different folder for each configuration sets.
 +
 +Inside each configuration sets folder will be placed the configuration files that need override default parameters.
 +
 +Here is a very simple example regarding [[documentation/​examples/​forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580#​igb_4_drivers_tunning_with_82546gb|igb NIC drivers tuning benchs]] were differents parameters in /​boot/​loader.conf.local were tested.
 +
 +We want 3 configuration sets:
 +  * First called "​xd1024.proc_lim-1"​
 +  * Second called "​xd2048.proc_lim-1"​
 +  * Third called "​xd4096.proc_lim-1"​
 +
 +<​code>​
 +mkdir -p /​root/​benchs/​configurations/​xd1024.proc_lim-1/​boot
 +cat > /​root/​benchs/​configurations/​xd1024.proc_lim-1/​boot/​loader.conf.local <<EOF
 +hw.igb.rxd="​1024"​
 +hw.igb.txd="​1024"​
 +hw.igb.rx_process_limit="​-1"​
 +EOF
 +mkdir -p /​root/​benchs/​configurations/​xd2048.proc_lim-1/​boot
 +cat > /​root/​benchs/​configurations/​xd2048.proc_lim-1/​boot/​loader.conf.local <<EOF
 +hw.igb.rxd="​2048"​
 +hw.igb.txd="​2048"​
 +hw.igb.rx_process_limit="​-1"​
 +EOF
 +mkdir -p /​root/​benchs/​configurations/​xd4096.proc_lim-1/​boot
 +cat > /​root/​benchs/​configurations/​xd4096.proc_lim-1/​boot/​loader.conf.local <<EOF
 +hw.igb.rxd="​4096"​
 +hw.igb.txd="​4096"​
 +hw.igb.rx_process_limit="​-1"​
 +EOF
 +</​code>​
 +
 +===== Testing Lab =====
 +
 +Start by configuring all servers, and the DUT with the configurations parameters common to all configuration sets (but remember that files in configuration sets will totally overwrite existing file).
 +Install your ssh public-key on each servers for permitting bench script to automatically connect to them.
 +
 +There are 2 examples of configuration files available for a bench lab:
 +  * [[https://​github.com/​ocochard/​netbenches/​blob/​master/​AMD_GX-412TC_4Cores_Intel_i210AT/​bench-lab-2nodes.config|bench-lab-2nodes.config]]
 +  * [[https://​github.com/​ocochard/​netbenches/​blob/​master/​AMD_GX-412TC_4Cores_Intel_i210AT/​bench-lab-3nodes.pkt.config|bench-lab-3nodes.config]]
 +Put on one of these files, all the IP adresses, testing command lines, etc corresponding to your bench test.
 +
 +Then start [[https://​github.com/​ocochard/​netbenches/​blob/​master/​scripts/​bench-lab.sh|bench-lab.sh]] with some parameters.
 +Then start it:
 +<​code>​
 +./​bench-lab.sh -f bench-lab-parameters-file -c configuration-sets-dir -i nanobsd-images-dir -n iteration -d benchs-results-dir -r my@email.org
 +</​code>​
 +
 +And here is an example:
 +<​code>​
 +olivier@lame4:​~/​netbenches/​Xeon_E5-2650-8Cores-Chelsio_T540-CR % ../​scripts/​bench-lab.sh -f bench-lab-2nodes.config -i /​root/​images.2017/​ -c forwarding-pf-ipfw/​config
 +s/ -p ../​pktgen.configs/​dualstack-2k/​ -d forwarding-pf-ipfw/​results/​fbsd.2017/​ -r olivier@FreeBSD.org
 +BSDRP automatized upgrade/​configuration-sets/​benchs script
 +
 +This script will start 210 bench tests using:
 + - Multiples images to test: yes
 + - Multiples configuration-sets to test: yes
 + - Multiples pkt-gen configuration to test: yes
 + - Number of iteration for each set: 5
 + - Results dir: forwarding-pf-ipfw/​results/​fbsd.2017/​
 +
 +Do you want to continue ? (y/n): y
 +Testing ICMP connectivity to each devices:
 +  192.168.1.8...OK
 +  192.168.1.8...OK
 +  192.168.1.10...OK
 +Testing SSH connectivity with key to each devices:
 +  192.168.1.8...OK
 +  192.168.1.8...OK
 +  192.168.1.10...OK
 +Starting the benchs
 +Start firmware image set: /​root/​images.2017//​BSDRP-311014-upgrade-amd64-serial.img.xz
 +Upgrading...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start configuration set: forwarding
 +Uploading cfg forwarding-pf-ipfw/​configs//​forwarding to 192.168.1.10
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start pkt-gen set: ../​pktgen.configs/​dualstack-2k//​inet4
 +Start bench serie bench.311014.forwarding.inet4.1
 +Waiting for end of bench 1/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet4.2
 +Waiting for end of bench 2/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet4.3
 +Waiting for end of bench 3/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet4.4
 +Waiting for end of bench 4/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet4.5
 +Waiting for end of bench 5/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start pkt-gen set: ../​pktgen.configs/​dualstack-2k//​inet6
 +Start bench serie bench.311014.forwarding.inet6.1
 +Waiting for end of bench 6/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet6.2
 +Waiting for end of bench 7/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet6.3
 +Waiting for end of bench 8/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet6.4
 +Waiting for end of bench 9/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start bench serie bench.311014.forwarding.inet6.5
 +Waiting for end of bench 10/​210...done
 +Rebooting 192.168.1.10 and waiting device return online...done
 +Start configuration set: ipfw-statefull
 +Uploading cfg forwarding-pf-ipfw/​configs//​ipfw-statefull to 192.168.1.10
 +Rebooting 192.168.1.10 and waiting device return online...done
 +...
 +Rebooting DUT and waiting device return online...done
 +Waiting for end of bench 209/​210...done
 +Rebooting DUT and waiting device return online...done
 +Waiting for end of bench 210/​210...done
 +All bench tests were done, results in forwarding-pf-ipfw/​results/​fbsd.2017/​
 +</​code>​
 +===== Results generation =====
 +
 +Once bench finished, there are lot's of raw result files with these names: ​
 +  * bench.NANOBSD-SVN.CONFIGURATION-SETS.info:​ Give resumed information about the nanobsd image and configuration set name used
 +  * bench.NANOBSD-SVN.CONFIGURATION-SETS.ITERATION.receiver:​ raw output of the ssh session on the receiver
 +  * bench.NANOBSD-SVN.CONFIGURATION-SETS.ITERATION.sender:​ raw output of the ssh session on the sender
 +
 +Here is [[https://​github.com/​ocochard/​netbenches/​blob/​master/​AMD_GX-412TC_4Cores_Intel_i210AT/​bench-lab-2nodes.config|an example of these raw results]].
 +
 +These raw results, closely dependents of the bench tool used, need to be filtered (and synthesized) in a usable data file.
 +
 +BSDRP use the script [[https://​github.com/​ocochard/​netbenches/​blob/​master/​scripts/​bench-lab-ministat.sh|bench-lab-ministat.sh]] for filtering netmap pkt-gen output with a first pass of ministat and generate a list of files named SVN-REV.CONFIG-SETS-NAME that include N line of result (with N=number of test iterations) and gnuplot.data file.
documentation/examples/freebsd_performance_regression_lab.txt ยท Last modified: 2017/03/29 09:32 (external edit)