documentation:examples:forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580
no way to compare when less than two revisions
Differences
This shows you the differences between two versions of the page.
— | documentation:examples:forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580 [2018/04/18 16:39] (current) – created - external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Forwarding performance lab of an IBM System x3550 M3 with Intel 82580 ====== | ||
+ | {{description> | ||
+ | ===== Hardware detail ===== | ||
+ | This lab will test an [[IBM System x3550 M3]] with **quad** cores (Intel Xeon L5630 2.13GHz, hyper-threading disabled) and a quad NIC 82580 connected to the PCI-Express Bus. | ||
+ | |||
+ | ===== Lab set-up ===== | ||
+ | |||
+ | The lab is detailed here: [[documentation: | ||
+ | |||
+ | BSDRP-amd64 v1.51 (FreeBSD 10.0-BETA2 with autotune mbuf patch) is used on the DUT. | ||
+ | ==== Diagram ==== | ||
+ | |||
+ | < | ||
+ | +------------------------------------------+ +--------+ +------------------------------+ | ||
+ | | Device under test | | Cisco | | Packet generator & receiver | ||
+ | | | |Catalyst| | | | ||
+ | | igb2: 198.18.0.203/ | ||
+ | | 2001: | ||
+ | | (00: | ||
+ | | | | | | | | ||
+ | | igb3: 198.19.0.203/ | ||
+ | | 2001: | ||
+ | | (00: | ||
+ | | | | | | ||
+ | | static routes | ||
+ | | 192.18.0.0/ | ||
+ | | 192.19.0.0/ | ||
+ | | 2001: | ||
+ | | 2001: | ||
+ | | | | | | ||
+ | | static arp and ndp | | | | ||
+ | | 198.18.0.208 | ||
+ | | 2001: | ||
+ | | | | | | ||
+ | | 198.19.0.208 | ||
+ | | 2001: | ||
+ | +------------------------------------------+ | ||
+ | </ | ||
+ | |||
+ | The generator **MUST** generate lot's of smallest IP flows (multiple source/ | ||
+ | |||
+ | Here is an example for generating about 2000 flows: | ||
+ | < | ||
+ | pkt-gen -N -f tx -i igb1 -n 1000000000 -4 -d 198.19.10.1: | ||
+ | </ | ||
+ | |||
+ | Receiver will use this command: | ||
+ | < | ||
+ | pkt-gen -N -f rx -i igb2 -w 4 | ||
+ | </ | ||
+ | ===== Basic configuration ===== | ||
+ | |||
+ | ==== Disabling Ethernet flow-control === | ||
+ | |||
+ | First, disable Ethernet flow-control: | ||
+ | < | ||
+ | echo " | ||
+ | echo " | ||
+ | sysctl dev.igb.2.fc=0 | ||
+ | sysctl dev.igb.3.fc=0 | ||
+ | </ | ||
+ | |||
+ | ==== IP Configuration ==== | ||
+ | |||
+ | Configure IP addresses, static routes and static ARP entries. | ||
+ | |||
+ | A router [[Documentation: | ||
+ | |||
+ | < | ||
+ | sysrc static_routes=" | ||
+ | sysrc route_generator=" | ||
+ | sysrc route_receiver=" | ||
+ | sysrc ifconfig_igb2=" | ||
+ | sysrc ifconfig_igb3=" | ||
+ | sysrc static_arp_pairs=" | ||
+ | sysrc static_arp_generator=" | ||
+ | sysrc static_arp_receiver=" | ||
+ | </ | ||
+ | |||
+ | ===== Default forwarding speed ===== | ||
+ | |||
+ | With the default parameters, multi-flow traffic at 1.488Mpps (the maximum rate for GigaEthernet) are correctly forwarded without any loss: | ||
+ | |||
+ | < | ||
+ | [root@BSDRP]~# | ||
+ | input (Total) | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | </ | ||
+ | |||
+ | The traffic is correctly load-balanced between NIC-queue/ | ||
+ | < | ||
+ | [root@BSDRP]# | ||
+ | irq278: igb2:que 0 2759545998 | ||
+ | irq279: igb2:que 1 2587966938 | ||
+ | irq280: igb2:que 2 2589102074 | ||
+ | irq281: igb2:que 3 2598239184 | ||
+ | irq282: igb2: | ||
+ | irq283: igb3:que 0 3318777087 | ||
+ | irq284: igb3:que 1 3098055250 | ||
+ | irq285: igb3:que 2 3101570541 | ||
+ | irq286: igb3:que 3 3052431966 | ||
+ | irq287: igb3: | ||
+ | |||
+ | [root@BSDRP]/# | ||
+ | last pid: 8292; load averages: | ||
+ | 153 processes: 12 running, 97 sleeping, 44 waiting | ||
+ | |||
+ | Mem: 2212K Active, 24M Inact, 244M Wired, 18M Buf, 15G Free | ||
+ | Swap: | ||
+ | |||
+ | |||
+ | PID USERNAME PRI NICE | ||
+ | 11 root | ||
+ | 11 root | ||
+ | 11 root | ||
+ | 11 root | ||
+ | 11 root | ||
+ | 11 root | ||
+ | 11 root | ||
+ | 11 root | ||
+ | </ | ||
+ | |||
+ | ===== igb(4) drivers tunning with 82546GB ===== | ||
+ | |||
+ | ==== Disabling multi-queue ==== | ||
+ | |||
+ | For disabling multi-queue (this mean without IRQ load-sharing between CPUs), there are 2 methods: | ||
+ | |||
+ | The first method is to use pkt-gen for generating a one IP flow (same src/dst IP and same src/dst port) traffic like this: | ||
+ | < | ||
+ | pkt-gen -i igb2 -f tx -n 80000000 -l 42 -d 2.3.3.2 -D 00: | ||
+ | </ | ||
+ | => With this method, igb(4) can't do load-balancing input traffic and will use only one queue. | ||
+ | |||
+ | The second method is to disabling the multi-queue support of igb(4) drivers by forcing the usage of one queue: | ||
+ | < | ||
+ | mount -uw / | ||
+ | echo ' | ||
+ | mount -ur / | ||
+ | reboot | ||
+ | </ | ||
+ | |||
+ | And check on the dmesg or with number of IRQ assigned to the NIC that no multi-queue was enabled: | ||
+ | < | ||
+ | [root@BSDRP]~# | ||
+ | igb2: < | ||
+ | igb2: Using MSIX interrupts with 2 vectors | ||
+ | igb2: Ethernet address: 00: | ||
+ | 001.000011 netmap_attach [2244] success for igb2 | ||
+ | igb3: < | ||
+ | igb3: Using MSIX interrupts with 2 vectors | ||
+ | igb3: Ethernet address: 00: | ||
+ | 001.000012 netmap_attach [2244] success for igb3 | ||
+ | |||
+ | [root@BSDRP]~# | ||
+ | irq272: igb2:que 0 | ||
+ | irq273: igb2: | ||
+ | irq274: igb3:que 0 48517905 | ||
+ | irq275: igb3: | ||
+ | </ | ||
+ | |||
+ | Using any of theses method, the result will be the same: forwarding speed will decrease (to about 700Kpps) corresponding to the maximum input rate (100% CPU usage of the unique CPU bound to input NIC IRQ). | ||
+ | < | ||
+ | [root@BSDRP]~# | ||
+ | input (Total) | ||
+ | | ||
+ | 690541 797962 | ||
+ | 704171 797906 | ||
+ | 676522 797770 | ||
+ | 707373 797878 | ||
+ | 672983 797962 | ||
+ | 705339 797899 | ||
+ | 684930 798049 | ||
+ | [root@bsdrp2]~# | ||
+ | last pid: 2930; load averages: | ||
+ | 129 processes: 8 running, 84 sleeping, 37 waiting | ||
+ | |||
+ | Mem: 13M Active, 8856K Inact, 203M Wired, 9748K Buf, 15G Free | ||
+ | Swap: | ||
+ | |||
+ | |||
+ | PID USERNAME PRI NICE | ||
+ | 11 root | ||
+ | 11 root | ||
+ | 0 root | ||
+ | |||
+ | </ | ||
+ | ==== hw.igb.rx_process_limit and hw.igb.txd/ | ||
+ | |||
+ | What are the impact of modifying hw.igb.rx_process_limit and hw.igb.txd/ | ||
+ | |||
+ | We need to overload this NIC for this test, this meaning using this NIC without multi-queue. | ||
+ | |||
+ | === graphical result === | ||
+ | Here are the results of one-flow | ||
+ | |||
+ | {{: | ||
+ | |||
+ | === Ministat graphs === | ||
+ | |||
+ | == txd/rxd fixed at 1024, rx_process_limit variable == | ||
+ | |||
+ | < | ||
+ | x xd1024.proc_lim-1 | ||
+ | + xd1024.proc_lim100 | ||
+ | * xd1024.proc_lim500 | ||
+ | +----------------------------------------------------------------------------------------------+ | ||
+ | |* + + ++ | ||
+ | | | ||
+ | | |_________AM_______| | ||
+ | | | ||
+ | +----------------------------------------------------------------------------------------------+ | ||
+ | N | ||
+ | x | ||
+ | + | ||
+ | Difference at 95.0% confidence | ||
+ | -34305.2 +/- 8390.76 | ||
+ | -6.13154% +/- 1.49972% | ||
+ | (Student' | ||
+ | * | ||
+ | Difference at 95.0% confidence | ||
+ | -28985.6 +/- 11520 | ||
+ | -5.18074% +/- 2.05903% | ||
+ | (Student' | ||
+ | </ | ||
+ | |||
+ | == txd/rxd fixed at 2048, rx_process_limit variable == | ||
+ | |||
+ | < | ||
+ | x xd2048.proc_lim-1 | ||
+ | + xd2048.proc_lim100 | ||
+ | * xd2048.proc_lim500 | ||
+ | +----------------------------------------------------------------------------------------------+ | ||
+ | |+** +* | ||
+ | | |___AM__| | | ||
+ | ||______MA_______| | ||
+ | ||_______M_A_________| | ||
+ | +----------------------------------------------------------------------------------------------+ | ||
+ | N | ||
+ | x | ||
+ | + | ||
+ | Difference at 95.0% confidence | ||
+ | -34710.8 +/- 3990.14 | ||
+ | -6.13516% +/- 0.70526% | ||
+ | (Student' | ||
+ | * | ||
+ | Difference at 95.0% confidence | ||
+ | -33896.4 +/- 4765.66 | ||
+ | -5.99122% +/- 0.842335% | ||
+ | (Student' | ||
+ | |||
+ | </ | ||
+ | |||
+ | == txd/rxd fixed at 4096, rx_process_limit variable == | ||
+ | |||
+ | < | ||
+ | x xd4096.proc_lim-1 | ||
+ | + xd4096.proc_lim100 | ||
+ | * xd4096.proc_lim500 | ||
+ | +----------------------------------------------------------------------------------------------+ | ||
+ | | * | | ||
+ | | * | | ||
+ | |+ | ||
+ | | | ||
+ | | |________A_______| | ||
+ | | | ||
+ | +----------------------------------------------------------------------------------------------+ | ||
+ | N | ||
+ | x | ||
+ | + | ||
+ | Difference at 95.0% confidence | ||
+ | -33541.4 +/- 6382.78 | ||
+ | -5.96771% +/- 1.13563% | ||
+ | (Student' | ||
+ | * | ||
+ | Difference at 95.0% confidence | ||
+ | -31077.8 +/- 4924.78 | ||
+ | -5.52939% +/- 0.87622% | ||
+ | (Student' | ||
+ | </ | ||
+ | |||
+ | ===== Firewall impact ===== | ||
+ | |||
+ | Multi-queue is re-enabled for this test, and best value from the previous tests used: | ||
+ | * hw.igb.rxd=2048 | ||
+ | * hw.igb.txd=2048 | ||
+ | * hw.igb.rx_process_limit=-1 (disabled) | ||
+ | * hw.igb.num_queues=0 (automatically based on number of CPUs and max supported MSI-X messages = 4 on this lab hardware) | ||
+ | |||
+ | This test will generate 2000 different flows by using 2000 different UDP destination ports: | ||
+ | < | ||
+ | pkt-gen -i igb2 -f tx -l 42 -d 2.3.3.1: | ||
+ | </ | ||
+ | ==== IPFW ==== | ||
+ | |||
+ | === Stateless === | ||
+ | Now we will test the impact of enabling simple stateless IPFW rules: | ||
+ | |||
+ | < | ||
+ | cat > / | ||
+ | #!/bin/sh | ||
+ | fwcmd="/ | ||
+ | # Flush out the list before we begin. | ||
+ | ${fwcmd} -f flush | ||
+ | ${fwcmd} add 3000 allow ip from any to any | ||
+ | ' | ||
+ | |||
+ | sysrc firewall_enable=" | ||
+ | sysrc firewall_script="/ | ||
+ | |||
+ | </ | ||
+ | |||
+ | === Statefull === | ||
+ | |||
+ | Now we will test the impact of enabling simple statefull IPFW rules: | ||
+ | |||
+ | < | ||
+ | cat > / | ||
+ | #!/bin/sh | ||
+ | fwcmd="/ | ||
+ | # Flush out the list before we begin. | ||
+ | ${fwcmd} -f flush | ||
+ | ${fwcmd} add 3000 allow ip from any to any keep-state | ||
+ | ' | ||
+ | |||
+ | service ipfw restart | ||
+ | </ | ||
+ | ==== PF ==== | ||
+ | |||
+ | === Stateless === | ||
+ | Now we will test the impact of enabling simple stateless PF rules: | ||
+ | |||
+ | < | ||
+ | cat >/ | ||
+ | set skip on lo0 | ||
+ | pass no state | ||
+ | ' | ||
+ | |||
+ | sysrc pf_enable=" | ||
+ | </ | ||
+ | |||
+ | === Statefull === | ||
+ | Now we will test the impact of enabling simple statefull PF rules: | ||
+ | |||
+ | < | ||
+ | cat >/ | ||
+ | set skip on lo0 | ||
+ | pass | ||
+ | ' | ||
+ | |||
+ | sysrc pf_enable=" | ||
+ | </ | ||
+ | ==== Results ==== | ||
+ | |||
+ | === Graph === | ||
+ | |||
+ | scale information: | ||
+ | |||
+ | {{documentation: | ||
+ | |||
+ | === ministat === | ||
+ | |||
+ | < | ||
+ | x pps.fastforwarding | ||
+ | + pps.ipfw-stateless | ||
+ | * pps.ipfw-statefull | ||
+ | % pps.pf-stateless | ||
+ | # pps.pf-statefull | ||
+ | +------------------------------------------------------------------------------------------------+ | ||
+ | |% %%% % # #O # * * # | ||
+ | | A| | ||
+ | | A| | ||
+ | | | ||
+ | | |__A__| | ||
+ | | | ||
+ | +------------------------------------------------------------------------------------------------+ | ||
+ | N | ||
+ | x | ||
+ | + | ||
+ | No difference proven at 95.0% confidence | ||
+ | * | ||
+ | Difference at 95.0% confidence | ||
+ | -156475 +/- 30196.9 | ||
+ | -10.5148% +/- 2.02917% | ||
+ | (Student' | ||
+ | % | ||
+ | Difference at 95.0% confidence | ||
+ | -210987 +/- 7130.55 | ||
+ | -14.1779% +/- 0.479158% | ||
+ | (Student' | ||
+ | # | ||
+ | Difference at 95.0% confidence | ||
+ | -183253 +/- 12649.1 | ||
+ | -12.3142% +/- 0.849995% | ||
+ | (Student' | ||
+ | </ |
documentation/examples/forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580.txt · Last modified: 2018/04/18 16:39 by 127.0.0.1