documentation:examples:setting_up_a_forwarding_performance_benchmark_lab
no way to compare when less than two revisions
Differences
This shows you the differences between two versions of the page.
Next revision | |||
— | documentation:examples:setting_up_a_forwarding_performance_benchmark_lab [2017/04/04 16:32] – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Setting-up a forwarding performance benchmark lab ====== | ||
+ | {{description> | ||
+ | ===== Benchmark Methodology ===== | ||
+ | |||
+ | Before to start a router benchmark, some RFC to read: | ||
+ | * Benchmarking Terminology for Network Interconnection Devices: [[https:// | ||
+ | * Benchmarking Methodology for Network Interconnect Devices: | ||
+ | * Terminology for Forwarding Information Base (FIB) based Router Performance: | ||
+ | * IPv6 Benchmarking Methodology for Network Interconnect Devices: [[https:// | ||
+ | * Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful [[https:// | ||
+ | |||
+ | But, we will start by a more simple lab by focusing on a wire-speed (or line-rate) packet-generator. | ||
+ | We didn't cover firewall benchmark ([[https:// | ||
+ | |||
+ | Reading [[documentation: | ||
+ | |||
+ | A wire-speed packet generator should be able to generate the [[http:// | ||
+ | |||
+ | ===== Simplified methodology ===== | ||
+ | |||
+ | Method used here is to bench only the " | ||
+ | - Only smallest size packet (20 bytes for IPv4, 40 bytes for IPv6) | ||
+ | - Offered load is the maximum line-rate of the medium (like under a Denial of Service attack) | ||
+ | |||
+ | And the value we are interested by will be the maximum forwarding rate measured at the receiving side. This mean without take care of the dropped packet. | ||
+ | |||
+ | Once obtain this " | ||
+ | |||
+ | ===== Diagram ===== | ||
+ | |||
+ | For cross-checking the packet counters, we will connect our devices to a non-blocking switch that have its own traffic counters. | ||
+ | |||
+ | We can use the same device for packet generator/ | ||
+ | |||
+ | < | ||
+ | +---------------------------+ | ||
+ | | Packet generator/ | ||
+ | +---------------------------+ | ||
+ | | ||
+ | | ||
+ | +----------------------------------------------------------+ | ||
+ | | Non-blocking switch with counters | ||
+ | +----------------------------------------------------------+ | ||
+ | </ | ||
+ | |||
+ | or using two different device for packet generator and receiver: | ||
+ | |||
+ | < | ||
+ | +------------------+ | ||
+ | | Packet generator | | Device Under Test | | Packet receiver | | ||
+ | +------------------+ | ||
+ | | ||
+ | | ||
+ | +------------------------------------------------------------------+ | ||
+ | | Non-blocking switch with counters | ||
+ | +------------------------------------------------------------------+ | ||
+ | </ | ||
+ | |||
+ | Or for a lab without switch: | ||
+ | Still the same device for packet generator/ | ||
+ | |||
+ | < | ||
+ | +---------------------------+ | ||
+ | | Packet generator/ | ||
+ | +---------------------------+ | ||
+ | | ||
+ | | ||
+ | +-------------------------+ | ||
+ | | Device Under Test (DUT) | | ||
+ | +-------------------------+ | ||
+ | </ | ||
+ | |||
+ | |||
+ | ===== Switch configuration ===== | ||
+ | |||
+ | We need to take care of the switch configuration: | ||
+ | * The device used as " | ||
+ | * The switch need to be configured for not sending non-desirable frame too (spanning tree, keepalive, CDP, etc…) that will fake our counters; | ||
+ | * Allowing Ethernet flow control is not a good idea (it's allays better to drop some packet and let TCP windowing slow down the connection speed), but it's even worse when we want to build a high speed packet generator. | ||
+ | |||
+ | ==== Disabling Spanning-Tree ==== | ||
+ | |||
+ | Disable STP (if you know what you are doing) on the vlan or interfaces used for the bench. | ||
+ | |||
+ | === Cisco switch === | ||
+ | |||
+ | < | ||
+ | no spanning-tree vlan 2 | ||
+ | no spanning-tree vlan 3 | ||
+ | </ | ||
+ | |||
+ | === Juniper switch === | ||
+ | |||
+ | < | ||
+ | set protocols rstp interface xe0/0/0 disable | ||
+ | </ | ||
+ | |||
+ | ==== Mac-address Table aging or static MAC ==== | ||
+ | |||
+ | Then disabling mac-address-table aging too or configure static MAC entry (this avoid to use a ping before starting the test for populating the dynamic MAC table): | ||
+ | |||
+ | === Cisco switch === | ||
+ | < | ||
+ | mac address-table aging-time 0 vlan 2 | ||
+ | mac address-table aging-time 0 vlan 3 | ||
+ | </ | ||
+ | |||
+ | or: | ||
+ | |||
+ | < | ||
+ | mac address-table static 000e.0cde.45df vlan 2 interface GigabitEthernet0/ | ||
+ | mac address-table static 001a.6478.267a vlan 2 interface GigabitEthernet0/ | ||
+ | mac address-table static 001b.21d5.660e vlan 3 interface GigabitEthernet0/ | ||
+ | mac address-table static 001b.21d5.6615 vlan 3 interface GigabitEthernet0/ | ||
+ | </ | ||
+ | |||
+ | === Juniper switch === | ||
+ | |||
+ | < | ||
+ | set vlans bench switch-options interface xe0/0/0 static-mac 000e.0cde.45df | ||
+ | </ | ||
+ | |||
+ | ==== Port==== | ||
+ | |||
+ | Disable CDP, LLDP, DTP, keep-alive and flow-control on all ports connected to the testers and DUT. | ||
+ | Reduce the load-interval for statistics calculation to the minimum (30seconds here). | ||
+ | |||
+ | === Cisco switch === | ||
+ | |||
+ | < | ||
+ | interface GigabitEthernet0/ | ||
+ | description DUT-em0 | ||
+ | switchport access vlan 2 | ||
+ | switchport mode access | ||
+ | switchport nonegotiate | ||
+ | load-interval 30 | ||
+ | no keepalive | ||
+ | no cdp enable | ||
+ | no lldp transmit | ||
+ | flowcontrol send off | ||
+ | flowcontrol receive off | ||
+ | </ | ||
+ | |||
+ | === Juniper switch === | ||
+ | |||
+ | < | ||
+ | set protocols lldp interface xe-0/0/0 disable | ||
+ | set protocols lldp-med interface xe-0/0/0 disable | ||
+ | </ | ||
+ | ===== Configuring the packet generator/ | ||
+ | |||
+ | ==== Hardware ==== | ||
+ | |||
+ | NIC supported by [[http:// | ||
+ | ==== Static ARP ==== | ||
+ | |||
+ | It's better to avoid ARP resolution on the packet generator: | ||
+ | |||
+ | < | ||
+ | sysrc static_arp_pairs=" | ||
+ | sysrc static_arp_tester2=" | ||
+ | sysrc static_arp_dut=" | ||
+ | service static_arp start | ||
+ | </ | ||
+ | |||
+ | ==== NIC drivers tunning ==== | ||
+ | |||
+ | === Disabling Flow-control === | ||
+ | |||
+ | This step depends of the NIC used, igb(4) uses this: | ||
+ | |||
+ | < | ||
+ | cat >> / | ||
+ | dev.igb.0.fc=0 | ||
+ | dev.igb.1.fc=0 | ||
+ | dev.igb.2.fc=0 | ||
+ | dev.igb.3.fc=0 | ||
+ | EOF | ||
+ | </ | ||
+ | |||
+ | === Unleashing the power of NIC chipset === | ||
+ | |||
+ | By default FreeBSD use timid drivers values, but BSDRP increase them ([[documentation: | ||
+ | |||
+ | As example for em(4) or igb(4) drivers: | ||
+ | * Disable the limit of maximum number of received packets to process at a time: | ||
+ | * hw.em|igb.rx_process_limit 100 => -1 | ||
+ | * Increase number of transmit and receive descriptors per queue to their maximum: | ||
+ | * hw.em|igb.txd 1024 => 2048 | ||
+ | * hw.em|igb.rxd 1024 => 2048 | ||
+ | * Increase the maximum number of interrupts per second generated | ||
+ | * hw.igb.max_interrupt_rate 8000 => 16000 | ||
+ | |||
+ | If you understand all the parameters on the document [[http:// | ||
+ | ===== Testing the packet generator and receiver ===== | ||
+ | |||
+ | We need to learn to use the packet generator/ | ||
+ | |||
+ | ==== Diagram ==== | ||
+ | |||
+ | < | ||
+ | +------------------+ | ||
+ | | Packet generator |=====>| Packet receiver | | ||
+ | +------------------+ | ||
+ | </ | ||
+ | |||
+ | ==== Preparation ==== | ||
+ | |||
+ | Before to start, if you didn't use static MAC entry on the switch or static ARP on your generator, ping each other for forcing them to emitting at last one packet. | ||
+ | |||
+ | Then clear the counter on the switch | ||
+ | < | ||
+ | switch# | ||
+ | Clear "show interface" | ||
+ | </ | ||
+ | |||
+ | |||
+ | ==== netmap pkt-gen ==== | ||
+ | |||
+ | <note warning> | ||
+ | You need to use a FreeBSD -head with svn revision of 257758 minimum and if you are using Intel NIC you need [[https:// | ||
+ | </ | ||
+ | |||
+ | We need to generate: | ||
+ | * multi-flows (important for using NIC multi-queue features), these mean using multiple IP as source and destination | ||
+ | * Smallest Ethernet frame size (64 bytes including CRC) | ||
+ | |||
+ | Here is an example for: | ||
+ | * 2000 flows by using a source range of IP 198.18.0.1-198.18.0.100 and destination 198.19.0.1 to 198.18.20 | ||
+ | * The source and destination UDP port are 2000 (it's important to specify the port source and destination for avoiding the usage of port number 0 filtered by pf) | ||
+ | * The destination MAC address needs to be given | ||
+ | * We generate 1 Billion packets (10 Giga ethernet links are fast) | ||
+ | * with a 4 second timer for the link be ready (pkt-gen down/up the link) | ||
+ | * 60 bytes Ethernet Frame (pkt-gen didn't include the 4 bytes CRC size) | ||
+ | |||
+ | < | ||
+ | [root@generator]~# | ||
+ | 641.226408 main [1750] interface is ix0 | ||
+ | 641.226502 extract_ip_range [293] range is 198.18.0.1: | ||
+ | 641.226510 extract_ip_range [293] range is 198.19.0.1: | ||
+ | 641.283643 main [1950] mapped 334980KB at 0x801c00000 | ||
+ | Sending on netmap:ix0: 4 queues, 1 threads and 1 cpus. | ||
+ | 198.18.0.1 -> 198.19.0.1 (00: | ||
+ | 641.283669 main [2012] --- SPECIAL OPTIONS: copy | ||
+ | |||
+ | 641.283671 main [2034] Sending 512 packets every 0.000000000 s | ||
+ | 641.283674 main [2036] Wait 4 secs for phy reset | ||
+ | 645.285310 main [2038] Ready... | ||
+ | 645.285352 nm_open [456] overriding ifname ix0 ringid 0x0 flags 0x1 | ||
+ | 645.285433 sender_body [1106] start, fd 4 main_fd 3 | ||
+ | 646.287313 main_thread [1547] 12384692 pps (12408099 pkts in 1001890 usec) | ||
+ | 647.289311 main_thread [1547] 12099481 pps (12123644 pkts in 1001997 usec) | ||
+ | 648.290311 main_thread [1547] 12479753 pps (12492233 pkts in 1001000 usec) | ||
+ | 649.291310 main_thread [1547] 12478188 pps (12490666 pkts in 1001000 usec) | ||
+ | 650.293310 main_thread [1547] 12475803 pps (12500755 pkts in 1002000 usec) | ||
+ | 651.294310 main_thread [1547] 12478549 pps (12491028 pkts in 1001000 usec) | ||
+ | 652.295984 main_thread [1547] 12480828 pps (12501721 pkts in 1001674 usec) | ||
+ | 653.297310 main_thread [1547] 12481843 pps (12498394 pkts in 1001326 usec) | ||
+ | 654.299310 main_thread [1547] 12478788 pps (12503746 pkts in 1002000 usec) | ||
+ | 655.301310 main_thread [1547] 12475746 pps (12500697 pkts in 1002000 usec) | ||
+ | 656.303309 main_thread [1547] 12477035 pps (12501977 pkts in 1001999 usec) | ||
+ | 657.305309 main_thread [1547] 12479925 pps (12504885 pkts in 1002000 usec) | ||
+ | 658.306310 main_thread [1547] 12481029 pps (12493510 pkts in 1001000 usec) | ||
+ | 659.307312 main_thread [1547] 12479999 pps (12492516 pkts in 1001003 usec) | ||
+ | 660.308309 main_thread [1547] 12477386 pps (12489826 pkts in 1000997 usec) | ||
+ | ... | ||
+ | </ | ||
+ | |||
+ | => It sends at about 12.4Mpps (the line-rate is 14.8Mpps). | ||
+ | ==== Don't use netstat ' | ||
+ | |||
+ | During the tests, we meet a problem with netstat: | ||
+ | |||
+ | The receiver measure about 565Kpps: | ||
+ | < | ||
+ | main_thread [1078] 564842 pps (565406 pkts in 1000998 usec) | ||
+ | main_thread [1078] 564443 pps (565009 pkts in 1001002 usec) | ||
+ | main_thread [1078] 564822 pps (565387 pkts in 1001000 usec) | ||
+ | main_thread [1078] 565082 pps (565647 pkts in 1001000 usec) | ||
+ | main_thread [1078] 565139 pps (565704 pkts in 1000999 usec) | ||
+ | main_thread [1078] 565122 pps (565686 pkts in 1000998 usec) | ||
+ | main_thread [1078] 564752 pps (565318 pkts in 1001002 usec) | ||
+ | </ | ||
+ | |||
+ | Now what about the router stats: | ||
+ | |||
+ | < | ||
+ | [root@BSDRP]~# | ||
+ | input (Total) | ||
+ | | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | 551k 906k | ||
+ | </ | ||
+ | |||
+ | The router display a forwarding rate of 552Kpps: There is a 10Kpps gap between the receiver and router stats. | ||
+ | |||
+ | We need to check the switch stats for a tie: | ||
+ | |||
+ | < | ||
+ | switch#sh int GigabitEthernet 1/0/6 | i output rate | ||
+ | 30 second output rate 289701000 bits/sec, 565821 packets/sec | ||
+ | </ | ||
+ | |||
+ | => Switch stats confirm the number of 565Kpps received. | ||
+ | |||
+ | There were a problem with FreeBSD self-counters that misses about 10Kpps in this case. Hopefully a contributer give me an hint: 564842 / 1024 = 551.6 Kpps. The [[http:// | ||
+ | |||
+ | If we call netstat without ' | ||
+ | < | ||
+ | [root@BSDRP]~# | ||
+ | input (Total) | ||
+ | | ||
+ | 564224 927744 | ||
+ | 564444 927822 | ||
+ | 564504 928024 | ||
+ | 564842 929645 | ||
+ | </ | ||
+ | |||
+ | Now we can try to bench your servers for a router usage, like [[documentation: | ||
+ | |||
+ | ===== Bench reproducibility and ministat ===== | ||
+ | |||
+ | Once your test ready you need to run it multiple time without human interaction: | ||
+ | * Need to script the test like on the [[documentation: | ||
+ | * Doing multiples times and publishing the [[https:// |
documentation/examples/setting_up_a_forwarding_performance_benchmark_lab.txt · Last modified: 2020/12/30 08:42 by olivier