documentation:examples:forwarding_performance_lab_of_a_netgate_rcc-ve_4860
no way to compare when less than two revisions
Differences
This shows you the differences between two versions of the page.
Next revision | |||
— | documentation:examples:forwarding_performance_lab_of_a_netgate_rcc-ve_4860 [2019/03/25 12:38] – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== Forwarding performance lab of Netgate RCC-VE 4860 ====== | ||
+ | {{description> | ||
+ | ===== Hardware detail ===== | ||
+ | This lab will test a [[http:// | ||
+ | * Quad cores Intel Atom C2558 (2.40GHz) | ||
+ | * 2 Gigabit Intel i211 | ||
+ | * 4 Gigabit Intel i350 | ||
+ | * 8Gb of RAM | ||
+ | ===== Lab set-up ===== | ||
+ | |||
+ | For more information about full setup of this lab: [[documentation: | ||
+ | |||
+ | BSDRP 1.92 is used (based on FreeBSD 12-stable r345325) is used. | ||
+ | |||
+ | ==== Diagram ==== | ||
+ | |||
+ | < | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | | 2001: | ||
+ | | 2001: | ||
+ | | ||
+ | | ||
+ | | 198.18.0.203 | ||
+ | | 2001: | ||
+ | | ||
+ | | 198.19.0.203 | ||
+ | | 2001: | ||
+ | | ||
+ | | ||
+ | | ||
+ | |||
+ | </ | ||
+ | |||
+ | This device use 2 kinds of Intel NIC: | ||
+ | * igb0 and igb1: Intel i211 with 2 queues, should be used for admin purpose | ||
+ | * igb2 to igb5: Intel i350 with 4 queues (and iPXE support) should be used for forwarding/ | ||
+ | |||
+ | |||
+ | The generator **MUST** generate lot's of IP flows (multiple source/ | ||
+ | |||
+ | Multiple source/ | ||
+ | < | ||
+ | pkt-gen -i igb2 -f tx -n 80000000 -l 60 -d 198.19.10.1: | ||
+ | </ | ||
+ | |||
+ | And the same with IPv6 flows (minimum frame size of 62 for having a correct empty UDP packet): | ||
+ | < | ||
+ | pkt-gen -f tx -i igb2 -n 1000000000 -l 62 -6 -d " | ||
+ | </ | ||
+ | |||
+ | Receiver will use these commands: | ||
+ | < | ||
+ | pkt-gen -i igb3 -f rx -w 4 | ||
+ | </ | ||
+ | |||
+ | ===== Basic configuration and tuning ===== | ||
+ | |||
+ | ==== Disabling Ethernet flow-control === | ||
+ | |||
+ | First, disable Ethernet flow-control: | ||
+ | < | ||
+ | echo " | ||
+ | echo " | ||
+ | service sysctl restart | ||
+ | </ | ||
+ | |||
+ | ====Enabling Tx abdicate (iflib drivers) === | ||
+ | |||
+ | Second, if NIC drivers is iflib based you should enable TX abdicate (automatically done with BSDRP but not on a generic FreeBSD): | ||
+ | < | ||
+ | cat <<EOF >> / | ||
+ | # Enabling Tx abdicate | ||
+ | dev.igb.0.iflib.tx_abdicate=1 | ||
+ | dev.igb.1.iflib.tx_abdicate=1 | ||
+ | dev.igb.2.iflib.tx_abdicate=1 | ||
+ | dev.igb.3.iflib.tx_abdicate=1 | ||
+ | dev.igb.4.iflib.tx_abdicate=1 | ||
+ | dev.igb.5.iflib.tx_abdicate=1 | ||
+ | EOF | ||
+ | service sysctl restart | ||
+ | </ | ||
+ | |||
+ | ==== Disabling ICMP redirect === | ||
+ | |||
+ | ICMP redirect is enabled by default on a generic FreeBSD (it's disabled by default on BSDRP), and this feature disable the fast tryforward code path. | ||
+ | |||
+ | < | ||
+ | cat <<EOF >> / | ||
+ | # Enabling fastforwarding by disabling ICMP redirect | ||
+ | net.inet.ip.redirect=0 | ||
+ | net.inet6.ip6.redirect=0 | ||
+ | EOF | ||
+ | service sysctl restart | ||
+ | </ | ||
+ | ==== IP Configuration ==== | ||
+ | |||
+ | Configure IP addresses, static ARP entries and static routes. | ||
+ | |||
+ | A router [[Documentation: | ||
+ | |||
+ | < | ||
+ | # IPv4 router | ||
+ | gateway_enable=" | ||
+ | ifconfig_igb2=" | ||
+ | ifconfig_igb3=" | ||
+ | static_routes=" | ||
+ | route_generator=" | ||
+ | route_receiver=" | ||
+ | static_arp_pairs=" | ||
+ | static_arp_generator=" | ||
+ | static_arp_receiver=" | ||
+ | |||
+ | # IPv6 router | ||
+ | ipv6_gateway_enable=" | ||
+ | ipv6_activate_all_interfaces=" | ||
+ | ifconfig_igb2_ipv6=" | ||
+ | ifconfig_igb3_ipv6=" | ||
+ | ipv6_static_routes=" | ||
+ | ipv6_route_generator=" | ||
+ | ipv6_route_receiver=" | ||
+ | static_ndp_pairs=" | ||
+ | static_ndp_generator=" | ||
+ | static_ndp_receiver=" | ||
+ | </ | ||
+ | |||
+ | ===== forwarding rate ===== | ||
+ | |||
+ | We start the first test by starting one packet generator at gigabit line-rate (1.488Mpps) the pkt-gen receiver output: | ||
+ | |||
+ | < | ||
+ | 378.931863 main_thread [2514] 1.126 Mpps (1.127 Mpkts 541.181 Mbps in 1001661 usec) 10.97 avg_batch 1014 min_space | ||
+ | 379.933862 main_thread [2514] 1.124 Mpps (1.126 Mpkts 540.628 Mbps in 1002000 usec) 10.79 avg_batch 1015 min_space | ||
+ | 380.934863 main_thread [2514] 1.122 Mpps (1.123 Mpkts 539.050 Mbps in 1001000 usec) 10.87 avg_batch 1015 min_space | ||
+ | 381.936861 main_thread [2514] 1.121 Mpps (1.123 Mpkts 539.214 Mbps in 1001999 usec) 10.94 avg_batch 1015 min_space | ||
+ | 382.938075 main_thread [2514] 1.122 Mpps (1.123 Mpkts 539.200 Mbps in 1001213 usec) 11.11 avg_batch 1015 min_space | ||
+ | 383.938862 main_thread [2514] 1.121 Mpps (1.122 Mpkts 538.636 Mbps in 1000788 usec) 11.26 avg_batch 1014 min_space | ||
+ | 384.940861 main_thread [2514] 1.121 Mpps (1.123 Mpkts 539.078 Mbps in 1001999 usec) 11.40 avg_batch 1015 min_space | ||
+ | 385.979862 main_thread [2514] 1.121 Mpps (1.165 Mpkts 558.969 Mbps in 1039001 usec) 11.38 avg_batch 1014 min_space | ||
+ | 386.980862 main_thread [2514] 1.121 Mpps (1.122 Mpkts 538.379 Mbps in 1000999 usec) 11.69 avg_batch 1015 min_space | ||
+ | 387.982862 main_thread [2514] 1.121 Mpps (1.123 Mpkts 539.269 Mbps in 1002001 usec) 12.01 avg_batch 1015 min_space | ||
+ | 388.983863 main_thread [2514] 1.120 Mpps (1.121 Mpkts 538.263 Mbps in 1001000 usec) 12.33 avg_batch 1016 min_space | ||
+ | 389.998864 main_thread [2514] 1.121 Mpps (1.138 Mpkts 546.116 Mbps in 1015001 usec) 11.92 avg_batch 1014 min_space | ||
+ | </ | ||
+ | |||
+ | It's able to forward at about about 1.12 Mpps. | ||
+ | |||
+ | ===== Firewalls impact ===== | ||
+ | |||
+ | This test will generate 2000 different flows by using a mix of 20 different sources IP and 100 different destination IP. | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | ==== Graph ==== | ||
+ | |||
+ | scale information about Gigabit Ethernet: | ||
+ | * 1.488Mpps is the maximum paquet-per-second (pps) rate with smallest 46 bytes packets. | ||
+ | * 81Kpps is the minimum pps rate with biggest 1500 bytes packets. | ||
+ | |||
+ | {{bench.ipfw.pf.netgate.png|forwarding and firewalling rate on a Netgate RCC-VE 4860 running FreeBSD 10.1}} | ||
+ | |||
+ | |||
+ | |||
+ | ===== Netmap performance ===== | ||
+ | |||
+ | ==== pkt-gen | ||
+ | |||
+ | Intel NIC drivers are netmap ready… what's about the rate with the netmap' | ||
+ | |||
+ | As a receiver (the sender is emitting at 1.48 Mpps): | ||
+ | < | ||
+ | [root@BSDRP]~# | ||
+ | 186.632724 main [1705] interface is igb2 | ||
+ | 186.632937 extract_ip_range [289] range is 10.0.0.1:0 to 10.0.0.1:0 | ||
+ | 186.632963 extract_ip_range [289] range is 10.1.0.1:0 to 10.1.0.1:0 | ||
+ | 186.957406 main [1896] mapped 334980KB at 0x801dff000 | ||
+ | Receiving from netmap: | ||
+ | 186.957504 main [1982] Wait 4 secs for phy reset | ||
+ | 190.958543 main [1984] Ready... | ||
+ | 190.958678 nm_open [456] overriding ifname igb2 ringid 0x0 flags 0x1 | ||
+ | 190.958863 receiver_body [1238] reading from netmap:igb2 fd 4 main_fd 3 | ||
+ | 191.960540 main_thread [1502] 0 pps (0 pkts in 1001704 usec) | ||
+ | 192.022046 receiver_body [1245] waiting for initial packets, poll returns 0 0 | ||
+ | 192.962539 main_thread [1502] 0 pps (0 pkts in 1001999 usec) | ||
+ | 193.040557 receiver_body [1245] waiting for initial packets, poll returns 0 0 | ||
+ | 193.964552 main_thread [1502] 329506 pps (330169 pkts in 1002013 usec) | ||
+ | 194.966546 main_thread [1502] 1488135 pps (1491104 pkts in 1001995 usec) | ||
+ | 195.968552 main_thread [1502] 1488144 pps (1491128 pkts in 1002005 usec) | ||
+ | 196.970547 main_thread [1502] 1488091 pps (1491060 pkts in 1001995 usec) | ||
+ | 197.972545 main_thread [1502] 1488169 pps (1491142 pkts in 1001998 usec) | ||
+ | 198.974550 main_thread [1502] 1488141 pps (1491125 pkts in 1002005 usec) | ||
+ | 199.976507 main_thread [1502] 1488138 pps (1491052 pkts in 1001958 usec) | ||
+ | 200.978546 main_thread [1502] 1488118 pps (1491151 pkts in 1002038 usec) | ||
+ | 201.980545 main_thread [1502] 1488143 pps (1491118 pkts in 1001999 usec) | ||
+ | 202.982551 main_thread [1502] 1488145 pps (1491130 pkts in 1002006 usec) | ||
+ | 203.984386 main_thread [1502] 1488136 pps (1490867 pkts in 1001835 usec) | ||
+ | 204.985545 main_thread [1502] 1488137 pps (1489862 pkts in 1001159 usec) | ||
+ | 205.987544 main_thread [1502] 1488135 pps (1491110 pkts in 1001999 usec) | ||
+ | 206.989545 main_thread [1502] 1488113 pps (1491091 pkts in 1002001 usec) | ||
+ | 207.991543 main_thread [1502] 1488143 pps (1491116 pkts in 1001998 usec) | ||
+ | 208.993545 main_thread [1502] 1488147 pps (1491126 pkts in 1002002 usec) | ||
+ | 209.994551 main_thread [1502] 1488135 pps (1489632 pkts in 1001006 usec) | ||
+ | 210.995544 main_thread [1502] 1488116 pps (1489595 pkts in 1000994 usec) | ||
+ | 211.997543 main_thread [1502] 1488145 pps (1491120 pkts in 1001999 usec) | ||
+ | 212.998542 main_thread [1502] 1488153 pps (1489638 pkts in 1000998 usec) | ||
+ | </ | ||
+ | |||
+ | It support the maximum Ethernet frame rate (1.48Mpps) with netmap as a receiver. | ||
+ | |||
+ | As a packet generator: | ||
+ | < | ||
+ | [root@BSDRP]~# | ||
+ | 550.392443 main [1705] interface is igb2 | ||
+ | 550.392739 extract_ip_range [289] range is 1.1.3.3:0 to 1.1.3.100:0 | ||
+ | 550.392771 extract_ip_range [289] range is 2.1.3.1:0 to 2.1.3.20:0 | ||
+ | 550.717236 main [1896] mapped 334980KB at 0x801dff000 | ||
+ | Sending on netmap: | ||
+ | 1.1.3.3 -> 2.1.3.1 (00: | ||
+ | 550.717342 main [1958] --- SPECIAL OPTIONS: copy | ||
+ | |||
+ | 550.717352 main [1980] Sending 512 packets every 0.000000000 s | ||
+ | 550.717361 main [1982] Wait 4 secs for phy reset | ||
+ | 554.743524 main [1984] Ready... | ||
+ | 554.743659 nm_open [456] overriding ifname igb2 ringid 0x0 flags 0x1 | ||
+ | 554.743854 sender_body [1064] start, fd 4 main_fd 3 | ||
+ | 555.753521 main_thread [1502] 1466573 pps (1480787 pkts in 1009692 usec) | ||
+ | 556.754522 main_thread [1502] 1488145 pps (1489635 pkts in 1001001 usec) | ||
+ | 557.756031 main_thread [1502] 1488145 pps (1490391 pkts in 1001509 usec) | ||
+ | 558.756528 main_thread [1502] 1488112 pps (1488852 pkts in 1000497 usec) | ||
+ | 559.757526 main_thread [1502] 1488130 pps (1489615 pkts in 1000998 usec) | ||
+ | 560.759520 main_thread [1502] 1488142 pps (1491109 pkts in 1001994 usec) | ||
+ | 561.761527 main_thread [1502] 1488130 pps (1491105 pkts in 1001999 usec) | ||
+ | 562.763520 main_thread [1502] 1488135 pps (1491113 pkts in 1002001 usec) | ||
+ | 563.764527 main_thread [1502] 1488119 pps (1489619 pkts in 1001008 usec) | ||
+ | 564.766519 main_thread [1502] 1488164 pps (1491127 pkts in 1001991 usec) | ||
+ | 565.768520 main_thread [1502] 1488116 pps (1491094 pkts in 1002001 usec) | ||
+ | 566.770520 main_thread [1502] 1488151 pps (1491127 pkts in 1002000 usec) | ||
+ | 567.771520 main_thread [1502] 1488112 pps (1489600 pkts in 1001000 usec) | ||
+ | 568.773518 main_thread [1502] 1488140 pps (1491113 pkts in 1001998 usec) | ||
+ | ^C568.885112 sigint_h [326] received control-C on thread 0x801806800 | ||
+ | 568.885184 sender_body [1162] flush tail 350 head 350 on thread 0x801806800 | ||
+ | 568.885222 sender_body [1170] pending tx tail 628 head 582 on ring 0 | ||
+ | 568.885280 sender_body [1170] pending tx tail 637 head 582 on ring 0 | ||
+ | 568.885315 sender_body [1170] pending tx tail 659 head 582 on ring 0 | ||
+ | 568.885393 sender_body [1170] pending tx tail 685 head 582 on ring 0 | ||
+ | 568.885450 sender_body [1170] pending tx tail 709 head 582 on ring 0 | ||
+ | 569.775515 main_thread [1502] 165737 pps (166068 pkts in 1001997 usec) | ||
+ | Sent 21022355 packets, 60 bytes each, in 14.15 seconds. | ||
+ | Speed: 1.49 Mpps Bandwidth: 713.30 Mbps (raw 998.62 Mbps) | ||
+ | </ | ||
+ | |||
+ | No problem in this direction too: Gigabit line-rate is a piece of cake with netmap on this platform. |
documentation/examples/forwarding_performance_lab_of_a_netgate_rcc-ve_4860.txt · Last modified: 2019/12/16 20:51 by olivier