User Tools

Site Tools


documentation:examples:forwarding_performance_lab_of_a_pc_engines_apu2

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

documentation:examples:forwarding_performance_lab_of_a_pc_engines_apu2 [2019/03/25 12:37] (current)
Line 1: Line 1:
 +====== Forwarding performance lab of a PC Engines APU 2 ======
 +{{description>​Forwarding performance lab of a dual core  AMD GX-412TC (1 GHz) with 3 with Intel i210AT Gigabit}}
 +===== Hardware detail =====
 +
 +This lab will test a [[http://​www.pcengines.ch/​apu2.htm|PC Engines APU 2C4]] ([[PC Engines APU2|dmesg]]):​
 +   * Quad core [[https://​www.amd.com/​Documents/​AMDGSeriesSOCProductBrief.pdf|AMD GX-412TC Processor]] (1 GHz and AESNI)
 +   * 3 with Intel i210AT Gigabit
 +   * 4Gb of RAM
 +
 +===== Lab set-up =====
 +
 +For more information about full setup of this lab: [[documentation:​examples:​Setting up a forwarding performance benchmark lab]] (switch configuration,​ etc.).
 +
 +BSDRP release 1.92 based on FreeBSD 12-STABLE is used.
 +==== Diagram ====
 +
 +<​code>​
 + ​+------------------------------------------+ ​     +-----------------------+
 + ​| ​            ​Device under Test            |      |  Packet gen & receiv ​ |
 + ​| ​                                         |      |                       |
 + ​| ​               igb1:    198.18.0.205/​24 ​ |<=====| igb2: 198.18.0.203/​24 |
 + ​| ​                           2001:​2::​8/​64 ​ |      |        2001:​2::​203/​64 |
 + ​| ​                      ​00:​0d:​b9:​41:​ca:​3d ​ |      |     ​00:​1b:​21:​c4:​95:​7a |
 + ​| ​                                         |      |                       |
 + ​| ​               igb2:    198.19.0.205/​24 ​ |=====>| igb3: 198.19.0.203/​24 |
 + ​| ​                    ​2001:​2:​0:​8000::​8/​64 ​ |      | 2001:​2:​0:​8000::​203/​64 |
 + ​| ​                      ​00:​0d:​b9:​41:​ca:​3e ​ |      |     ​00:​1b:​21:​c4:​95:​7b |
 + ​| ​                                         |      |                       |
 + ​| ​              ​static routes ​             |      |                       |
 + ​| ​     198.19.0.0/​16 => 198.19.0.203 ​      ​| ​     |                       |
 + ​| ​     198.18.0.0/​16 => 198.18.0.203 ​      ​| ​     |                       |
 + ​| ​       2001:2::/49 => 2001:​2::​203 ​       |      |                       |
 + | 2001:​2:​0:​8000::/​49 => 2001:​2:​0:​8000::​203 |      |                       |
 + ​| ​                                         |      |                       |
 + ​| ​           static arp and ndp            |      |                       |
 + | 198.18.0.203 ​       => 00:​1b:​21:​c4:​95:​7a |      |                       |
 + | 2001:​2::​203 ​                             |      |                       |
 + ​| ​                                         |      |                       |
 + | 198.19.0.203 ​       => 00:​1b:​21:​c4:​95:​7b |      |                       |
 + | 2001:​2:​0:​8000::​203 ​                      ​| ​     |                       |
 + ​+------------------------------------------+ ​     +-----------------------+
 +
 +</​code>​
 +
 +The generator **MUST** generate lot's of IP flows (multiple source/​destination IP addresses and/or UDP src/dst port) and minimum packet size (for generating maximum packet rate) with one of these commands:
 +
 +Multiple source/​destination IP addresses (don't forget to precise port to use for avoiding to use port number 0 filtered by pf):
 +<​code>​
 +pkt-gen -U -i igb2 -f tx -n 80000000 -l 60 -d 198.19.10.1:​2000-198.19.10.20 -D 00:​0d:​b9:​41:​ca:​3d -s 198.18.10.1:​2000-198.18.10.100 -w 4
 +</​code>​
 +
 +And the same with IPv6 flows:
 +<​code>​
 +pkt-gen -N -f tx -i igb2 -n 1000000000 -l 60 -6 -d "​[2001:​2:​0:​8001::​1]-[2001:​2:​0:​8001::​64]"​ -D 00:​0d:​b9:​41:​ca:​3d -s "​[2001:​2:​0:​1::​1]-[2001:​2:​0:​1::​14]"​ -S 00:​07:​43:​2e:​e4:​72 -w 4
 +</​code>​
 +
 +
 +Receiver will use these commands:
 +<​code>​
 +pkt-gen -i igb3 -f rx -w 4
 +</​code>​
 +===== Basic configuration =====
 +
 +==== Disabling Ethernet flow-control ===
 +
 +<​code>​
 +echo "​dev.igb.1.fc=0"​ >> /​etc/​sysctl.conf
 +echo "​dev.igb.2.fc=0"​ >> /​etc/​sysctl.conf
 +service sysctl restart
 +</​code>​
 +
 +====Enabling Tx abdicate (iflib drivers) ===
 +
 +Second, if NIC drivers is iflib based you should enable TX abdicate (automatically done with BSDRP but not on a generic FreeBSD):
 +<​code>​
 +cat <<EOF >> /​etc/​sysctl.conf
 +# Enabling Tx abdicate
 +dev.igb.1.iflib.tx_abdicate=1
 +dev.igb.2.iflib.tx_abdicate=1
 +EOF
 +service sysctl restart
 +</​code>​
 +
 +==== Disabling ICMP redirect ===
 +
 +ICMP redirect is enabled by default on a generic FreeBSD (it's disabled by default on BSDRP), and this feature disable the fast tryforward code path.
 +
 +<​code>​
 +cat <<EOF >> /​etc/​sysctl.conf
 +# Enabling fastforwarding by disabling ICMP redirect
 +net.inet.ip.redirect=0
 +net.inet6.ip6.redirect=0
 +EOF
 +service sysctl restart
 +</​code>​
 +
 +
 +
 +==== Static routes and ARP/NDP entries ==== 
 +
 +Configure static routes, configure IP addresses, static ARP and some entropy sources are excluded:
 +
 +A router [[Documentation:​Technical docs:​Performance|should not use LRO and TSO]]. BSDRP disable by default using a RC script (disablelrotso_enable="​YES"​ in /​etc/​rc.conf.misc).
 +
 +<​code>​
 +sysrc gateway_enable="​YES"​
 +sysrc static_routes="​generator receiver"​
 +sysrc route_generator="​-net 198.18.0.0/​16 198.18.0.203"​
 +sysrc route_receiver="​-net 198.19.0.0/​16 198.19.0.203"​
 +sysrc static_arp_pairs="​receiver generator"​
 +sysrc static_arp_generator="​198.18.0.203 00:​1b:​21:​c4:​95:​7a"​
 +sysrc static_arp_receiver="​198.19.0.203 00:​1b:​21:​c4:​95:​7b"​
 +sysrc ifconfig_igb1="​inet 198.18.0.205/​24 -tso4 -tso6 -lro -vlanhwtso"​
 +sysrc ifconfig_igb2="​inet 198.19.0.205/​24 -tso4 -tso6 -lro -vlanhwtso"​
 +
 +sysrc ipv6_gateway_enable="​YES"​
 +sysrc ipv6_activate_all_interfaces="​YES"​
 +sysrc ipv6_static_routes="​generator receiver"​
 +sysrc ipv6_route_generator="​2001:​2::​ -prefixlen 49 2001:​2::​203"​
 +sysrc ipv6_route_receiver="​2001:​2:​0:​8000::​ -prefixlen 49 2001:​2:​0:​8000::​203"​
 +sysrc ifconfig_igb1_ipv6="​inet6 2001:2::205 prefixlen 64"
 +sysrc ifconfig_igb2_ipv6="​inet6 2001:​2:​0:​8000::​205 prefixlen 64"
 +sysrc static_ndp_pairs="​receiver generator"​
 +sysrc static_ndp_generator="​2001:​2::​203 00:​1b:​21:​c4:​95:​7a"​
 +sysrc static_ndp_receiver="​2001:​2:​0:​8000::​203 00:​1b:​21:​c4:​95:​7b"​
 +
 +# Reduce number of source for entropy device harvests
 +harvest_mask="​351"​
 +</​code>​
 +
 +===== Default forwarding rate =====
 +
 +We start the first test by starting one packet generator at gigabit line-rate (1.488Mpps) and found:
 +  * APU2 is not responsive during this test: NIC multiqueue correcly distribute load to all cores;
 +  * About 794Kpps are accepted by the igb(4) Ethernet interface.
 +
 +<​code>​
 +110.375986 main_thread [2514] 794985 pps (795780 pkts 381974400 bps in 1001000 usec) 7.63 avg_batch 1001 min_space
 +111.377986 main_thread [2514] 794692 pps (796281 pkts 382214880 bps in 1002000 usec) 7.58 avg_batch 1001 min_space
 +112.378985 main_thread [2514] 794606 pps (795400 pkts 381792000 bps in 1000999 usec) 7.07 avg_batch 1005 min_space
 +113.379987 main_thread [2514] 796200 pps (796998 pkts 382559040 bps in 1001002 usec) 6.78 avg_batch 999 min_space
 +114.380986 main_thread [2514] 800465 pps (801265 pkts 384607200 bps in 1000999 usec) 6.88 avg_batch 1001 min_space
 +</​code>​
 +===== Firewalls impact =====
 +
 +This test will generate 2000 different flows by using 2000 different UDP destination ports.
 +
 +pf and ipfw configurations used are detailed on the previous [[documentation:​examples:​forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580#​firewall_impact|Forwarding performance lab of an IBM System x3550 M3 with Intel 82580]].
 +
 +{{bench.forwarding.and.firewalling.rate.on.pc.engines.apu2.png|forwarding and firewalling rate with a PC Engines APU running FreeBSD 10.3}}
 +
 +
 +
 +
  
documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2.txt · Last modified: 2019/03/25 12:37 (external edit)