User Tools

Site Tools


documentation:examples:forwarding_performance_lab_of_a_pc_engines_apu2

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

documentation:examples:forwarding_performance_lab_of_a_pc_engines_apu2 [2019/03/25 12:37] (current)
Line 1: Line 1:
 +====== Forwarding performance lab of a PC Engines APU 2 ======
 +{{description>Forwarding performance lab of a dual core  AMD GX-412TC (1 GHz) with 3 with Intel i210AT Gigabit}}
 +===== Hardware detail =====
 +
 +This lab will test a [[http://www.pcengines.ch/apu2.htm|PC Engines APU 2C4]] ([[PC Engines APU2|dmesg]]):
 +   * Quad core [[https://www.amd.com/Documents/AMDGSeriesSOCProductBrief.pdf|AMD GX-412TC Processor]] (1 GHz and AESNI)
 +   * 3 with Intel i210AT Gigabit
 +   * 4Gb of RAM
 +
 +===== Lab set-up =====
 +
 +For more information about full setup of this lab: [[documentation:examples:Setting up a forwarding performance benchmark lab]] (switch configuration, etc.).
 +
 +BSDRP release 1.92 based on FreeBSD 12-STABLE is used.
 +==== Diagram ====
 +
 +<code>
 + +------------------------------------------+      +-----------------------+
 +             Device under Test            |      |  Packet gen & receiv  |
 +                                          |      |                       |
 +                igb1:    198.18.0.205/24  |<=====| igb2: 198.18.0.203/24 |
 +                            2001:2::8/64  |      |        2001:2::203/64 |
 +                       00:0d:b9:41:ca:3d  |      |     00:1b:21:c4:95:7a |
 +                                          |      |                       |
 +                igb2:    198.19.0.205/24  |=====>| igb3: 198.19.0.203/24 |
 +                     2001:2:0:8000::8/64  |      | 2001:2:0:8000::203/64 |
 +                       00:0d:b9:41:ca:3e  |      |     00:1b:21:c4:95:7b |
 +                                          |      |                       |
 +               static routes              |      |                       |
 +      198.19.0.0/16 => 198.19.0.203            |                       |
 +      198.18.0.0/16 => 198.18.0.203            |                       |
 +        2001:2::/49 => 2001:2::203        |      |                       |
 + | 2001:2:0:8000::/49 => 2001:2:0:8000::203 |      |                       |
 +                                          |      |                       |
 +            static arp and ndp            |      |                       |
 + | 198.18.0.203        => 00:1b:21:c4:95:7a |      |                       |
 + | 2001:2::203                              |      |                       |
 +                                          |      |                       |
 + | 198.19.0.203        => 00:1b:21:c4:95:7b |      |                       |
 + | 2001:2:0:8000::203                            |                       |
 + +------------------------------------------+      +-----------------------+
 +
 +</code>
 +
 +The generator **MUST** generate lot's of IP flows (multiple source/destination IP addresses and/or UDP src/dst port) and minimum packet size (for generating maximum packet rate) with one of these commands:
 +
 +Multiple source/destination IP addresses (don't forget to precise port to use for avoiding to use port number 0 filtered by pf):
 +<code>
 +pkt-gen -U -i igb2 -f tx -n 80000000 -l 60 -d 198.19.10.1:2000-198.19.10.20 -D 00:0d:b9:41:ca:3d -s 198.18.10.1:2000-198.18.10.100 -w 4
 +</code>
 +
 +And the same with IPv6 flows:
 +<code>
 +pkt-gen -N -f tx -i igb2 -n 1000000000 -l 60 -6 -d "[2001:2:0:8001::1]-[2001:2:0:8001::64]" -D 00:0d:b9:41:ca:3d -s "[2001:2:0:1::1]-[2001:2:0:1::14]" -S 00:07:43:2e:e4:72 -w 4
 +</code>
 +
 +
 +Receiver will use these commands:
 +<code>
 +pkt-gen -i igb3 -f rx -w 4
 +</code>
 +===== Basic configuration =====
 +
 +==== Disabling Ethernet flow-control ===
 +
 +<code>
 +echo "dev.igb.1.fc=0" >> /etc/sysctl.conf
 +echo "dev.igb.2.fc=0" >> /etc/sysctl.conf
 +service sysctl restart
 +</code>
 +
 +====Enabling Tx abdicate (iflib drivers) ===
 +
 +Second, if NIC drivers is iflib based you should enable TX abdicate (automatically done with BSDRP but not on a generic FreeBSD):
 +<code>
 +cat <<EOF >> /etc/sysctl.conf
 +# Enabling Tx abdicate
 +dev.igb.1.iflib.tx_abdicate=1
 +dev.igb.2.iflib.tx_abdicate=1
 +EOF
 +service sysctl restart
 +</code>
 +
 +==== Disabling ICMP redirect ===
 +
 +ICMP redirect is enabled by default on a generic FreeBSD (it's disabled by default on BSDRP), and this feature disable the fast tryforward code path.
 +
 +<code>
 +cat <<EOF >> /etc/sysctl.conf
 +# Enabling fastforwarding by disabling ICMP redirect
 +net.inet.ip.redirect=0
 +net.inet6.ip6.redirect=0
 +EOF
 +service sysctl restart
 +</code>
 +
 +
 +
 +==== Static routes and ARP/NDP entries ==== 
 +
 +Configure static routes, configure IP addresses, static ARP and some entropy sources are excluded:
 +
 +A router [[Documentation:Technical docs:Performance|should not use LRO and TSO]]. BSDRP disable by default using a RC script (disablelrotso_enable="YES" in /etc/rc.conf.misc).
 +
 +<code>
 +sysrc gateway_enable="YES"
 +sysrc static_routes="generator receiver"
 +sysrc route_generator="-net 198.18.0.0/16 198.18.0.203"
 +sysrc route_receiver="-net 198.19.0.0/16 198.19.0.203"
 +sysrc static_arp_pairs="receiver generator"
 +sysrc static_arp_generator="198.18.0.203 00:1b:21:c4:95:7a"
 +sysrc static_arp_receiver="198.19.0.203 00:1b:21:c4:95:7b"
 +sysrc ifconfig_igb1="inet 198.18.0.205/24 -tso4 -tso6 -lro -vlanhwtso"
 +sysrc ifconfig_igb2="inet 198.19.0.205/24 -tso4 -tso6 -lro -vlanhwtso"
 +
 +sysrc ipv6_gateway_enable="YES"
 +sysrc ipv6_activate_all_interfaces="YES"
 +sysrc ipv6_static_routes="generator receiver"
 +sysrc ipv6_route_generator="2001:2:: -prefixlen 49 2001:2::203"
 +sysrc ipv6_route_receiver="2001:2:0:8000:: -prefixlen 49 2001:2:0:8000::203"
 +sysrc ifconfig_igb1_ipv6="inet6 2001:2::205 prefixlen 64"
 +sysrc ifconfig_igb2_ipv6="inet6 2001:2:0:8000::205 prefixlen 64"
 +sysrc static_ndp_pairs="receiver generator"
 +sysrc static_ndp_generator="2001:2::203 00:1b:21:c4:95:7a"
 +sysrc static_ndp_receiver="2001:2:0:8000::203 00:1b:21:c4:95:7b"
 +
 +# Reduce number of source for entropy device harvests
 +harvest_mask="351"
 +</code>
 +
 +===== Default forwarding rate =====
 +
 +We start the first test by starting one packet generator at gigabit line-rate (1.488Mpps) and found:
 +  * APU2 is not responsive during this test: NIC multiqueue correcly distribute load to all cores;
 +  * About 794Kpps are accepted by the igb(4) Ethernet interface.
 +
 +<code>
 +110.375986 main_thread [2514] 794985 pps (795780 pkts 381974400 bps in 1001000 usec) 7.63 avg_batch 1001 min_space
 +111.377986 main_thread [2514] 794692 pps (796281 pkts 382214880 bps in 1002000 usec) 7.58 avg_batch 1001 min_space
 +112.378985 main_thread [2514] 794606 pps (795400 pkts 381792000 bps in 1000999 usec) 7.07 avg_batch 1005 min_space
 +113.379987 main_thread [2514] 796200 pps (796998 pkts 382559040 bps in 1001002 usec) 6.78 avg_batch 999 min_space
 +114.380986 main_thread [2514] 800465 pps (801265 pkts 384607200 bps in 1000999 usec) 6.88 avg_batch 1001 min_space
 +</code>
 +===== Firewalls impact =====
 +
 +This test will generate 2000 different flows by using 2000 different UDP destination ports.
 +
 +pf and ipfw configurations used are detailed on the previous [[documentation:examples:forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580#firewall_impact|Forwarding performance lab of an IBM System x3550 M3 with Intel 82580]].
 +
 +{{bench.forwarding.and.firewalling.rate.on.pc.engines.apu2.png|forwarding and firewalling rate with a PC Engines APU running FreeBSD 10.3}}
 +
 +
 +
 +
  
documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2.txt · Last modified: 2019/03/25 12:37 (external edit)