====== Forwarding performance lab of a PC Engines APU 2 ======
{{description>Forwarding performance lab of a dual core AMD GX-412TC (1 GHz) with 3 with Intel i210AT Gigabit}}
===== Hardware detail =====
This lab will test a [[http://www.pcengines.ch/apu2.htm|PC Engines APU 2C4]] ([[PC Engines APU2|dmesg]]):
* Quad core [[https://www.amd.com/Documents/AMDGSeriesSOCProductBrief.pdf|AMD GX-412TC Processor]] (1 GHz and AESNI)
* 3 with Intel i210AT Gigabit
* 4Gb of RAM
===== Lab set-up =====
For more information about full setup of this lab: [[documentation:examples:Setting up a forwarding performance benchmark lab]] (switch configuration, etc.).
BSDRP release 1.92 based on FreeBSD 12-STABLE is used.
==== Diagram ====
+------------------------------------------+ +-----------------------+
| Device under Test | | Packet gen & receiv |
| | | |
| igb1: 198.18.0.205/24 |<=====| igb2: 198.18.0.203/24 |
| 2001:2::8/64 | | 2001:2::203/64 |
| 00:0d:b9:41:ca:3d | | 00:1b:21:c4:95:7a |
| | | |
| igb2: 198.19.0.205/24 |=====>| igb3: 198.19.0.203/24 |
| 2001:2:0:8000::8/64 | | 2001:2:0:8000::203/64 |
| 00:0d:b9:41:ca:3e | | 00:1b:21:c4:95:7b |
| | | |
| static routes | | |
| 198.19.0.0/16 => 198.19.0.203 | | |
| 198.18.0.0/16 => 198.18.0.203 | | |
| 2001:2::/49 => 2001:2::203 | | |
| 2001:2:0:8000::/49 => 2001:2:0:8000::203 | | |
| | | |
| static arp and ndp | | |
| 198.18.0.203 => 00:1b:21:c4:95:7a | | |
| 2001:2::203 | | |
| | | |
| 198.19.0.203 => 00:1b:21:c4:95:7b | | |
| 2001:2:0:8000::203 | | |
+------------------------------------------+ +-----------------------+
The generator **MUST** generate lot's of IP flows (multiple source/destination IP addresses and/or UDP src/dst port) and minimum packet size (for generating maximum packet rate) with one of these commands:
Multiple source/destination IP addresses (don't forget to precise port to use for avoiding to use port number 0 filtered by pf):
pkt-gen -U -i igb2 -f tx -n 80000000 -l 60 -d 198.19.10.1:2000-198.19.10.20 -D 00:0d:b9:41:ca:3d -s 198.18.10.1:2000-198.18.10.100 -w 4
And the same with IPv6 flows:
pkt-gen -N -f tx -i igb2 -n 1000000000 -l 60 -6 -d "[2001:2:0:8001::1]-[2001:2:0:8001::64]" -D 00:0d:b9:41:ca:3d -s "[2001:2:0:1::1]-[2001:2:0:1::14]" -S 00:07:43:2e:e4:72 -w 4
Receiver will use these commands:
pkt-gen -i igb3 -f rx -w 4
===== Basic configuration =====
==== Disabling Ethernet flow-control ===
echo "dev.igb.1.fc=0" >> /etc/sysctl.conf
echo "dev.igb.2.fc=0" >> /etc/sysctl.conf
service sysctl restart
====Enabling Tx abdicate (iflib drivers) ===
Second, if NIC drivers is iflib based you should enable TX abdicate (automatically done with BSDRP but not on a generic FreeBSD):
cat <> /etc/sysctl.conf
# Enabling Tx abdicate
dev.igb.1.iflib.tx_abdicate=1
dev.igb.2.iflib.tx_abdicate=1
EOF
service sysctl restart
==== Disabling ICMP redirect ===
ICMP redirect is enabled by default on a generic FreeBSD (it's disabled by default on BSDRP), and this feature disable the fast tryforward code path.
cat <> /etc/sysctl.conf
# Enabling fastforwarding by disabling ICMP redirect
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0
EOF
service sysctl restart
==== Static routes and ARP/NDP entries ====
Configure static routes, configure IP addresses, static ARP and some entropy sources are excluded:
A router [[Documentation:Technical docs:Performance|should not use LRO and TSO]]. BSDRP disable by default using a RC script (disablelrotso_enable="YES" in /etc/rc.conf.misc).
sysrc gateway_enable="YES"
sysrc static_routes="generator receiver"
sysrc route_generator="-net 198.18.0.0/16 198.18.0.203"
sysrc route_receiver="-net 198.19.0.0/16 198.19.0.203"
sysrc static_arp_pairs="receiver generator"
sysrc static_arp_generator="198.18.0.203 00:1b:21:c4:95:7a"
sysrc static_arp_receiver="198.19.0.203 00:1b:21:c4:95:7b"
sysrc ifconfig_igb1="inet 198.18.0.205/24 -tso4 -tso6 -lro -vlanhwtso"
sysrc ifconfig_igb2="inet 198.19.0.205/24 -tso4 -tso6 -lro -vlanhwtso"
sysrc ipv6_gateway_enable="YES"
sysrc ipv6_activate_all_interfaces="YES"
sysrc ipv6_static_routes="generator receiver"
sysrc ipv6_route_generator="2001:2:: -prefixlen 49 2001:2::203"
sysrc ipv6_route_receiver="2001:2:0:8000:: -prefixlen 49 2001:2:0:8000::203"
sysrc ifconfig_igb1_ipv6="inet6 2001:2::205 prefixlen 64"
sysrc ifconfig_igb2_ipv6="inet6 2001:2:0:8000::205 prefixlen 64"
sysrc static_ndp_pairs="receiver generator"
sysrc static_ndp_generator="2001:2::203 00:1b:21:c4:95:7a"
sysrc static_ndp_receiver="2001:2:0:8000::203 00:1b:21:c4:95:7b"
# Reduce number of source for entropy device harvests
harvest_mask="351"
===== Default forwarding rate =====
We start the first test by starting one packet generator at gigabit line-rate (1.488Mpps) and found:
* APU2 is not responsive during this test: NIC multiqueue correcly distribute load to all cores;
* About 794Kpps are accepted by the igb(4) Ethernet interface.
110.375986 main_thread [2514] 794985 pps (795780 pkts 381974400 bps in 1001000 usec) 7.63 avg_batch 1001 min_space
111.377986 main_thread [2514] 794692 pps (796281 pkts 382214880 bps in 1002000 usec) 7.58 avg_batch 1001 min_space
112.378985 main_thread [2514] 794606 pps (795400 pkts 381792000 bps in 1000999 usec) 7.07 avg_batch 1005 min_space
113.379987 main_thread [2514] 796200 pps (796998 pkts 382559040 bps in 1001002 usec) 6.78 avg_batch 999 min_space
114.380986 main_thread [2514] 800465 pps (801265 pkts 384607200 bps in 1000999 usec) 6.88 avg_batch 1001 min_space
===== Firewalls impact =====
This test will generate 2000 different flows by using 2000 different UDP destination ports.
pf and ipfw configurations used are detailed on the previous [[documentation:examples:forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580#firewall_impact|Forwarding performance lab of an IBM System x3550 M3 with Intel 82580]].
{{bench.forwarding.and.firewalling.rate.on.pc.engines.apu2.png|forwarding and firewalling rate with a PC Engines APU running FreeBSD 10.3}}