User Tools

Site Tools


documentation:examples:forwarding_performance_lab_of_a_pc_engines_apu2

Forwarding performance lab of a PC Engines APU 2

Hardware detail

This lab will test a PC Engines APU 2C4 (dmesg):

Lab set-up

For more information about full setup of this lab: Setting up a forwarding performance benchmark lab (switch configuration, etc.).

BSDRP release based on FreeBSD 11.1 is used.

Diagram

 +------------------------------------------+      +-----------------------+
 |             Device under Test            |      |  Packet gen & receiv  |
 |                                          |      |                       |
 |                igb1:    198.18.0.205/24  |<=====| igb2: 198.18.0.203/24 |
 |                            2001:2::8/64  |      |        2001:2::203/64 |
 |                       00:0d:b9:41:ca:3d  |      |     00:1b:21:c4:95:7a |
 |                                          |      |                       |
 |                igb2:    198.19.0.205/24  |=====>| igb3: 198.19.0.203/24 |
 |                     2001:2:0:8000::8/64  |      | 2001:2:0:8000::203/64 |
 |                       00:0d:b9:41:ca:3e  |      |     00:1b:21:c4:95:7b |
 |                                          |      |                       |
 |               static routes              |      |                       |
 |      198.19.0.0/16 => 198.19.0.203       |      |                       |
 |      198.18.0.0/16 => 198.18.0.203       |      |                       |
 |        2001:2::/49 => 2001:2::203        |      |                       |
 | 2001:2:0:8000::/49 => 2001:2:0:8000::203 |      |                       |
 |                                          |      |                       |
 |            static arp and ndp            |      |                       |
 | 198.18.0.203        => 00:1b:21:c4:95:7a |      |                       |
 | 2001:2::203                              |      |                       |
 |                                          |      |                       |
 | 198.19.0.203        => 00:1b:21:c4:95:7b |      |                       |
 | 2001:2:0:8000::203                       |      |                       |
 +------------------------------------------+      +-----------------------+

The generator MUST generate lot's of IP flows (multiple source/destination IP addresses and/or UDP src/dst port) and minimum packet size (for generating maximum packet rate) with one of these commands:

Multiple source/destination IP addresses (don't forget to precise port to use for avoiding to use port number 0 filtered by pf):

pkt-gen -U -i igb2 -f tx -n 80000000 -l 60 -d 198.19.10.1:2000-198.19.10.20 -D 00:0d:b9:41:ca:3d -s 198.18.10.1:2000-198.18.10.100 -w 4

And the same with IPv6 flows:

pkt-gen -N -f tx -i igb2 -n 1000000000 -l 60 -6 -d "[2001:2:0:8001::1]-[2001:2:0:8001::64]" -D 00:0d:b9:41:ca:3d -s "[2001:2:0:1::1]-[2001:2:0:1::14]" -S 00:07:43:2e:e4:72 -w 4

Receiver will use these commands:

pkt-gen -i igb3 -f rx -w 4

Basic configuration

Disabling Ethernet flow-control

First, disable Ethernet flow-control:

echo "dev.igb.1.fc=0" >> /etc/sysctl.conf
echo "dev.igb.2.fc=0" >> /etc/sysctl.conf
sysctl dev.igb.1.fc=0
sysctl dev.igb.2.fc=0

Static routes and ARP/NDP entries

Configure static routes, configure IP addresses, static ARP and some entropy sources are excluded:

A router should not use LRO and TSO. BSDRP disable by default using a RC script (disablelrotso_enable=“YES” in /etc/rc.conf.misc).

sysrc gateway_enable="YES"
sysrc static_routes="generator receiver"
sysrc route_generator="-net 198.18.0.0/16 198.18.0.203"
sysrc route_receiver="-net 198.19.0.0/16 198.19.0.203"
sysrc static_arp_pairs="receiver generator"
sysrc static_arp_generator="198.18.0.203 00:1b:21:c4:95:7a"
sysrc static_arp_receiver="198.19.0.203 00:1b:21:c4:95:7b"
sysrc ifconfig_igb1="inet 198.18.0.205/24 -tso4 -tso6 -lro -vlanhwtso"
sysrc ifconfig_igb2="inet 198.19.0.205/24 -tso4 -tso6 -lro -vlanhwtso"

sysrc ipv6_gateway_enable="YES"
sysrc ipv6_activate_all_interfaces="YES"
sysrc ipv6_static_routes="generator receiver"
sysrc ipv6_route_generator="2001:2:: -prefixlen 49 2001:2::203"
sysrc ipv6_route_receiver="2001:2:0:8000:: -prefixlen 49 2001:2:0:8000::203"
sysrc ifconfig_igb1_ipv6="inet6 2001:2::205 prefixlen 64"
sysrc ifconfig_igb2_ipv6="inet6 2001:2:0:8000::205 prefixlen 64"
sysrc static_ndp_pairs="receiver generator"
sysrc static_ndp_generator="2001:2::203 00:1b:21:c4:95:7a"
sysrc static_ndp_receiver="2001:2:0:8000::203 00:1b:21:c4:95:7b"

# Reduce number of source for entropy device harvests
harvest_mask="351"

Default forwarding rate

We start the first test by starting one packet generator at gigabit line-rate (1.488Mpps) and found:

  • APU2 is not responsive during this test: NIC multiqueue correcly distribute load to all cores;
  • About 752Kpps are accepted by the igb(4) Ethernet interface.
635.441481 main_thread [2277] 758409 pps (759314 pkts 364470720 bps in 1001193 usec) 2.92 avg_batch 2040 min_space
636.442289 main_thread [2277] 757341 pps (757951 pkts 363816480 bps in 1000806 usec) 2.83 avg_batch 2039 min_space
637.444284 main_thread [2277] 757516 pps (759029 pkts 364333920 bps in 1001997 usec) 2.98 avg_batch 2040 min_space
638.446284 main_thread [2277] 754751 pps (756261 pkts 363005280 bps in 1002001 usec) 2.85 avg_batch 2040 min_space
639.448284 main_thread [2277] 757203 pps (758717 pkts 364184160 bps in 1002000 usec) 2.88 avg_batch 2039 min_space
640.449355 main_thread [2277] 757388 pps (758199 pkts 363935520 bps in 1001071 usec) 2.94 avg_batch 2039 min_space
641.451284 main_thread [2277] 757826 pps (759288 pkts 364458240 bps in 1001929 usec) 2.98 avg_batch 2040 min_space
642.453284 main_thread [2277] 754653 pps (756162 pkts 362957760 bps in 1002000 usec) 2.88 avg_batch 2039 min_space
643.454295 main_thread [2277] 757383 pps (758148 pkts 363911040 bps in 1001010 usec) 2.94 avg_batch 2040 min_space
644.455284 main_thread [2277] 755891 pps (756639 pkts 363186720 bps in 1000990 usec) 3.00 avg_batch 2039 min_space
645.456284 main_thread [2277] 757006 pps (757763 pkts 363726240 bps in 1001000 usec) 2.98 avg_batch 2039 min_space
646.458285 main_thread [2277] 755869 pps (757381 pkts 363542880 bps in 1002001 usec) 2.91 avg_batch 2039 min_space
647.459284 main_thread [2277] 758039 pps (758796 pkts 364222080 bps in 1000999 usec) 2.86 avg_batch 2040 min_space
648.461284 main_thread [2277] 755032 pps (756542 pkts 363140160 bps in 1002000 usec) 2.87 avg_batch 2039 min_space
649.462284 main_thread [2277] 758446 pps (759204 pkts 364417920 bps in 1001000 usec) 2.90 avg_batch 2039 min_space
650.464284 main_thread [2277] 755844 pps (757355 pkts 363530400 bps in 1001999 usec) 2.93 avg_batch 2040 min_space

Firewalls impact

This test will generate 2000 different flows by using 2000 different UDP destination ports.

pf and ipfw configurations used are detailed on the previous Forwarding performance lab of an IBM System x3550 M3 with Intel 82580.

forwarding and firewalling rate with a PC Engines APU running FreeBSD 10.3

documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2.txt · Last modified: 2017/09/03 10:21 by olivier