Forwarding performance lab of a 8 cores Atom C2758 at 2.41GHz with 10-Gigabit Chelsio T540-CR

This lab will test a SuperMicro SuperServer 5018A-FTN4:

• Intel Rangeley: Atom C2758 (8 cores) at 2.4GHz
• 8Gb of RAM
• Quad port Chelsio 10-Gigabit T540-CR and OPT SFP (SFP-10G-LR)

For more information about full setup of this lab: Setting up a forwarding performance benchmark lab (switch configuration, etc.).

BSDRP used for this bench is based on FreeBSD 11.1.

+------------------------------------------+ +-------+ +------------------------------+
|        Device under test                 | |Juniper| | Packet generator & receiver  |
|                                          | |  QFX  | |                              |
|                cxl0: 198.18.0.8/24       |=|   <   |=| vcxl0: 198.18.0.110/24       |
|                      2001:2::8/64        | |       | |        2001:2::110/64        |
|                      (00:07:43:2e:e5:90) | |       | |        (00:07:43:2e:e4:72)   |
|                                          | |       | |                              |
|                cxl1: 198.19.0.8/24       |=|   >   |=| vcxl1: 198.19.0.110/24       |
|                      2001:2:0:8000::8/64 | |       | |        2001:2:0:8000::110/64 |
|                      (00:07:43:2e:e5:98) | +-------+ |        (00:07:43:2e:e4:7a)   |
|                                          |           |                              |
|            static routes                 |           |                              |
| 192.18.0.0/16      => 198.18.0.10        |           |                              |
| 192.19.0.0/16      => 198.19.0.10        |           |                              |
| 2001:2::/49        => 2001:2::110        |           |                              |
| 2001:2:0:8000::/49 => 2001:2:0:8000::110 |           |                              |
|                                          |           |                              |
|        static arp and ndp                |           | /boot/loader.conf:           |
| 198.18.0.10        => 00:07:43:2e:e4:72  |           |      hw.cxgbe.num_vis=2      |
| 2001:2::110                              |           |                              |
|                                          |           |                              |
| 198.19.0.10        => 00:07:43:2e:e4:7a  |           |                              |
| 2001:2:0:8000::110                       |           |                              |
+------------------------------------------+           +------------------------------+

The generator MUST generate lot's of smallest IP flows (multiple source/destination IP addresses and/or UDP src/dst port).

Here is an example for generating 2000 IPv4 flows (100 destination IP addresses * 20 source IP addresses) with a Chelsio NIC:

pkt-gen -i vcxl0 -f tx -n 1000000000 -l 60 -d 198.19.10.1:2000-198.19.10.100 -D 00:07:43:2e:e5:90 -s 198.18.10.1:2000-198.18.10.20 -w 4 -p 2

And the same with IPv6 flows (minimum frame size of 62 here):

pkt-gen -f tx -i vcxl0 -n 1000000000 -l 62 -6 -d "[2001:2:0:8010::1]-[2001:2:0:8010::64]" -D 00:07:43:2e:e5:90 -s "[2001:2:0:10::1]-[2001:2:0:10::14]" -S 00:07:43:2e:e4:72 -w 4 -p 2

Receiver will use this command:

pkt-gen -i vcxl1 -f rx -w 4

First, disable Ethernet flow-control on both servers. Chelsio T540 are configured like this:

echo "dev.cxl.0.pause_settings=0" >> /etc/sysctl.conf
echo "dev.cxl.0.pause_settings=0" >> /etc/sysctl.conf
service sysctl reload

A router should not use LRO and TSO. BSDRP disable by default using a RC script (disablelrotso_enable=“YES” in /etc/rc.conf.misc).

Configure static routes:

sysrc static_routes="generator receiver"
sysrc route_generator="-net 198.18.0.0/16 198.18.8.2"
sysrc route_receiver="-net 198.19.0.0/16 198.19.8.2"
sysrc ifconfig_cxl0="inet 198.18.0.8/24 -tso4 -tso6 -lro"
sysrc ifconfig_cxl1="inet 198.19.0.8/24 -tso4 -tso6 -lro"
sysrc static_arp_pairs="HPncxl0 HPncxl1"
sysrc static_arp_HPncxl0="198.18.0.110 00:07:43:2e:e4:72"
sysrc static_arp_HPncxl1="198.19.0.110 00:07:43:2e:e4:7a"

Behaviour in front of a multi-flow traffic generator at line-rate 14.8Mpps (thanks Chelsio!), pkt-gen on the receiver reports:

705.046821 main_thread [2277] 3173758 pps (3176351 pkts 1524648480 bps in 1000817 usec) 19.66 avg_batch 0 min_space
706.048593 main_thread [2277] 3179960 pps (3185595 pkts 1529085600 bps in 1001772 usec) 19.65 avg_batch 823 min_space
707.050593 main_thread [2277] 3180102 pps (3186459 pkts 1529500320 bps in 1001999 usec) 19.63 avg_batch 1000 min_space
708.052592 main_thread [2277] 3182707 pps (3189072 pkts 1530754560 bps in 1002000 usec) 19.69 avg_batch 768 min_space
709.054593 main_thread [2277] 3166689 pps (3173022 pkts 1523050560 bps in 1002000 usec) 19.59 avg_batch 1001 min_space
710.055593 main_thread [2277] 3126576 pps (3129706 pkts 1502258880 bps in 1001001 usec) 19.33 avg_batch 999 min_space
711.057592 main_thread [2277] 3138706 pps (3144980 pkts 1509590400 bps in 1001999 usec) 19.41 avg_batch 768 min_space
712.058592 main_thread [2277] 3147761 pps (3150909 pkts 1512436320 bps in 1001000 usec) 19.47 avg_batch 768 min_space
713.059592 main_thread [2277] 3156042 pps (3159195 pkts 1516413600 bps in 1000999 usec) 19.51 avg_batch 768 min_space
714.060592 main_thread [2277] 3174469 pps (3177647 pkts 1525270560 bps in 1001001 usec) 19.58 avg_batch 768 min_space
715.061593 main_thread [2277] 3172119 pps (3175291 pkts 1524139680 bps in 1001000 usec) 19.60 avg_batch 1001 min_space

About 3.1M packets-per-seconds forwarded.