User Tools

Site Tools


documentation:examples:forwarding_performance_lab_of_a_superserver_5018a-ftn4_with_10-gigabit_chelsio_t540-cr

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

documentation:examples:forwarding_performance_lab_of_a_superserver_5018a-ftn4_with_10-gigabit_chelsio_t540-cr [2017/09/04 09:03] (current)
Line 1: Line 1:
 +====== Forwarding performance lab of a SuperServer 5018A-FTN4 with 10-Gigabit Chelsio T540-CR ======
 +{{description>​Forwarding performance lab of a 8 cores Atom C2758 at 2.41GHz with 10-Gigabit Chelsio T540-CR}}
 +===== Hardware detail =====
 +
 +This lab will test a [[http://​www.supermicro.com/​products/​system/​1U/​5018/​SYS-5018A-FTN4.cfm|SuperMicro]] [[SuperServer 5018A-FTN4]]:​
 +   * Intel Rangeley:​ [[http://​ark.intel.com/​products/​77988/​Intel-Atom-Processor-C2758-4M-Cache-2_40-GHz|Atom C2758 (8 cores) at 2.4GHz]]
 +   * 8Gb of RAM
 +   * Quad port Chelsio 10-Gigabit T540-CR and OPT SFP (SFP-10G-LR)
 +
 +===== Lab set-up =====
 +
 +For more information about full setup of this lab: [[documentation:​examples:​Setting up a forwarding performance benchmark lab]] (switch configuration,​ etc.).
 +
 +BSDRP used for this bench is based on FreeBSD 11.1.
 +
 +==== Diagram ====
 +
 +<​code>​
 ++------------------------------------------+ +-------+ +------------------------------+
 +|        Device under test                 | |Juniper| | Packet generator & receiver ​ |
 +|                                          | |  QFX  | |                              |
 +|                cxl0: 198.18.0.8/​24 ​      ​|=| ​  < ​  |=| vcxl0: 198.18.0.110/​24 ​      |
 +|                      2001:​2::​8/​64 ​       | |       | |        2001:​2::​110/​64 ​       |
 +|                      (00:​07:​43:​2e:​e5:​90) | |       | |        (00:​07:​43:​2e:​e4:​72) ​  |
 +|                                          | |       | |                              |
 +|                cxl1: 198.19.0.8/​24 ​      ​|=| ​  > ​  |=| vcxl1: 198.19.0.110/​24 ​      |
 +|                      2001:​2:​0:​8000::​8/​64 | |       | |        2001:​2:​0:​8000::​110/​64 |
 +|                      (00:​07:​43:​2e:​e5:​98) | +-------+ |        (00:​07:​43:​2e:​e4:​7a) ​  |
 +|                                          |           ​| ​                             |
 +|            static routes ​                ​| ​          ​| ​                             |
 +| 192.18.0.0/​16 ​     => 198.18.0.10 ​       |           ​| ​                             |
 +| 192.19.0.0/​16 ​     => 198.19.0.10 ​       |           ​| ​                             |
 +| 2001:​2::/​49 ​       => 2001:​2::​110 ​       |           ​| ​                             |
 +| 2001:​2:​0:​8000::/​49 => 2001:​2:​0:​8000::​110 |           ​| ​                             |
 +|                                          |           ​| ​                             |
 +|        static arp and ndp                |           | /​boot/​loader.conf: ​          |
 +| 198.18.0.10 ​       => 00:​07:​43:​2e:​e4:​72 ​ |           ​| ​     hw.cxgbe.num_vis=2 ​     |
 +| 2001:​2::​110 ​                             |           ​| ​                             |
 +|                                          |           ​| ​                             |
 +| 198.19.0.10 ​       => 00:​07:​43:​2e:​e4:​7a ​ |           ​| ​                             |
 +| 2001:​2:​0:​8000::​110 ​                      ​| ​          ​| ​                             |
 ++------------------------------------------+ ​          ​+------------------------------+
 +</​code>​
 +
 +The generator **MUST** generate lot's of smallest IP flows (multiple source/​destination IP addresses and/or UDP src/dst port).
 +
 +Here is an example for generating 2000 IPv4 flows (100 destination IP addresses * 20 source IP addresses) with a Chelsio NIC:
 +<​code>​
 +pkt-gen -i vcxl0 -f tx -n 1000000000 -l 60 -d 198.19.10.1:​2000-198.19.10.100 -D 00:​07:​43:​2e:​e5:​90 -s 198.18.10.1:​2000-198.18.10.20 -w 4 -p 2
 +</​code>​
 +
 +And the same with IPv6 flows (minimum frame size of 62 here):
 +<​code>​
 +pkt-gen -f tx -i vcxl0 -n 1000000000 -l 62 -6 -d "​[2001:​2:​0:​8010::​1]-[2001:​2:​0:​8010::​64]"​ -D 00:​07:​43:​2e:​e5:​90 -s "​[2001:​2:​0:​10::​1]-[2001:​2:​0:​10::​14]"​ -S 00:​07:​43:​2e:​e4:​72 -w 4 -p 2
 +</​code>​
 +
 +<note warning>
 +This version of pkt-gen is improved with: IPv6 support, software checksum and optional unit normalization. [[https://​raw.githubusercontent.com/​ocochard/​BSDRP/​master/​BSDRPcur/​patches/​freebsd.pkt-gen.ae-ipv6.patch|BSDRP'​s patch to netmap pkt-gen ]].
 +</​note>​
 +Receiver will use this command:
 +<​code>​
 +pkt-gen -i vcxl1 -f rx -w 4
 +</​code>​
 +===== Basic configuration =====
 +
 +==== Disabling Ethernet flow-control ===
 +
 +First, disable Ethernet flow-control on both servers. Chelsio T540 are configured like this:
 +<​code>​
 +echo "​dev.cxl.0.pause_settings=0"​ >> /​etc/​sysctl.conf
 +echo "​dev.cxl.0.pause_settings=0"​ >> /​etc/​sysctl.conf
 +service sysctl reload
 +</​code>​
 +
 +=== Disabling LRO and TSO ===
 +
 +A router [[Documentation:​Technical docs:​Performance|should not use LRO and TSO]]. BSDRP disable by default using a RC script (disablelrotso_enable="​YES"​ in /​etc/​rc.conf.misc).
 +==== IP Configuration on DUT ==== 
 +
 +Configure static routes:
 +<​code>​
 +sysrc static_routes="​generator receiver"​
 +sysrc route_generator="​-net 198.18.0.0/​16 198.18.8.2"​
 +sysrc route_receiver="​-net 198.19.0.0/​16 198.19.8.2"​
 +sysrc ifconfig_cxl0="​inet 198.18.0.8/​24 -tso4 -tso6 -lro"
 +sysrc ifconfig_cxl1="​inet 198.19.0.8/​24 -tso4 -tso6 -lro"
 +sysrc static_arp_pairs="​HPncxl0 HPncxl1"​
 +sysrc static_arp_HPncxl0="​198.18.0.110 00:​07:​43:​2e:​e4:​72"​
 +sysrc static_arp_HPncxl1="​198.19.0.110 00:​07:​43:​2e:​e4:​7a"​
 +</​code>​
 +===== Default forwarding rate =====
 +
 +Behaviour in front of a multi-flow traffic generator at line-rate 14.8Mpps (thanks Chelsio!), pkt-gen on the receiver reports:
 +
 +<​code>​
 +705.046821 main_thread [2277] 3173758 pps (3176351 pkts 1524648480 bps in 1000817 usec) 19.66 avg_batch 0 min_space
 +706.048593 main_thread [2277] 3179960 pps (3185595 pkts 1529085600 bps in 1001772 usec) 19.65 avg_batch 823 min_space
 +707.050593 main_thread [2277] 3180102 pps (3186459 pkts 1529500320 bps in 1001999 usec) 19.63 avg_batch 1000 min_space
 +708.052592 main_thread [2277] 3182707 pps (3189072 pkts 1530754560 bps in 1002000 usec) 19.69 avg_batch 768 min_space
 +709.054593 main_thread [2277] 3166689 pps (3173022 pkts 1523050560 bps in 1002000 usec) 19.59 avg_batch 1001 min_space
 +710.055593 main_thread [2277] 3126576 pps (3129706 pkts 1502258880 bps in 1001001 usec) 19.33 avg_batch 999 min_space
 +711.057592 main_thread [2277] 3138706 pps (3144980 pkts 1509590400 bps in 1001999 usec) 19.41 avg_batch 768 min_space
 +712.058592 main_thread [2277] 3147761 pps (3150909 pkts 1512436320 bps in 1001000 usec) 19.47 avg_batch 768 min_space
 +713.059592 main_thread [2277] 3156042 pps (3159195 pkts 1516413600 bps in 1000999 usec) 19.51 avg_batch 768 min_space
 +714.060592 main_thread [2277] 3174469 pps (3177647 pkts 1525270560 bps in 1001001 usec) 19.58 avg_batch 768 min_space
 +715.061593 main_thread [2277] 3172119 pps (3175291 pkts 1524139680 bps in 1001000 usec) 19.60 avg_batch 1001 min_space
 +</​code>​
 +
 +About 3.1M packets-per-seconds forwarded.
 +
 +===== Firewall impact on forwarding rate =====
 +
 +{{:​documentation:​examples:​bench-lab-ss5018a-ftn4-chelsio-11-stable.png}}
  
documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4_with_10-gigabit_chelsio_t540-cr.txt · Last modified: 2017/09/04 09:03 (external edit)