IPSec performance lab of a 8 cores Atom

This lab will test a SuperMicro SuperServer 5018A-FTN4:

• Intel Rangeley: Atom C2758 (8 cores) at 2.4GHz
• 8Gb of RAM
• Quad port Chelsio 10-Gigabit T540-CR and OPT SFP (SFP-10G-LR)

This CPU includes AES-NI: AES-CBC,AES-XTS,AES-GCM,AES-ICM.

The benchmarking method used here is detailed in Setting up a VPN (IPSec, GRE, etc...) performance benchmark lab.

+--------------------+   +-------------------------------------+   +------------------------------------+
|         r630       |   |          Atom C2758-Chelsio         |   |                  HP                |
|  Packet generator  |   |           Device under Test         |   |           IPSec endpoint           |
|     and receiver   |   |                                     |   |              (AES-NI)              |
|                    |   |                                     |   |                                    |
|vcxl0: 198.18.0.2/24|=>=| cxl0: 198.18.0.208/24               |   |                                    |
|       2001:2::2/64 |   | 2001:2::208/64                      |   |                                    |
|  00:07:43:2f:fe:b2 |   | 00:07:43:2e:e5:90                   |   |                                    |
|                    |   |                                     |   |                                    |
|                    |   |               cxl1: 198.18.1.208/24 |=>=| cxl0: 198.18.1.210/24              |
|                    |   |                  2001:2:0:1::208/64 |   |    2001:2:0:1::210/64              |
|                    |   |                   00:07:43:2e:e5:98 |   |     00:07:43:2e:e4:70              |
|                    |   |                                     |   |                                    |
|                    |   |              static routes          |   |            static routes           |
|                    |   |     198.19.0.0/16 => 198.18.1.210   |   |    198.19.0.0/16 => 198.19.0.2     |
|                    |   |     198.18.0.0/16 => 198.18.0.2     |   |    198.18.0.0/16 => 198.18.1.208   |
|                    |   |       2001:2::/49 => 2001:2::2      |   |      2001:2::/49 => 2001:2:0:1::208|
|                    |   |2001:2:0:8000::/49 => 2001:2:0:1::210|   |2001:2:0:8000::/49=>2001:2:0:8000::2|
|                    |   |                                     |   |                                    |
|vcxl1: 198.19.0.2/24|   |                                     |   |        cxl1: 198.19.0.210/24       |
| 2001:2:0:8000::2/64|   |                                     |   |        2001:2:0:8000::210/64       |
| 00:07:43:2f:fe:ba  |   |                                     |   |         00:07:43:2e:e4:78          |
+--------------------+   +-------------------------------------+   +------------------------------------+
          ||                                                                          ||
          ==================================<===========================================

Almost the same as on the forwarding performance lab.

Configure IP address, routes and static IPSec.

/etc/rc.conf:

# IPv4 router
gateway_enable="YES"
static_routes="generator receiver"
route_generator="-net 198.18.0.0/16 198.18.0.2"
route_receiver="-net 198.19.0.0/16 198.18.1.210"
ifconfig_cxl0="inet 198.18.0.208/24 -tso4 -tso6 -lro"
ifconfig_cxl1="inet 198.18.1.208/24 -tso4 -tso6 -lro"
static_arp_pairs="generator receiver"
static_arp_generator="198.18.0.2 00:07:43:2f:fe:b2"
static_arp_receiver="198.18.1.210 00:07:43:2e:e4:70"

# IPv6 router
ipv6_gateway_enable="YES"
ipv6_activate_all_interfaces="YES"
ipv6_static_routes="generator receiver"
ipv6_route_generator="2001:2:: -prefixlen 49 2001:2::2"
ipv6_route_receiver="2001:2:0:8000:: -prefixlen 49 2001:2:0:1::210"
ifconfig_cxl0_ipv6="inet6 2001:2::208 prefixlen 64"
ifconfig_cxl1_ipv6="inet6 2001:2:0:1::208 prefixlen 64"
static_ndp_pairs="generator receiver"
static_ndp_generator="2001:2::2 00:07:43:2f:fe:b2"
static_ndp_receiver="2001:2:0:1::210 00:07:43:2e:e4:70"

# Enabling IPSec
kld_list="aesni"
ipsec_enable="YES"

/etc/ipsec.conf

flush;
spdflush;
spdadd 198.18.0.0/16 198.19.0.0/16 any -P out ipsec esp/tunnel/198.18.1.208-198.18.1.210/require;
spdadd 198.19.0.0/16 198.18.0.0/16 any -P in ipsec esp/tunnel/198.18.1.210-198.18.1.208/require;
add 198.18.1.208 198.18.1.210 esp 0x1000 -E aes-gcm-16 "12345678901234567890";
add 198.18.1.210 198.18.1.208 esp 0x1001 -E aes-gcm-16 "12345678901234567890";
spdadd 2001:2::/49 2001:2:0:8000::/49 any -P out ipsec esp/tunnel/2001:2:0:1::208-2001:2:0:1::210/require;
spdadd 2001:2:0:8000::/49 2001:2::/49 any -P in ipsec esp/tunnel/2001:2:0:1::210-2001:2:0:1::208/require;
add 2001:2:0:1::208 2001:2:0:1::210 esp 0x1002 -E aes-gcm-16 "12345678901234567890";
add 2001:2:0:1::210 2001:2:0:1::208 esp 0x1003 -E aes-gcm-16 "12345678901234567890";

Configure IP address, routes and static IPSec:

# IPv4 router
gateway_enable="YES"
ifconfig_cxl0="inet 198.18.1.210/24 -tso4 -tso6 -lro -vlanhwtso"
ifconfig_cxl1="inet 198.19.0.210/24 -tso4 -tso6 -lro -vlanhwtso"
static_routes="generator receiver"
route_generator="-net 198.18.0.0/16 198.18.1.208"
route_receiver="-net 198.19.0.0/16 198.19.0.2"
static_arp_pairs="generator receiver"
static_arp_generator="198.18.1.208 00:07:43:2e:e5:98"
static_arp_receiver="198.19.0.2 00:07:43:2f:fe:ba"

# IPv6 router
ipv6_gateway_enable="YES"
ipv6_activate_all_interfaces="YES"
ifconfig_cxl0_ipv6="inet6 2001:2:0:1::210 prefixlen 64"
ifconfig_cxl1_ipv6="inet6 2001:2:0:8000::210 prefixlen 64"
ipv6_static_routes="generator receiver"
ipv6_route_generator="2001:2:: -prefixlen 49 2001:1::208"
ipv6_route_receiver="2001:2:0:8000:: -prefixlen 49 2001:2:0:8000::2"
static_ndp_pairs="generator receiver"
static_ndp_generator="2001:2:0:1::208 00:07:43:2e:e5:98"
static_ndp_receiver="2001:2:0:8000::2 00:07:43:2f:fe:ba"

# Enabling IPSec
kld_list="aesni"
ipsec_enable="YES"

/etc/ipsec.conf:

flush;
spdflush;
spdadd 198.18.0.0/16 198.19.0.0/16 any -P in ipsec esp/tunnel/198.18.1.208-198.18.1.210/require;
spdadd 198.19.0.0/16 198.18.0.0/16 any -P out ipsec esp/tunnel/198.18.1.210-198.18.1.208/require;
add 198.18.1.208 198.18.1.210 esp 0x1000 -E aes-gcm-16 "12345678901234567890";
add 198.18.1.210 198.18.1.208 esp 0x1001 -E aes-gcm-16 "12345678901234567890";
spdadd 2001:2::/49 2001:2:0:8000::/49 any -P in ipsec esp/tunnel/2001:2:0:1::208-2001:2:0:1::210/require;
spdadd 2001:2:0:8000::/49 2001:2::/49 any -P out ipsec esp/tunnel/2001:2:0:1::210-2001:2:0:1::208/require;
add 2001:2:0:1::208 2001:2:0:1::210 esp 0x1002 -E aes-gcm-16 "12345678901234567890";
add 2001:2:0:1::210 2001:2:0:1::208 esp 0x1003 -E aes-gcm-16 "12345678901234567890";

Once done, we start using a fast method for measuring the “IPsec equilibrium throughput” of the DUT.

From the packet generator/receiver a simple script that use netmap-pktgen will do the job:

[root@pkt-gen]~# equilibrium -4 -d 00:07:43:2e:e5:90 -t vcxl0 -r vcxl1 -l 10000
Benchmark tool using equilibrium throughput method
- Benchmark mode: Bandwitdh (bps) for VPN gateway
- UDP load = 500B, IPv4 packet size=528B, Ethernet frame size=542B
- Link rate = 10000 Mb/s
- Tolerance = 0.01
Iteration 1
  - Offering load = 5000 Mb/s
  - Step = 2500 Mb/s
  - Measured forwarding rate = 1383 Mb/s
  - Forwared rate too low, forcing OLOAD=FWRATE and STEP=FWRATE/2
Iteration 2
  - Offering load = 1383 Mb/s
  - Step = 691 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 1384 Mb/s
  - forwarding rate greater than offering load! (forcing FWRATE=OLOAD)
Iteration 3
  - Offering load = 1728 Mb/s
  - Step = 345 Mb/s
  - Trend = increasing
  - Measured forwarding rate = 1383 Mb/s
Iteration 4
  - Offering load = 1556 Mb/s
  - Step = 172 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 1386 Mb/s
Iteration 5
  - Offering load = 1470 Mb/s
  - Step = 86 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 1384 Mb/s
Iteration 6
  - Offering load = 1427 Mb/s
  - Step = 43 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 1385 Mb/s
Iteration 7
  - Offering load = 1406 Mb/s
  - Step = 21 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 1384 Mb/s
Estimated Equilibrium Ethernet throughput= 1384 Mb/s (maximum value seen: 1386 Mb/s)

⇒ We reach about 1.386Gb/s of encrypted traffic (notice the equilibrium script bug at step 2 that could stop here).

TO DO:

~/netbenches/Atom_C2758_8Cores-Chelsio_T540-CR % ../scripts/bench-lab.sh -f bench-lab-3nodes.config -c ipsec/configs/ -p ../pktgen.configs/dualstack-vpn/ -d ipsec/results/fbsd11.1/