OpenVPN performance lab of an IBM System x3550 M3 with Intel 82580

OPenVPN performance lab of a quad cores Xeon 2.13GHz and quad-port gigabit Intel 82580

Hardware detail

This lab will test an IBM System x3550 M3 with quad cores (Intel Xeon L5630 2.13GHz, hyper-threading disabled) and a quad NIC 82580 connected to the PCI-Express Bus.

This CPU includes AES-NI: AES-CBC,AES-XTS,AES-GCM,AES-ICM.

Method used

The benchmarking method used here is detailed in Setting up a VPN (IPSec, GRE, etc...) performance benchmark lab and OpenVPN example lab.

Diagram

+---------------------+   +-------------------------------------+    +----------------------------------------+
|          R1         |   |             IBM x3550 M3            |    |                     R3                 |
|   Packet generator  |   |           Device under Test         |    |              IPSec endpoint            |
|     and receiver    |   |                                     |    |                 (AES-NI)               |
|                     |   |                                     |    |                                        |
|igb2: 198.18.0.201/24|=>=| igb2: 198.18.0.202/24               |    |                                        |
|       2001:2::201/64|   | 2001:2::202/64                      |    |                                        |
|    00:1b:21:d4:3f:2a|   | 00:1b:21:d3:8f:3e                   |    |                                        |
|                     |   |                                     |    |                                        |
|                     |   |               igb3: 198.18.1.202/24 |==>=| igb2: 198.18.1.203/24                  |
|                     |   |                  2001:2:0:1::202/64 |    |    2001:2:0:1::203/64                  |
|                     |   |                   00:1b:21:d3:8f:3f |    |     00:1b:21:c4:95:7a                  |
|                     |   |                                     |    |                                        |
|                     |   |              static routes          |    |             static routes              |
|                     |   |     198.19.0.0/16 => 198.18.1.203   |    |     198.19.0.0/16 => 198.19.0.201      |
|                     |   |     198.18.0.0/16 => 198.18.0.201   |    |     198.18.0.0/16 => 198.18.1.202      |
|                     |   |       2001:2::/49 => 2001:2::201    |    |       2001:2::/49 => 2001:2:0:1::202   |
|                     |   |2001:2:0:8000::/49 => 2001:2:0:1::203|    | 2001:2:0:8000::/49=>2001:2:0:8000::201 |
|                     |   |                                     |    |                                        |
|igb3: 198.19.0.201/24|   |                                     |    |         igb3: 198.19.0.203/24          |
|2001:2:0:8000::201/64|   |                                     |    |         2001:2:0:8000::203/64          |
|   00:1b:21:d4:3f:2b |   |                                     |    |          00:1b:21:c4:95:7b             |
+---------------------+   +-------------------------------------+    +----------------------------------------+
          ||                                                                           ||
	  ==================================<============================================

Devices configuration

Almost the same as on the forwarding performance lab but with fastforwarding disabled (not compatible with IPsec).

R2 (DUT)

Disable fastforwarding (not compliant with IPSec), configure IP address, routes and static IPSec.

/etc/rc.conf:

# IPv4 router
gateway_enable="YES"
ifconfig_igb2="198.18.0.202/24 -tso4 -tso6 -lro"
ifconfig_igb3="198.18.1.202/24 -tso4 -tso6 -lro"
# Route toward receiver will be pushed by OpenVPN client
static_routes="generator"
route_generator="-net 198.18.0.0/16 198.18.0.201"
static_arp_pairs="receiver generator"
static_arp_generator="198.18.0.201 00:1b:21:d4:3f:2a"
static_arp_receiver="198.18.1.203 00:1b:21:c4:95:7a"

# IPv6 router
ipv6_gateway_enable="YES"
ipv6_activate_all_interfaces="YES"
ifconfig_igb2_ipv6="inet6 2001:2::202 prefixlen 64"
ifconfig_igb3_ipv6="inet6 2001:2:0:1::202 prefixlen 64"
ipv6_static_routes="generator"
ipv6_route_generator="2001:2:: -prefixlen 49 2001:2::201"
static_ndp_pairs="receiver generator"
static_ndp_generator="2001:2::201 00:1b:21:d4:3f:2a"
static_ndp_receiver="2001:2:0:1::203 00:1b:21:c4:95:7a"

# Enabling OpenVPN
openvpn_enable="YES"

# Enabling AES-NI
kld_list="aesni"

/usr/local/etc/openvpn/openvpn.conf

client
dev tun
remote 198.18.1.203
ca ca.crt
cert client.crt
key client.key
ncp-disable
cipher AES-128-GCM
engine cryptodev
fast-io

R3 (reference)

Disable fastforwarding (not compliant with IPSec), configure IP address, routes and static IPSec:

# IPv4 router
gateway_enable="YES"
ifconfig_igb2="inet 198.18.1.203/24"
ifconfig_igb3="inet 198.19.0.203/24"

# routes toward generator will be installed by OpenVPN
static_routes="receiver"
route_receiver="-net 198.19.0.0/16 198.19.0.201"
static_arp_pairs="receiver generator"
static_arp_generator="198.18.1.202 00:1b:21:d3:8f:3f"
static_arp_receiver="198.19.0.201 00:1b:21:d4:3f:2b"

# IPv6 router
ipv6_gateway_enable="YES"
ipv6_activate_all_interfaces="YES"
ifconfig_igb2_ipv6="inet6 2001:2:0:1::203 prefixlen 64"
ifconfig_igb3_ipv6="inet6 2001:2:0:8000::203 prefixlen 64"

ipv6_static_routes="receiver"
ipv6_route_receiver="2001:2:0:8000:: -prefixlen 49 2001:2:0:8000::201"
static_ndp_pairs="receiver generator"
static_ndp_generator="2001:2:0:1::202 00:1b:21:d3:8f:3f"
static_ndp_receiver="2001:2:0:8000::201 00:1b:21:d4:3f:2b"

# Enabling OpenVPN
kld_list="aesni"
openvpn_enable="YES"

/usr/local/etc/openvpn/openvpn.conf:

dev tun
ca /usr/local/etc/easy-rsa/pki/ca.crt
cert /usr/local/etc/easy-rsa/pki/issued/server.crt
key /usr/local/etc/easy-rsa/pki/private/server.key
dh /usr/local/etc/easy-rsa/pki/dh.pem
server 172.16.254.0 255.255.255.0
server-ipv6 2001:db8:172::/64
ncp-disable
cipher AES-128-GCM
engine cryptodev
ifconfig-pool-persist ipp.txt
client-config-dir ccd
push "route 198.19.0.0 255.255.0.0"
push "route-ipv6 2001:2:0:8000::/49"
route 198.18.0.0 255.255.0.0
route-ipv6 2001:2::/49
keepalive 60 720
fast-io

Equilibrium throughput benchmark methodology

Once done, we start using a fast method for measuring the “equilibrium throughput” of the DUT.

From the packet generator/receiver a simple script that use netmap-pktgen will do the job:

[root@R1]/tmp# equilibrium -u -d 00:1b:21:d3:8f:3e -t igb2 -r igb3 -o 0.001
Benchmark tool using equilibrium throughput method
- Benchmark mode: Bandwitdh (bps) for VPN gateway
- UDP load = 500B, IPv4 packet size=528B, Ethernet frame size=542B
- Link rate = 1000 Mb/s
- Tolerance = 0.001
Iteration 1
  - Offering load = 500 Mb/s
  - Step = 250 Mb/s
  - Measured forwarding rate = 500 Mb/s
Iteration 2
  - Offering load = 750 Mb/s
  - Step = 250 Mb/s
  - Trend = increasing
  - Measured forwarding rate = 539 Mb/s
Iteration 3
  - Offering load = 625 Mb/s
  - Step = 125 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 543 Mb/s
Iteration 4
  - Offering load = 563 Mb/s
  - Step = 62 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 545 Mb/s
Iteration 5
  - Offering load = 532 Mb/s
  - Step = 31 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 532 Mb/s
Iteration 6
  - Offering load = 547 Mb/s
  - Step = 15 Mb/s
  - Trend = increasing
  - Measured forwarding rate = 545 Mb/s
Iteration 7
  - Offering load = 540 Mb/s
  - Step = 7 Mb/s
  - Trend = decreasing
  - Measured forwarding rate = 540 Mb/s
Iteration 8
  - Offering load = 543 Mb/s
  - Step = 3 Mb/s
  - Trend = increasing
  - Measured forwarding rate = 543 Mb/s
Iteration 9
  - Offering load = 544 Mb/s
  - Step = 1 Mb/s
  - Trend = increasing
  - Measured forwarding rate = 543 Mb/s
Estimated Equilibrium Ethernet throughput= 543 Mb/s (maximum value seen: 545 Mb/s)

⇒ OpenVPN (userspace) reach about 454Mb/s which is about half IPSec performance (kernel space)

BSD Router Project

Table of Contents