Forwarding performance lab of an one core IBM server and dual-port gigabit Intel NIC

This lab will test an IBM eServer xSeries 306m with one core (Intel Pentium4 3.00GHz, hyper-threading disabled) and a dual NIC 82546GB connected to the PCI-X Bus. Overloading a one core server should be easy.

The lab is detailed here: Setting up a forwarding performance benchmark lab.

BSDRP-amd64 v1.4 (FreeBSD 9.1) is used on the router.

+-------------------+      +----------------------------------------+      +-------------------+
| Packet generator  |      |            Device under Test           |      |  Packet receiver  |
|    em0: 2.2.2.2   |=====>| em1: 2.2.2.3              em0: 1.1.1.3 |=====>|    em0: 1.1.1.1   |
| 00:1b:21:d5:66:15 |      | 00:0e:0c:de:45:df    00:0e:0c:de:45:de |      | 00:1b:21:d5:66:0e |
+-------------------+      +----------------------------------------+      +-------------------+

Generator will use this command:

pkt-gen -i em0 -t 0 -l 42 -d 1.1.1.1 -D 00:0e:0c:de:45:df -s 2.2.2.2 -w 10

Receiver will use this command:

pkt-gen -i em0 -w 10

For this small lab, we will configure the router.

First, disable Ethernet flow-control:

echo "hw.em.0.fc=0" >> /etc/sysctl.conf
echo "hw.em.1.fc=0" >> /etc/sysctl.conf
sysctl hw.em.0.fc=0
sysctl hw.em.1.fc=0

Here is the modified value of the default BSDRP /etc/rc.conf for static ARP:

ifconfig_em0="inet 1.1.1.3/24"
ifconfig_em1="inet 2.2.2.3/24"
static_arp_pairs="receiver generator"
static_arp_receiver="1.1.1.1 00:1b:21:d5:66:0e"
static_arp_generator="2.2.2.2 00:1b:21:d5:66:15"

Default FreeBSD NIC parameters are used for this first test.

Edit BSDRP /boot/loader.conf.local and comment all NIC tunning, then reboot.

[root@BSDRP]~# sysctl hw.em.
hw.em.rx_process_limit: 100
hw.em.txd: 1024
hw.em.rxd: 1024

The generator will push packet at 1.4Mpps:

root@generator:~ # pkt-gen -i em0 -t 0 -l 42 -d 1.1.1.1 -D 00:0e:0c:de:45:df -s 2.2.2.2 -w 10
main [832] ether_aton(00:0e:0c:de:45:df) gives 0x800f9b292
main [900] map size is 334980 Kb
main [922] mmapping 334980 Kbytes
Sending on em0: 1 queues, 1 threads and 1 cpus.
2.2.2.2 -> 1.1.1.1 (00:1b:21:d5:66:15 -> 00:0e:0c:de:45:df)
main [975] Wait 10 secs for phy reset
main [977] Ready...
sender_body [479] start
main [1085] 1405293 pps
main [1085] 1406363 pps
main [1085] 1406409 pps
main [1085] 1406509 pps
main [1085] 1406560 pps
main [1085] 1406439 pps
main [1085] 1405242 pps
...

Meanwhile the receiver show this:

root@receiver:~ # pkt-gen -i em0 -w 10
main [900] map size is 334980 Kb
main [922] mmapping 334980 Kbytes
Receiving from em0: 1 queues, 1 threads and 1 cpus.
main [975] Wait 10 secs for phy reset
main [977] Ready...
receiver_body [621] waiting for initial packets, poll returns 0 0
main [1085] 0 pps
receiver_body [621] waiting for initial packets, poll returns 0 0
main [1085] 0 pps
main [1085] 159098 pps
main [1085] 400260 pps
main [1085] 400247 pps
...
main [1085] 400198 pps
main [1085] 400287 pps
main [1085] 400240 pps
main [1085] 400235 pps
main [1085] 400245 pps
main [1085] 400232 pps
main [1085] 215381 pps
main [1085] 0 pps
Received 30859400 packets, in 77.13 seconds.
Speed: 400.07Kpps.

Receiver results in Kpps for 5 tests (with a reboot between them):

400.15
400.10
400.09
399.88
399.93

⇒ the receiver measure 400Kpps.

Now what about the router stats:

[root@BSDRP]~# netstat -ihw 1
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
      387k  983k     0        22M       387k     0        15M     0
      389k  981k     0        22M       389k     0        16M     0
      390k  982k     0        22M       390k     0        16M     0
      390k  982k     0        22M       390k     0        16M     0
      390k  983k     0        22M       390k     0        16M     0
      390k  979k     0        22M       390k     0        16M     0
      390k  982k     0        22M       390k     0        16M     0
      390k  982k     0        22M       390k     0        16M     0
      390k  983k     0        22M       390k     0        16M     0
[root@BSDRP]~# vmstat -i | head -1 ; vmstat -i | grep em
interrupt                          total       rate
irq17: em0 bge1                  2555941       2745
irq18: em1 uhci2                 2555879       2745
[root@BSDRP]~# top -nCHSIzs1
last pid:  1334;  load averages:  0.00,  0.02,  0.05  up 0+00:15:38    10:17:26
76 processes:  2 running, 58 sleeping, 16 waiting

Mem: 9984K Active, 8432K Inact, 64M Wired, 17M Buf, 894M Free
Swap:


  PID USERNAME PRI NICE   SIZE    RES STATE    TIME    CPU COMMAND
    0 root     -92    0     0K   176K -        9:30 93.65% kernel{em1 taskq}
    0 root     -92    0     0K   176K -        0:27  3.76% kernel{em0 taskq}
   11 root     -92    -     0K   256K WAIT     0:07  0.68% intr{irq17: em0 bge1}
   11 root     -92    -     0K   256K WAIT     0:06  0.59% intr{irq18: em1 uhci2}

[root@BSDRP]~# vmstat -z | head -1 ; vmstat -z | grep -i mbuf
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
mbuf_packet:            256,      0,    1024,     128,263652366,   0,   0
mbuf:                   256,      0,       1,     137,     137,   0,   0
mbuf_cluster:          2048, 262144,    1152,       6,    1152,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0

⇒ The router is still very well responding, but it display a forwarding rate of 390Kpps. The receiver measured 400Kpps, there is a 10Kpps gap between them.

We need to check the switch stats for a tie:

switch>sh int Gi0/10
GigabitEthernet0/10 is up, line protocol is up
  Hardware is Gigabit Ethernet, address is 000c.307c.208a (bia 000c.307c.208a)
  Description: Receiver-em0
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 52/255, rxload 1/255
  (etc...)
  30 second input rate 0 bits/sec, 0 packets/sec
  30 second output rate 204464000 bits/sec, 399348 packets/sec

⇒ Switch stats confirm the number of 400Kpps received: There is a problem with FreeBSD self-counters that misses about 10Kpps in this case.

BSDRP NIC parameters are used for this first test:

• Maximum number of received packets to process at a time is increase to 500
• Number of transmit/received descriptors per queue are increase to their maxium (4096)

[root@BSDRP]~# sysctl hw.em.
hw.em.rx_process_limit: 500
hw.em.txd: 4096
hw.em.rxd: 4096

Receiver results in Kpps for 5 tests (with a reboot between them):

405.29
405.23
406.25
405.20
404.98

⇒ Throughput increase to 405Kpps: A small 5Kpps gain.

Some router counters:

[root@BSDRP]~# vmstat -i | head -1 ; vmstat -i | grep em
interrupt                          total       rate
irq17: em0 bge1                   181885        301
irq18: em1 uhci2                  181884        301

[root@BSDRP]~# top -nCHSIzs1
last pid:  1344;  load averages:  0.00,  0.05,  0.07  up 0+00:10:19    10:31:13
76 processes:  2 running, 58 sleeping, 16 waiting

Mem: 9976K Active, 8300K Inact, 85M Wired, 17M Buf, 873M Free
Swap:

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME    CPU COMMAND
    0 root     -92    0     0K   176K -        3:39 93.07% kernel{em1 taskq}
    0 root     -92    0     0K   176K -        0:11  4.20% kernel{em0 taskq}

[root@BSDRP]~# vmstat -z | head -1 ; vmstat -z | grep -i mbuf
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
mbuf_packet:            256,      0,    8704,     512,110372794,   0,   0
mbuf:                   256,      0,       1,     128,     135,   0,   0
mbuf_cluster:          2048, 262144,    9216,       6,    9216,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0

⇒ The server is still very well responding.

There is no IRQ storm neither mbuf problem: It's taskq em that consume all CPU resources.

For this test we completely disable the maximum number of received packets to process at a time (-1):

[root@BSDRP]~# sysctl hw.em.
hw.em.rx_process_limit: -1
hw.em.txd: 4096
hw.em.rxd: 4096

Receiver results in Kpps for 5 tests (with a reboot between them):

410.52
409.65
410.31
408.92
410.52

⇒ Performance increased to 410Kpps (10Kpps gain regarding default value, and 5Kpps gain regarding tuned-but-still-limited value).

What's about the router stats now:

[root@BSDRP3]~# vmstat -i | head -1 ; vmstat -i | grep em
interrupt                          total       rate
irq17: em0 bge1                    81956         84
irq18: em1 uhci2                   81928         84
[root@BSDRP3]~# top -nCHSIzs1
last pid:  1343;  load averages:  3.05,  2.29,  1.49  up 0+00:19:57    10:56:06
80 processes:  5 running, 59 sleeping, 16 waiting

Mem: 10M Active, 8348K Inact, 93M Wired, 17M Buf, 865M Free
Swap:

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME    CPU COMMAND
    0 root     -92    0     0K   176K -       17:02 100.00% kernel{em1 taskq}

⇒ The router is very very slow to respond, almost unusable. Receiving packet consume all its CPU.

Ministat graphs:

x hw.em.txd-rxd=1024.hw.em.rx_proc_lim=100
+ hw.em.txd-rxd=4096.hw.em.rx_proc_lim=500
* hw.em.txd-rxd=4096.hw.em.rx_proc_lim=-1
+----------------------------------------------------------------------------------------------------+
|x x                                              +                                                * |
|x x                                            + ++       +                        *      *     * * |
||AM                                           |__M_A___|                              |______A__M__||
+----------------------------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5        399.88        400.15        400.09        400.03    0.11768602
+   5        404.98        406.25        405.23        405.39     0.4948232
Difference at 95.0% confidence
        5.36 +/- 0.524533
        1.3399% +/- 0.131123%
        (Student's t, pooled s = 0.359653)
*   5        408.92        410.52        410.31       409.984    0.69363535
Difference at 95.0% confidence
        9.954 +/- 0.725551
        2.48831% +/- 0.181374%
        (Student's t, pooled s = 0.497484)

Now we will test the impact of enabling a simple IPFW rules:

cat > /etc/ipfw.rules <<'EOF'
#!/bin/sh
fwcmd="/sbin/ipfw"
# Flush out the list before we begin.
${fwcmd} -f flush
${fwcmd} add 3000 allow ip from any to any
'EOF'

echo 'firewall_enable="YES"' >> /etc/rc.conf
echo 'firewall_script="/etc/ipfw.rules"  >> /etc/rc.conf

Receiver results in Kpps for 5 tests (with a reboot between them):

320.63
320.13
320.79
320.18
320.52

⇒ Throughput reduced to 320Kpps: Enabling ipfw add an impact of about 80Kpps on this server. Router still respond perfectly on the CLI.

cat >/etc/pf.conf <<'EOF'
set skip on lo0
pass
'EOF'

echo 'pf_enable="YES"' >> /etc/rc.conf

Receiver results in Kpps for 5 tests (with a reboot between them):

272.40
274.78
272.56
275.65
274.51

⇒ Very big performance impact here ! Drop to 274Kpps and router is not responsive at all: If watchdog is enabled It will trigger a reboot of the router.

ministat graphs:

x ipfw
+ pf
+----------------------------------------------------------------------------------------------------+
|+                                                                                                 xx|
|+   ++ +                                                                                          xx|
||__AM_|                                                                                           A||
+----------------------------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5        320.13        320.79        320.52        320.45    0.28644371
+   5         272.4        275.65        274.51        273.98     1.4337538
Difference at 95.0% confidence
        -46.47 +/- 1.50781
        -14.5015% +/- 0.47053%
        (Student's t, pooled s = 1.03385)