BSD Router Project

User Tools

Site Tools

Trace: • Equal-cost multi-path routing (ECMP)
documentation:examples:ecmp

This is an old revision of the document!

Table of Contents

Equal-cost multi-path routing (ECMP)

This lab is testing new ECMP routing code from FreeBSD's review D26449

Presentation

Bhyve doesn't support emulating multiqueue NIC, so RSS flow-id could not be tested using a bhyve based lab: Had to use physical lab

Network diagram

Here is the logical and physical view:

Setting-up the lab

Downloading BSD Router Project images

Download BSDRP serial image on Sourceforge and upload them to the 2 ECMP routers.

Static routing setup

Client

A simple host with multiples aliases: 

sysrc hostname=client \
  gateway_enable=NO \
  ipv6_gateway_enable=NO \
  ifconfig_igb1="inet 10.0.31.3/24" \
  ifconfig_igb1_alias1="inet 10.0.31.13/32" \
  ifconfig_igb1_alias2="inet 10.0.31.23/32" \
  ifconfig_igb1_alias3="inet 10.0.31.33/32" \
  ifconfig_igb1_alias4="inet 10.0.31.43/32" \
  ifconfig_igb1_alias5="inet 10.0.31.53/32" \
  ifconfig_igb1_ipv6="inet6 2001:db8:31::3 prefixlen 64" \
  ifconfig_igb1_alias61="inet6 2001:db8:31::13 prefixlen 64" \
  ifconfig_igb1_alias62="inet6 2001:db8:31::23 prefixlen 64" \
  ifconfig_igb1_alias63="inet6 2001:db8:31::33 prefixlen 64" \
  ifconfig_igb1_alias64="inet6 2001:db8:31::43 prefixlen 64" \
  ifconfig_igb1_alias65="inet6 2001:db8:31::53 prefixlen 64" \
  static_routes="LAB" \ 
  route_LAB="-net 10.0.0.0/16 10.0.31.1" \ 
  ipv6_static_routes="LAB" \ 
  ipv6_route_LAB="2001:db8:: -prefixlen 32 2001:db8:31::1"
service hostname restart
service netif restart
service routing restart
config save

R1 (ECMP router)

R1 is a router with ECMP: 2 static routes toward the same destination but using 2 different next-hop. 

sysrc hostname=R1 \
  gateway_enable=YES \
  ipv6_gateway_enable=YES \
  ifconfig_igb0="inet 10.0.31.1/24" \ 
  ifconfig_igb0_ipv6="inet6 2001:db8:31::1 prefixlen 64" \
  ifconfig_igb1="inet 10.0.112.1/24" \
  ifconfig_igb1_ipv6="inet6 2001:db8:112::1 prefixlen 64" \
  ifconfig_igb2="inet 10.0.212.1/24" \
  ifconfig_igb2_ipv6="inet6 2001:db8:212::1 prefixlen 64" \
  static_routes="MPATH1 MPATH2" \
  route_MPATH1="-net 10.0.0.0/16 10.0.112.2" \
  route_MPATH2="-net 10.0.0.0/16 10.0.212.2" \
  ipv6_static_routes="MPATH1 MPATH2" \
  ipv6_route_MPATH1="2001:db8:: -prefixlen 32 2001:db8:112::2" \
  ipv6_route_MPATH2="2001:db8:: -prefixlen 32 2001:db8:212::2"
service hostname restart
service netif restart
service routing restart
config save

Checking static route with multiple next-hop: 

[root@R1]~# # netstat -rn4 | grep 10.0.0.0/16
10.0.0.0/16        10.0.112.2         UGS        igb1
10.0.0.0/16        10.0.212.2         UGS        igb2

[root@R1]~# route -n get 10.0.0.0/16
   route to: 10.0.0.0
destination: 10.0.0.0
       mask: 255.255.0.0
    gateway: 10.0.112.2
        fib: 0
  interface: igb1
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         0         0

[root@R1]~# netstat -4onW
Nexthop data

Internet:
Idx   Type         IFA                Gateway             Flags      Use Mtu         Netif     Addrif Refcnt Prepend
1       v4/resolve 127.0.0.1          lo0/resolve        HS            0  16384        lo0               1
2       v4/resolve 10.0.31.1          igb0/resolve                     0   1500       igb0               1
3       v4/resolve 127.0.0.1          lo0/resolve        HS            0  16384        lo0      igb0     1
4       v4/resolve 10.0.112.1         igb1/resolve                     0   1500       igb1               1
5       v4/resolve 127.0.0.1          lo0/resolve        HS            0  16384        lo0      igb1     1
6       v4/resolve 10.0.212.1         igb2/resolve                     0   1500       igb2               1
7       v4/resolve 127.0.0.1          lo0/resolve        HS            0  16384        lo0      igb2     1
8            v4/gw 10.0.112.1         10.0.112.2         GS            0   1500       igb1               1
9            v4/gw 10.0.212.1         10.0.212.2         GS            0   1500       igb2               1
    
[root@R1]~# netstat -rn6 | grep 2001:db8::/32
2001:db8::/32                     2001:db8:212::2               UGS        igb2
2001:db8::/32                     2001:db8:112::2               UGS        igb1
      
[root@R1]~# route -n6 get 2001:db8::/32
   route to: 2001:db8::
destination: 2001:db8::
       mask: ffff:ffff::
    gateway: 2001:db8:212::2
        fib: 0
  interface: igb2
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         0         0

[root@R1]~# netstat -6onW
Nexthop data

Internet6:
Idx   Type         IFA                           Gateway                        Flags      Use Mtu       Netif   Addrif Refcnt Prepend
1       v6/resolve ::1                           lo0/resolve                   H             0  16384      lo0             1
2       v6/resolve fe80::1%lo0                   lo0/resolve                   HS            0  16384      lo0             1
3       v6/resolve fe80::1%lo0                   lo0/resolve                                 0  16384      lo0             1
4       v6/resolve ::1                           lo0/resolve                   HS            0  16384      lo0    igb0     2
5       v6/resolve fe80::20d:b9ff:fe45:79e4%igb0 igb0/resolve                                0   1500     igb0             1
6       v6/resolve 2001:db8:31::1                igb0/resolve                                0   1500     igb0             1
7       v6/resolve fe80::20d:b9ff:fe45:79e5%igb1 igb1/resolve                                0   1500     igb1             1
8       v6/resolve ::1                           lo0/resolve                   HS            0  16384      lo0    igb1     2
9       v6/resolve 2001:db8:112::1               igb1/resolve                                0   1500     igb1             1
10      v6/resolve fe80::20d:b9ff:fe45:79e6%igb2 igb2/resolve                                0   1500     igb2             1
11      v6/resolve ::1                           lo0/resolve                   HS            0  16384      lo0    igb2     2
12           v6/gw ::1                           ::1                           GRS           0  16384      lo0             4
13           v6/gw 2001:db8:112::1               2001:db8:112::2               GS            0   1500     igb1             1
14           v6/gw 2001:db8:212::1               2001:db8:212::2               GS            0   1500     igb2             1
15      v6/resolve 2001:db8:212::1               igb2/resolve                                0   1500     igb2             1

Look like this router will prefer igb1 for inet4 and igb2 for inet6.

R5 (ECMP router)

R5 is like R2, a router with ECMP: 2 static routing toward the same destination but using 2 different next-hop.. 

sysrc hostname=R2 \
  gateway_enable=YES \
  ipv6_gateway_enable=YES \
  ifconfig_igb0="inet 10.0.24.2/24" \
  ifconfig_igb0_ipv6="inet6 2001:db8:24::2 prefixlen 64" \
  ifconfig_igb1="inet 10.0.112.2/24" \
  ifconfig_igb1_ipv6="inet6 2001:db8:112::2 prefixlen 64" \
  ifconfig_igb2="inet 10.0.212.2/24" \
  ifconfig_igb2_ipv6="inet6 2001:db8:212::2 prefixlen 64" \
  static_routes="MPATH1 MPATH2" \
  route_MPATH1="-net 10.0.0.0/16 10.0.112.1" \
  route_MPATH2="-net 10.0.0.0/16 10.0.212.1" \
  ipv6_static_routes="MPATH1 MPATH2" \
  ipv6_route_MPATH1="2001:db8:: -prefixlen 32 2001:db8:112::1" \
  ipv6_route_MPATH2="2001:db8:: -prefixlen 32 2001:db8:212::1"
service hostname restart
service netif restart
service routing restart
config save

And checking: 

# route get 10.0.0.0/16
   route to: 10.0.0.0
destination: 10.0.0.0
       mask: 255.255.0.0
    gateway: 10.0.212.1
        fib: 0
  interface: igb2
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         0         0


# route -6 get 2001:db8::/32
   route to: 2001:db8::
destination: 2001:db8::
       mask: ffff:ffff::
    gateway: 2001:db8:112::1
        fib: 0
  interface: igb1
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         0         0

Look like this router will prefer to use igb2 for inet4 and igb1 for inet6.

Server

A simple host with multiple aliases: 

sysrc hostname=server \
  gateway_enable=NO \
  ipv6_gateway_enable=NO \
  ifconfig_vr0="inet 10.0.24.4/24" \
  ifconfig_vr0_alias1="inet 10.0.24.14/32" \
  ifconfig_vr0_alias2="inet 10.0.24.24/32" \
  ifconfig_vr0_alias3="inet 10.0.24.34/32" \
  ifconfig_vr0_alias4="inet 10.0.24.44/32" \
  ifconfig_vr0_alias5="inet 10.0.24.54/32" \
  ifconfig_vr0_ipv6="inet6 2001:db8:24::4 prefixlen 64" \
  ifconfig_vr0_alias61="inet6 2001:db8:24::14 prefixlen 64" \
  ifconfig_vr0_alias62="inet6 2001:db8:24::24 prefixlen 64" \
  ifconfig_vr0_alias63="inet6 2001:db8:24::34 prefixlen 64" \
  ifconfig_vr0_alias64="inet6 2001:db8:24::44 prefixlen 64" \
  ifconfig_vr0_alias65="inet6 2001:db8:24::54 prefixlen 64" \
  static_routes="LAB" \
  route_LAB="-net 10.0.0.0/16 10.0.24.2" \
  ipv6_static_routes="LAB" \
  ipv6_route_LAB="2001:db8:: -prefixlen 32 2001:db8:24::2"
service hostname restart
service netif restart
service routing restart
config save

FRR Multipath setup

Replacing static routes by FRR (OSPF) compiled with MULTIPATH option.

R1 (ECMP router)

In place of static routes, FRR is enabled: 

sysrc frr_vtysh_boot="YES" \
  frr_enable="YES" \
  frr_daemons="zebra ospfd ospf6d" \
  watchfrr_flags=" -d -r /usr/sbin/servicebBfrrbBrestartbB%s -s /usr/sbin/servicebBfrrbBstartbB%s -k /usr/sbin/servicebBfrrbBstopbB%s -b bB -t 30 zebra ospfd ospf6d" \
  watchfrr_enable="YES"

cat > /usr/local/etc/frr/frr.conf <EOF
frr version 7.4
frr defaults traditional
hostname homelab1
!
interface igb0
 ipv6 ospf6 passive
!
router ospf
 ospf router-id 1.1.1.1
 passive-interface igb0
 network 10.0.31.0/24 area 0
 network 10.0.112.0/24 area 0
 network 10.0.212.0/24 area 0
!
router ospf6
 interface igb0 area 0.0.0.0
 interface igb1 area 0.0.0.0
 interface igb2 area 0.0.0.0
!
line vty
!
EOF
service frr start
service watchfrr start

R2 (ECMP router)

In place of static routes, FRR is enabled: 

sysrc frr_vtysh_boot="YES" \
  frr_enable="YES" \
  frr_daemons="zebra ospfd ospf6d" \
  watchfrr_flags=" -d -r /usr/sbin/servicebBfrrbBrestartbB%s -s /usr/sbin/servicebBfrrbBstartbB%s -k /usr/sbin/servicebBfrrbBstopbB%s -b bB -t 30 zebra ospfd ospf6d" \
  watchfrr_enable="YES"

cat > /usr/local/etc/frr/frr.conf <EOF
frr version 7.4
frr defaults traditional
hostname router2
!
router ospf
 ospf router-id 2.2.2.2
 passive-interface igb0
 network 10.0.24.0/24 area 0
 network 10.0.112.0/24 area 0
 network 10.0.212.0/24 area 0
!
line vty
!
EOF
service frr start
service watchfrr start

Checking routes installed

On R1: 

[root@router1]~# vtysh

Hello, this is FRRouting (version 7.4).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

homelab1# sh ip route 10.0.24.0/24
Routing entry for 10.0.24.0/24
  Known via "ospf", distance 110, metric 20, best
  Last update 00:06:39 ago
  * 10.0.112.2, via igb1, weight 1
  * 10.0.212.2, via igb2, weight 1

router1# exit
[root@router1]~# netstat -rn4 | grep 10.0.24.0/24
10.0.24.0/24       10.0.212.2         UG1        igb2
10.0.24.0/24       10.0.112.2         UG1        igb1

Test Load balancing IP packets

Flows from the client to the server should be “flow-id shared” between the 2 paths. Let's check using multiple sources and destination IP addresses.

Source Destination Ongoing path selected by R1 Returning path selected by R2
10.0.31.3 10.0.24.4 igb1 igb2
10.0.31.3 10.0.24.14 igb2 igb1
10.0.31.13 10.0.24.14 igb2 igb1
10.0.31.23 10.0.24.24 igb1 igb2
10.0.31.33 10.0.24.34 igb2 igb2
10.0.31.43 10.0.24.44 igb1 igb2
10.0.31.53 10.0.24.54 igb1 igb1
2001:db8:31::3 2001:db8:24::4 igb1 igb1
2001:db8:31::13 2001:db8:24::14 igb1 igb2
2001:db8:31::23 2001:db8:24::24 igb2 igb2
2001:db8:31::33 2001:db8:24::34 igb2 igb1
2001:db8:31::43 2001:db8:24::44 igb2 igb1
documentation/examples/ecmp.1600866077.txt.gz · Last modified: 2020/09/23 15:01 by olivier
Except where otherwise noted, content on this wiki is licensed under the following license: BSD 2-Clause
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki