User Tools

Site Tools


documentation:examples:ecmp

Equal-cost multi-path routing (ECMP)

This lab show an ECMP (feature called RADIX_MPATH on FreeBSD) routing example.

Presentation

Network diagram

Here is the logical and physical view:

Setting-up the lab

Downloading BSD Router Project images

Download BSDRP serial image (prevent to have to use an X display) on Sourceforge.

Download Lab scripts

More information on these BSDRP lab scripts available on How to build a BSDRP router lab.

Start the lab with 6 routers. With bhyve under FreeBSD:

user:~ # BSDRP-lab-bhyve.sh -i BSDRP.amd64/BSDRP-1.71-full-amd64-serial.img.xz -n 6
BSD Router Project (http://bsdrp.net) - bhyve full-meshed lab script
Setting-up a virtual lab with 6 VM(s):
- Working directory: /tmp/BSDRP
- Each VM have 1 core(s) and 256M RAM
- Switch mode: bridge + tap
- 0 LAN(s) between all VM
- Full mesh Ethernet links between each VM
VM 1 have the following NIC:
- vtnet0 connected to VM 2.
- vtnet1 connected to VM 3.
- vtnet2 connected to VM 4.
- vtnet3 connected to VM 5.
- vtnet4 connected to VM 6.
VM 2 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 3.
- vtnet2 connected to VM 4.
- vtnet3 connected to VM 5.
- vtnet4 connected to VM 6.
VM 3 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 4.
- vtnet3 connected to VM 5.
- vtnet4 connected to VM 6.
VM 4 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 3.
- vtnet3 connected to VM 5.
- vtnet4 connected to VM 6.
VM 5 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 3.
- vtnet3 connected to VM 4.
- vtnet4 connected to VM 6.
VM 6 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 3.
- vtnet3 connected to VM 4.
- vtnet4 connected to VM 5.
For connecting to VM'serial console, you can use:
- VM 1 : cu -l /dev/nmdm1B
- VM 2 : cu -l /dev/nmdm2B
- VM 3 : cu -l /dev/nmdm3B
- VM 4 : cu -l /dev/nmdm4B
- VM 5 : cu -l /dev/nmdm5B
- VM 6 : cu -l /dev/nmdm6B

Static routing setup

VM 1 (client)

VM1 is configured as a simple host with multiples aliases:

sysrc hostname=VM1
sysrc gateway_enable=NO
sysrc ipv6_gateway_enable=NO
sysrc ifconfig_vtnet0="inet 10.0.12.1/24"
sysrc ifconfig_vtnet0_alias1="inet 10.0.12.11/32"
sysrc ifconfig_vtnet0_alias2="inet 10.0.12.21/32"
sysrc ifconfig_vtnet0_alias3="inet 10.0.12.31/32"
sysrc ifconfig_vtnet0_alias4="inet 10.0.12.41/32"
sysrc ifconfig_vtnet0_alias5="inet 10.0.12.51/32"
sysrc ifconfig_vtnet0_ipv6="inet6 2001:db8:12::1 prefixlen 64"
sysrc ifconfig_vtnet0_alias61="inet6 2001:db8:12::11 prefixlen 64"
sysrc ifconfig_vtnet0_alias62="inet6 2001:db8:12::21 prefixlen 64"
sysrc ifconfig_vtnet0_alias63="inet6 2001:db8:12::31 prefixlen 64"
sysrc ifconfig_vtnet0_alias64="inet6 2001:db8:12::41 prefixlen 64"
sysrc ifconfig_vtnet0_alias65="inet6 2001:db8:12::51 prefixlen 64"
sysrc defaultrouter="10.0.12.2"
sysrc ipv6_defaultrouter="2001:db8:12::2"
service hostname restart
service netif restart
service routing restart
config save

VM 2 (ECMP router)

VM 2 is a router with ECMP: 2 static routes toward the same destination but using 2 different next-hop. Need to disable fastforward by re-enabling ICMP redirect (disabled on BSDRP by default).

sysrc hostname=VM2
sysrc ifconfig_vtnet0="inet 10.0.12.2/24"
sysrc ifconfig_vtnet1="inet 10.0.23.2/24"
sysrc ifconfig_vtnet2="inet 10.0.24.2/24"
sysrc ifconfig_vtnet0_ipv6="inet6 2001:db8:12::2 prefixlen 64"
sysrc ifconfig_vtnet1_ipv6="inet6 2001:db8:23::2 prefixlen 64"
sysrc ifconfig_vtnet2_ipv6="inet6 2001:db8:24::2 prefixlen 64"
sysrc static_routes="VM3VM6 VM3VM5 VM4VM6 VM4VM5"
sysrc route_VM3VM6="-net 10.0.56.0/24 10.0.23.3"
sysrc route_VM3VM5="-net 10.0.35.0/24 10.0.23.3"
sysrc route_VM4VM6="-net 10.0.56.0/24 10.0.24.4"
sysrc route_VM4VM5="-net 10.0.45.0/24 10.0.24.4"
sysrc ipv6_static_routes="VM3VM6 VM3VM5 VM4VM6 VM4VM5"
sysrc ipv6_route_VM3VM6="2001:db8:56:: -prefixlen 64 2001:db8:23::3"
sysrc ipv6_route_VM3VM5="2001:db8:35:: -prefixlen 64 2001:db8:23::3"
sysrc ipv6_route_VM4VM6="2001:db8:56:: -prefixlen 64 2001:db8:24::4"
sysrc ipv6_route_VM4VM5="2001:db8:45:: -prefixlen 64 2001:db8:24::4"
sed -i '' -e '/net.inet.ip.redirect=0/d' /etc/sysctl.conf
sed -i '' -e '/net.inet6.ip6.redirect=0/d' /etc/sysctl.conf
sysctl net.inet.ip.redirect=1
sysctl net.inet6.ip6.redirect=1
service hostname restart
service netif restart
service routing restart
config save

And first problem using static route:

add net 10.0.56.0: gateway 10.0.23.3 fib 0
add net 10.0.35.0: gateway 10.0.23.3 fib 0
add net 10.0.56.0: gateway 10.0.24.4 fib 0: route already in table
add net 10.0.45.0: gateway 10.0.24.4 fib 0
add net 2001:db8:56::: gateway 2001:db8:23::3 fib 0
add net 2001:db8:35::: gateway 2001:db8:23::3 fib 0
add net 2001:db8:56::: gateway 2001:db8:24::4 fib 0: route already in table
add net 2001:db8:45::: gateway 2001:db8:24::4 fib 0
[root@VM2]~# route get 10.0.56.0/24
   route to: 10.0.56.0
destination: 10.0.56.0
       mask: 255.255.255.0
    gateway: 10.0.23.3
        fib: 0
  interface: vtnet1
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

VM 3 (router)

VM 3 is a simple router.

sysrc hostname=VM3
sysrc ifconfig_vtnet1="inet 10.0.23.3/24"
sysrc ifconfig_vtnet3="inet 10.0.35.3/24"
sysrc ifconfig_vtnet1_ipv6="inet6 2001:db8:23::3 prefixlen 64"
sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:35::3 prefixlen 64"
sysrc static_routes="VM6 VM1"
sysrc route_VM1="-net 10.0.12.0/24 10.0.23.2"
sysrc route_VM6="-net 10.0.56.0/24 10.0.35.5"
sysrc ipv6_static_routes="VM6 VM1"
sysrc ipv6_route_VM1="2001:db8:12:: -prefixlen 64 2001:db8:23::2"
sysrc ipv6_route_VM6="2001:db8:56:: -prefixlen 64 2001:db8:35::5"
service hostname restart
service netif restart
service routing restart
config save

VM 4 (router)

VM 4 is like VM 3, a simple router.

sysrc hostname=VM4
sysrc ifconfig_vtnet1="inet 10.0.24.4/24"
sysrc ifconfig_vtnet3="inet 10.0.45.4/24"
sysrc ifconfig_vtnet1_ipv6="inet6 2001:db8:24::4 prefixlen 64"
sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:45::4 prefixlen 64"
sysrc static_routes="VM6 VM1"
sysrc route_VM1="-net 10.0.12.0/24 10.0.24.2"
sysrc route_VM6="-net 10.0.56.0/24 10.0.45.5"
sysrc ipv6_static_routes="VM6 VM1"
sysrc ipv6_route_VM1="2001:db8:12:: -prefixlen 64 2001:db8:24::2"
sysrc ipv6_route_VM6="2001:db8:56:: -prefixlen 64 2001:db8:45::5"
service hostname restart
service netif restart
service routing restart
config save

VM 5 (ECMP router)

VM 5 is like VM 2, a router with ECMP: 2 static routing toward the same destination but using 2 different next-hop..

sysrc hostname=VM5
sysrc ifconfig_vtnet2="inet 10.0.35.5/24"
sysrc ifconfig_vtnet3="inet 10.0.45.5/24"
sysrc ifconfig_vtnet4="inet 10.0.56.5/24"
sysrc ifconfig_vtnet2_ipv6="inet6 2001:db8:35::5 prefixlen 64"
sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:45::5 prefixlen 64"
sysrc ifconfig_vtnet4_ipv6="inet6 2001:db8:56::5 prefixlen 64"
sysrc static_routes="VM3VM1 VM3VM2 VM4VM1 VM4VM2"
sysrc route_VM3VM1="-net 10.0.12.0/24 10.0.35.3"
sysrc route_VM3VM2="-net 10.0.23.0/24 10.0.35.3"
sysrc route_VM4VM1="-net 10.0.12.0/24 10.0.45.4"
sysrc route_VM4VM2="-net 10.0.24.0/24 10.0.45.4"
sysrc ipv6_static_routes="VM3VM1 VM3VM2 VM4VM1 VM4VM2"
sysrc ipv6_route_VM3VM1="2001:db8:12:: -prefixlen 64 2001:db8:35::3"
sysrc ipv6_route_VM3VM2="2001:db8:23:: -prefixlen 64 2001:db8:35::3"
sysrc ipv6_route_VM4VM1="2001:db8:12:: -prefixlen 64 2001:db8:45::4"
sysrc ipv6_route_VM4VM2="2001:db8:24:: -prefixlen 64 2001:db8:45::4"
sed -i '' -e '/net.inet.ip.redirect=0/d' /etc/sysctl.conf
sed -i '' -e '/net.inet6.ip6.redirect=0/d' /etc/sysctl.conf
sysctl net.inet.ip.redirect=1
sysctl net.inet6.ip6.redirect=1
service hostname restart
service netif restart
service routing restart
config save

VM 6 (server)

VM 6, like VM 1 is configured as a simple host with multiple aliases:

sysrc hostname=VM6
sysrc gateway_enable=NO
sysrc ipv6_gateway_enable=NO
sysrc ifconfig_vtnet4="inet 10.0.56.6/24"
sysrc ifconfig_vtnet4_alias1="inet 10.0.56.16/32"
sysrc ifconfig_vtnet4_alias2="inet 10.0.56.26/32"
sysrc ifconfig_vtnet4_alias3="inet 10.0.56.36/32"
sysrc ifconfig_vtnet4_alias4="inet 10.0.56.46/32"
sysrc ifconfig_vtnet4_alias5="inet 10.0.56.56/32"
sysrc ifconfig_vtnet4_ipv6="inet6 2001:db8:56::6 prefixlen 64"
sysrc ifconfig_vtnet4_alias61="inet6 2001:db8:56::16 prefixlen 64"
sysrc ifconfig_vtnet4_alias62="inet6 2001:db8:56::26 prefixlen 64"
sysrc ifconfig_vtnet4_alias63="inet6 2001:db8:56::36 prefixlen 64"
sysrc ifconfig_vtnet4_alias64="inet6 2001:db8:56::46 prefixlen 64"
sysrc ifconfig_vtnet4_alias65="inet6 2001:db8:56::56 prefixlen 64"
sysrc defaultrouter="10.0.56.5"
sysrc ipv6_defaultrouter="2001:db8:56::5"
service hostname restart
service netif restart
service routing restart
config save

Testing Load balancing

Flows from VM1 to VM6 should be load-balanced by VM2 toward VM3 and VM4. Let's check using multiple sources and destination IP addresses. Using ping record feature like with this example:

[root@VM1]~# ping -c1 -R -S 10.0.12.21 10.0.56.26
PING 10.0.56.26 (10.0.56.26) from 10.0.12.21: 56 data bytes
64 bytes from 10.0.56.26: icmp_seq=0 ttl=61 time=0.358 ms
RR:     10.0.23.2
        10.0.35.3
        10.0.56.5
        10.0.56.26
        10.0.35.5
        10.0.23.3
        10.0.12.2
        10.0.12.21

--- 10.0.56.26 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.358/0.358/0.358/0.000 ms
Source Destination Ongoing path selected by VM2 Returning path selected by VM5
10.0.12.1 10.0.56.6 VM3 VM3
10.0.12.11 10.0.56.16 VM3 VM3
10.0.12.21 10.0.56.26 VM3 VM3
10.0.12.31 10.0.56.36 VM3 VM3
10.0.12.41 10.0.56.46 VM3 VM3
10.0.12.51 10.0.56.56 VM3 VM3
2001:db8:12::1 2001:db8:56::6 VM3 VM3
2001:db8:12::11 2001:db8:56::16 VM3 VM3
2001:db8:12::21 2001:db8:56::26 VM3 VM3
2001:db8:12::31 2001:db8:56::36 VM3 VM3
2001:db8:12::41 2001:db8:56::46 VM3 VM3
2001:db8:12::51 2001:db8:56::56 VM3 VM3
10.0.12.1 10.0.45.5 VM4 VM3
There is no flow load-balancing with RADIX MPATH since FreeBSD 11, seems broken with tryforward, put proposed patch disable tryforward(fastforward) if RADIX_MPATH is enabled.
documentation/examples/ecmp.txt · Last modified: 2019/11/01 08:49 by olivier