- en
- fr
This is an old revision of the document!
Table of Contents
Equal-cost multi-path routing (ECMP)
This lab is testing new ECMP routing code from FreeBSD's review D26449
Presentation
Network diagram
Setting-up the lab
Downloading BSD Router Project images
Download BSDRP serial image (prevent to have to use an X display) on Sourceforge.
Download Lab scripts
More information on these BSDRP lab scripts available on How to build a BSDRP router lab.
Start the lab with 6 routers and 2 cores each (important to have a minimum of 2 vtnet RX & TX queues). With bhyve under FreeBSD:
user:~ # BSDRP-lab-bhyve.sh -i BSDRP.amd64/BSDRP-1.71-full-amd64-serial.img.xz -n 6 -c 2 BSD Router Project (http://bsdrp.net) - bhyve full-meshed lab script Setting-up a virtual lab with 6 VM(s): - Working directory: /tmp/BSDRP - Each VM have 1 core(s) and 256M RAM - Switch mode: bridge + tap - 0 LAN(s) between all VM - Full mesh Ethernet links between each VM VM 1 have the following NIC: - vtnet0 connected to VM 2. - vtnet1 connected to VM 3. - vtnet2 connected to VM 4. - vtnet3 connected to VM 5. - vtnet4 connected to VM 6. VM 2 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 3. - vtnet2 connected to VM 4. - vtnet3 connected to VM 5. - vtnet4 connected to VM 6. VM 3 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 2. - vtnet2 connected to VM 4. - vtnet3 connected to VM 5. - vtnet4 connected to VM 6. VM 4 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 2. - vtnet2 connected to VM 3. - vtnet3 connected to VM 5. - vtnet4 connected to VM 6. VM 5 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 2. - vtnet2 connected to VM 3. - vtnet3 connected to VM 4. - vtnet4 connected to VM 6. VM 6 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 2. - vtnet2 connected to VM 3. - vtnet3 connected to VM 4. - vtnet4 connected to VM 5. For connecting to VM'serial console, you can use: - VM 1 : cu -l /dev/nmdm1B - VM 2 : cu -l /dev/nmdm2B - VM 3 : cu -l /dev/nmdm3B - VM 4 : cu -l /dev/nmdm4B - VM 5 : cu -l /dev/nmdm5B - VM 6 : cu -l /dev/nmdm6B
Static routing setup
VM 1 (client)
VM1 is configured as a simple host with multiples aliases:
sysrc hostname=VM1 \ gateway_enable=NO \ ipv6_gateway_enable=NO \ ifconfig_vtnet0="inet 10.0.12.1/24" \ ifconfig_vtnet0_alias1="inet 10.0.12.11/32" \ ifconfig_vtnet0_alias2="inet 10.0.12.21/32" \ ifconfig_vtnet0_alias3="inet 10.0.12.31/32" \ ifconfig_vtnet0_alias4="inet 10.0.12.41/32" \ ifconfig_vtnet0_alias5="inet 10.0.12.51/32" \ ifconfig_vtnet0_ipv6="inet6 2001:db8:12::1 prefixlen 64" \ ifconfig_vtnet0_alias61="inet6 2001:db8:12::11 prefixlen 64" \ ifconfig_vtnet0_alias62="inet6 2001:db8:12::21 prefixlen 64" \ ifconfig_vtnet0_alias63="inet6 2001:db8:12::31 prefixlen 64" \ ifconfig_vtnet0_alias64="inet6 2001:db8:12::41 prefixlen 64" \ ifconfig_vtnet0_alias65="inet6 2001:db8:12::51 prefixlen 64" \ defaultrouter="10.0.12.2" \ ipv6_defaultrouter="2001:db8:12::2" service hostname restart service netif restart service routing restart config save
VM 2 (ECMP router)
VM 2 is a router with ECMP: 2 static routes toward the same destination but using 2 different next-hop.
sysrc hostname=VM2 \ ifconfig_vtnet0="inet 10.0.12.2/24" \ ifconfig_vtnet1="inet 10.0.23.2/24" \ ifconfig_vtnet2="inet 10.0.24.2/24" \ ifconfig_vtnet0_ipv6="inet6 2001:db8:12::2 prefixlen 64" \ ifconfig_vtnet1_ipv6="inet6 2001:db8:23::2 prefixlen 64" \ ifconfig_vtnet2_ipv6="inet6 2001:db8:24::2 prefixlen 64" \ static_routes="VM3VM6 VM3VM5 VM4VM6 VM4VM5" \ route_VM3VM6="-net 10.0.56.0/24 10.0.23.3" \ route_VM3VM5="-net 10.0.35.0/24 10.0.23.3" \ route_VM4VM6="-net 10.0.56.0/24 10.0.24.4" \ route_VM4VM5="-net 10.0.45.0/24 10.0.24.4" \ ipv6_static_routes="VM3VM6 VM3VM5 VM4VM6 VM4VM5" \ ipv6_route_VM3VM6="2001:db8:56:: -prefixlen 64 2001:db8:23::3" \ ipv6_route_VM3VM5="2001:db8:35:: -prefixlen 64 2001:db8:23::3" \ ipv6_route_VM4VM6="2001:db8:56:: -prefixlen 64 2001:db8:24::4" \ ipv6_route_VM4VM5="2001:db8:45:: -prefixlen 64 2001:db8:24::4" service hostname restart service netif restart service routing restart config save
Checking static route with multiple next-hop:
[root@VM2]~# netstat -rn4 | grep 10.0.56.0 10.0.56.0/24 10.0.24.4 UGS vtnet2 10.0.56.0/24 10.0.23.3 UGS vtnet1 [root@VM2]~# route -n get 10.0.56.0/24 route to: 10.0.56.0 destination: 10.0.56.0 mask: 255.255.255.0 gateway: 10.0.24.4 fib: 0 interface: vtnet2 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 100 0 [root@VM2]~# netstat -4onW Nexthop data Internet: Idx Type IFA Gateway Flags Use Mtu Netif Addrif Refcnt Prepend 1 v4/resolve 127.0.0.1 lo0/resolve HS 2 16384 lo0 1 2 v4/resolve 10.0.12.2 vtnet0/resolve 7 1500 vtnet0 1 3 v4/resolve 127.0.0.1 lo0/resolve HS 0 16384 lo0 vtnet0 1 4 v4/resolve 10.0.23.2 vtnet1/resolve 0 1500 vtnet1 1 5 v4/resolve 127.0.0.1 lo0/resolve HS 0 16384 lo0 vtnet1 1 6 v4/resolve 10.0.24.2 vtnet2/resolve 0 1500 vtnet2 1 7 v4/resolve 127.0.0.1 lo0/resolve HS 0 16384 lo0 vtnet2 1 8 v4/gw 10.0.23.2 10.0.23.3 GS 1 1500 vtnet1 2 9 v4/gw 10.0.24.2 10.0.24.4 GS 6 1500 vtnet2 2 [root@VM2]~# route -n6 get 2001:db8:56::/64 route to: 2001:db8:56:: destination: 2001:db8:56:: mask: ffff:ffff:ffff:ffff:: gateway: 2001:db8:24::4 fib: 0 interface: vtnet2 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 0 0 [root@VM2]~# netstat -6onW Nexthop data Internet6: Idx Type IFA Gateway Flags Use Mtu Netif Addrif Refcnt Prepend 1 v6/resolve ::1 lo0/resolve H 2 16384 lo0 1 2 v6/resolve fe80::1%lo0 lo0/resolve HS 0 16384 lo0 1 3 v6/resolve fe80::1%lo0 lo0/resolve 0 16384 lo0 1 4 v6/gw ::1 ::1 GRS 0 16384 lo0 4 5 v6/resolve ::1 lo0/resolve HS 0 16384 lo0 vtnet0 2 6 v6/resolve fe80::5a9c:fcff:fe01:202%vtnet0 vtnet0/resolve 0 1500 vtnet0 1 7 v6/resolve 2001:db8:12::2 vtnet0/resolve 0 1500 vtnet0 1 8 v6/resolve ::1 lo0/resolve HS 0 16384 lo0 vtnet1 2 9 v6/resolve fe80::5a9c:fcff:fe02:302%vtnet1 vtnet1/resolve 0 1500 vtnet1 1 10 v6/resolve 2001:db8:23::2 vtnet1/resolve 0 1500 vtnet1 1 11 v6/resolve ::1 lo0/resolve HS 0 16384 lo0 vtnet2 2 12 v6/resolve fe80::5a9c:fcff:fe02:402%vtnet2 vtnet2/resolve 0 1500 vtnet2 1 13 v6/resolve 2001:db8:24::2 vtnet2/resolve 0 1500 vtnet2 1 14 v6/gw 2001:db8:23::2 2001:db8:23::3 GS 0 1500 vtnet1 2 15 v6/gw 2001:db8:24::2 2001:db8:24::4 GS 0 1500 vtnet2 2
VM 3 (router)
VM 3 is a simple router.
sysrc hostname=VM3 \ ifconfig_vtnet1="inet 10.0.23.3/24" \ ifconfig_vtnet3="inet 10.0.35.3/24" \ ifconfig_vtnet1_ipv6="inet6 2001:db8:23::3 prefixlen 64" \ ifconfig_vtnet3_ipv6="inet6 2001:db8:35::3 prefixlen 64" \ static_routes="VM6 VM1" \ route_VM1="-net 10.0.12.0/24 10.0.23.2" \ route_VM6="-net 10.0.56.0/24 10.0.35.5" \ ipv6_static_routes="VM6 VM1" \ ipv6_route_VM1="2001:db8:12:: -prefixlen 64 2001:db8:23::2" \ ipv6_route_VM6="2001:db8:56:: -prefixlen 64 2001:db8:35::5" service hostname restart service netif restart service routing restart config save
VM 4 (router)
VM 4 is like VM 3, a simple router.
sysrc hostname=VM4 \ ifconfig_vtnet1="inet 10.0.24.4/24" \ ifconfig_vtnet3="inet 10.0.45.4/24" \ ifconfig_vtnet1_ipv6="inet6 2001:db8:24::4 prefixlen 64" \ ifconfig_vtnet3_ipv6="inet6 2001:db8:45::4 prefixlen 64" \ static_routes="VM6 VM1" \ route_VM1="-net 10.0.12.0/24 10.0.24.2" \ route_VM6="-net 10.0.56.0/24 10.0.45.5" \ ipv6_static_routes="VM6 VM1" \ ipv6_route_VM1="2001:db8:12:: -prefixlen 64 2001:db8:24::2" \ ipv6_route_VM6="2001:db8:56:: -prefixlen 64 2001:db8:45::5" service hostname restart service netif restart service routing restart config save
VM 5 (ECMP router)
VM 5 is like VM 2, a router with ECMP: 2 static routing toward the same destination but using 2 different next-hop..
sysrc hostname=VM5 \ ifconfig_vtnet2="inet 10.0.35.5/24" \ ifconfig_vtnet3="inet 10.0.45.5/24" \ ifconfig_vtnet4="inet 10.0.56.5/24" \ ifconfig_vtnet2_ipv6="inet6 2001:db8:35::5 prefixlen 64" \ ifconfig_vtnet3_ipv6="inet6 2001:db8:45::5 prefixlen 64" \ ifconfig_vtnet4_ipv6="inet6 2001:db8:56::5 prefixlen 64" \ static_routes="VM3VM1 VM3VM2 VM4VM1 VM4VM2" \ route_VM3VM1="-net 10.0.12.0/24 10.0.35.3" \ route_VM3VM2="-net 10.0.23.0/24 10.0.35.3" \ route_VM4VM1="-net 10.0.12.0/24 10.0.45.4" \ route_VM4VM2="-net 10.0.24.0/24 10.0.45.4" \ ipv6_static_routes="VM3VM1 VM3VM2 VM4VM1 VM4VM2" \ ipv6_route_VM3VM1="2001:db8:12:: -prefixlen 64 2001:db8:35::3" \ ipv6_route_VM3VM2="2001:db8:23:: -prefixlen 64 2001:db8:35::3" \ ipv6_route_VM4VM1="2001:db8:12:: -prefixlen 64 2001:db8:45::4" \ ipv6_route_VM4VM2="2001:db8:24:: -prefixlen 64 2001:db8:45::4" service hostname restart service netif restart service routing restart config save
VM 6 (server)
VM 6, like VM 1 is configured as a simple host with multiple aliases:
sysrc hostname=VM6 \ gateway_enable=NO \ ipv6_gateway_enable=NO \ ifconfig_vtnet4="inet 10.0.56.6/24" \ ifconfig_vtnet4_alias1="inet 10.0.56.16/32" \ ifconfig_vtnet4_alias2="inet 10.0.56.26/32" \ ifconfig_vtnet4_alias3="inet 10.0.56.36/32" \ ifconfig_vtnet4_alias4="inet 10.0.56.46/32" \ ifconfig_vtnet4_alias5="inet 10.0.56.56/32" \ ifconfig_vtnet4_ipv6="inet6 2001:db8:56::6 prefixlen 64" \ ifconfig_vtnet4_alias61="inet6 2001:db8:56::16 prefixlen 64" \ ifconfig_vtnet4_alias62="inet6 2001:db8:56::26 prefixlen 64" \ ifconfig_vtnet4_alias63="inet6 2001:db8:56::36 prefixlen 64" \ ifconfig_vtnet4_alias64="inet6 2001:db8:56::46 prefixlen 64" \ ifconfig_vtnet4_alias65="inet6 2001:db8:56::56 prefixlen 64" \ defaultrouter="10.0.56.5" \ ipv6_defaultrouter="2001:db8:56::5" service hostname restart service netif restart service routing restart config save
Testing Load balancing
Flows from VM1 to VM6 should be “flow-id shared” by VM2 toward VM3 and VM4. Let's check using multiple sources and destination IP addresses. Using ping record feature like with this example:
[root@VM1]~# ping -c1 -R -S 10.0.12.21 10.0.56.26 PING 10.0.56.26 (10.0.56.26) from 10.0.12.21: 56 data bytes 64 bytes from 10.0.56.26: icmp_seq=0 ttl=61 time=0.455 ms RR: 10.0.24.2 10.0.45.4 10.0.56.5 10.0.56.26 10.0.45.5 10.0.24.4 10.0.12.2 10.0.12.21 --- 10.0.56.26 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.455/0.455/0.455/0.000 ms
Source | Destination | Ongoing path selected by VM2 | Returning path selected by VM5 |
---|---|---|---|
10.0.12.1 | 10.0.35.5 | VM3 | VM4 |
10.0.12.1 | 10.0.45.5 | VM4 | VM4 |
10.0.12.1 | 10.0.56.6 | VM4 | VM4 |
10.0.12.11 | 10.0.56.16 | VM4 | VM4 |
10.0.12.21 | 10.0.56.26 | VM4 | VM4 |
10.0.12.31 | 10.0.56.36 | VM4 | VM4 |
10.0.12.41 | 10.0.56.46 | VM4 | VM4 |
10.0.12.51 | 10.0.56.56 | VM4 | VM4 |
2001:db8:12::1 | 2001:db8:56::6 | VM4 | VM4 |
2001:db8:12::11 | 2001:db8:56::16 | VM4 | VM4 |
2001:db8:12::21 | 2001:db8:56::26 | VM4 | VM4 |
2001:db8:12::31 | 2001:db8:56::36 | VM4 | VM4 |
2001:db8:12::41 | 2001:db8:56::46 | VM4 | VM4 |
2001:db8:12::51 | 2001:db8:56::56 | VM4 | VM4 |