documentation:examples:ecmp
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
documentation:examples:ecmp [2019/10/30 14:50] – [Router 5 (ECMP router)] olivier | documentation:examples:ecmp [2022/12/11 18:40] – [Checking routes installed] olivier | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Equal-cost multi-path routing (ECMP) ====== | ====== Equal-cost multi-path routing (ECMP) ====== | ||
- | This lab show an ECMP (feature called RADIX_MPATH on FreeBSD) | + | This lab need a FreeBSD 13-head r366390 minimum ([[https:// |
===== Presentation ===== | ===== Presentation ===== | ||
+ | <note warning> | ||
==== Network diagram ==== | ==== Network diagram ==== | ||
Here is the logical and physical view: | Here is the logical and physical view: | ||
- | {{: | + | {{: |
===== Setting-up the lab ===== | ===== Setting-up the lab ===== | ||
Line 15: | Line 15: | ||
==== Downloading BSD Router Project images ==== | ==== Downloading BSD Router Project images ==== | ||
- | Download BSDRP serial image (prevent to have to use an X display) | + | Download BSDRP serial image on Sourceforge |
- | ==== Download Lab scripts ===== | ||
- | |||
- | More information on these BSDRP lab scripts available on [[documentation: | ||
- | |||
- | Start the lab with 6 routers. | ||
- | With bhyve under FreeBSD: | ||
- | < | ||
- | user:~ # BSDRP-lab-bhyve.sh -i BSDRP.amd64/ | ||
- | BSD Router Project (http:// | ||
- | Setting-up a virtual lab with 6 VM(s): | ||
- | - Working directory: /tmp/BSDRP | ||
- | - Each VM have 1 core(s) and 256M RAM | ||
- | - Switch mode: bridge + tap | ||
- | - 0 LAN(s) between all VM | ||
- | - Full mesh Ethernet links between each VM | ||
- | VM 1 have the following NIC: | ||
- | - vtnet0 connected to VM 2. | ||
- | - vtnet1 connected to VM 3. | ||
- | - vtnet2 connected to VM 4. | ||
- | - vtnet3 connected to VM 5. | ||
- | - vtnet4 connected to VM 6. | ||
- | VM 2 have the following NIC: | ||
- | - vtnet0 connected to VM 1. | ||
- | - vtnet1 connected to VM 3. | ||
- | - vtnet2 connected to VM 4. | ||
- | - vtnet3 connected to VM 5. | ||
- | - vtnet4 connected to VM 6. | ||
- | VM 3 have the following NIC: | ||
- | - vtnet0 connected to VM 1. | ||
- | - vtnet1 connected to VM 2. | ||
- | - vtnet2 connected to VM 4. | ||
- | - vtnet3 connected to VM 5. | ||
- | - vtnet4 connected to VM 6. | ||
- | VM 4 have the following NIC: | ||
- | - vtnet0 connected to VM 1. | ||
- | - vtnet1 connected to VM 2. | ||
- | - vtnet2 connected to VM 3. | ||
- | - vtnet3 connected to VM 5. | ||
- | - vtnet4 connected to VM 6. | ||
- | VM 5 have the following NIC: | ||
- | - vtnet0 connected to VM 1. | ||
- | - vtnet1 connected to VM 2. | ||
- | - vtnet2 connected to VM 3. | ||
- | - vtnet3 connected to VM 4. | ||
- | - vtnet4 connected to VM 6. | ||
- | VM 6 have the following NIC: | ||
- | - vtnet0 connected to VM 1. | ||
- | - vtnet1 connected to VM 2. | ||
- | - vtnet2 connected to VM 3. | ||
- | - vtnet3 connected to VM 4. | ||
- | - vtnet4 connected to VM 5. | ||
- | For connecting to VM' | ||
- | - VM 1 : cu -l /dev/nmdm1B | ||
- | - VM 2 : cu -l /dev/nmdm2B | ||
- | - VM 3 : cu -l /dev/nmdm3B | ||
- | - VM 4 : cu -l /dev/nmdm4B | ||
- | - VM 5 : cu -l /dev/nmdm5B | ||
- | - VM 6 : cu -l /dev/nmdm6B | ||
- | </ | ||
===== Static routing setup ===== | ===== Static routing setup ===== | ||
- | ==== VM 1 (client) | + | ==== Client |
- | VM1 is configured as a simple host with multiples aliases: | + | A simple host with static routes: |
< | < | ||
- | sysrc hostname=R1 | + | sysrc hostname=client \ |
- | sysrc gateway_enable=NO | + | gateway_enable=NO |
- | sysrc ipv6_gateway_enable=NO | + | ipv6_gateway_enable=NO |
- | sysrc ifconfig_vtnet0="inet 10.0.12.1/ | + | |
- | sysrc ifconfig_vtnet0_alias1="inet 10.0.12.11/32" | + | |
- | sysrc ifconfig_vtnet0_alias2="inet 10.0.12.21/ | + | |
- | sysrc ifconfig_vtnet0_alias3="inet 10.0.12.31/ | + | |
- | sysrc ifconfig_vtnet0_alias4=" | + | |
- | sysrc ifconfig_vtnet0_alias5=" | + | |
- | sysrc ifconfig_vtnet0_ipv6="inet6 2001: | + | |
- | sysrc ifconfig_vtnet0_alias61="inet6 2001:db8:12::11 prefixlen | + | |
- | sysrc ifconfig_vtnet0_alias62=" | + | |
- | sysrc ifconfig_vtnet0_alias63=" | + | |
- | sysrc ifconfig_vtnet0_alias64=" | + | |
- | sysrc ifconfig_vtnet0_alias65=" | + | |
- | sysrc defaultrouter=" | + | |
- | sysrc ipv6_defaultrouter=" | + | |
service hostname restart | service hostname restart | ||
service netif restart | service netif restart | ||
Line 107: | Line 40: | ||
</ | </ | ||
- | ==== VM 2 (ECMP router) ==== | + | ==== R1 (ECMP router) ==== |
- | VM 2 is a router with ECMP: 2 static routes toward the same destination but using 2 different next-hop. | + | R1 is a router with ECMP: 2 static routes toward the same destination but using 2 different next-hop. |
< | < | ||
- | sysrc hostname=R2 | + | |
- | sysrc ifconfig_vtnet0="inet 10.0.12.2/ | + | sysrc hostname=R1 \ |
- | sysrc ifconfig_vtnet1="inet 10.0.23.2/24" | + | |
- | sysrc ifconfig_vtnet2="inet 10.0.24.2/ | + | ipv6_gateway_enable=YES \ |
- | sysrc ifconfig_vtnet0_ipv6=" | + | ifconfig_igb0="inet 10.0.12.2/ |
- | sysrc ifconfig_vtnet1_ipv6="inet6 2001: | + | |
- | sysrc ifconfig_vtnet2_ipv6=" | + | |
- | sysrc static_routes=" | + | |
- | sysrc route_R3R6="-net 10.0.56.0/24 10.0.23.3" | + | |
- | sysrc route_R3R5=" | + | |
- | sysrc route_R4R6="-net 10.0.56.0/24 10.0.24.4" | + | static_routes=" |
- | sysrc route_R4R5=" | + | |
- | sysrc ipv6_static_routes=" | + | |
- | sysrc ipv6_route_R3R6=" | + | ipv6_static_routes=" |
- | sysrc ipv6_route_R3R5=" | + | |
- | sysrc ipv6_route_R4R6=" | + | |
- | sysrc ipv6_route_R4R5=" | + | |
service hostname restart | service hostname restart | ||
service netif restart | service netif restart | ||
Line 135: | Line 67: | ||
</ | </ | ||
- | ==== VM 3 (router) ==== | + | Checking static route with multiple next-hop: |
- | VM 3 is a simple | + | < |
+ | root@R1:~ # netstat -rn4 | grep 10.0.0.0/ | ||
+ | 10.0.0.0/ | ||
+ | 10.0.0.0/ | ||
+ | |||
+ | root@R1:~ # netstat -4onW | ||
+ | Nexthop data | ||
+ | |||
+ | Internet: | ||
+ | Idx | ||
+ | 1 | ||
+ | 2 | ||
+ | 3 | ||
+ | 4 | ||
+ | 5 | ||
+ | 6 | ||
+ | 7 | ||
+ | 8 v4/gw 10.0.231.2 | ||
+ | 9 v4/gw 10.0.232.2 | ||
+ | |||
+ | root@R1:~ # netstat -6onW | ||
+ | Nexthop data | ||
+ | |||
+ | Internet6: | ||
+ | Idx | ||
+ | 1 | ||
+ | 2 | ||
+ | 3 | ||
+ | 4 | ||
+ | 5 | ||
+ | 6 | ||
+ | 7 | ||
+ | 8 | ||
+ | 9 | ||
+ | 10 v6/gw ::1 :: | ||
+ | 11 v6/gw 2001: | ||
+ | 12 v6/gw 2001: | ||
+ | </ | ||
+ | ==== R2 (ECMP router) ==== | ||
+ | |||
+ | R2 is like R1, a router | ||
< | < | ||
- | sysrc hostname=R3 | + | sysrc hostname=R2 \ |
- | sysrc ifconfig_vtnet1="inet 10.0.23.3/ | + | |
- | sysrc ifconfig_vtnet3="inet 10.0.35.3/ | + | ipv6_gateway_enable=YES \ |
- | sysrc ifconfig_vtnet1_ipv6=" | + | ifconfig_igb0="inet 10.0.34.3/ |
- | sysrc ifconfig_vtnet3_ipv6=" | + | ifconfig_igb0_ipv6=" |
- | sysrc static_routes=" | + | |
- | sysrc route_R1="-net 10.0.12.0/ | + | |
- | sysrc route_R6="-net 10.0.56.0/24 10.0.35.5" | + | |
- | sysrc ipv6_static_routes=" | + | ifconfig_igb2_ipv6=" |
- | sysrc ipv6_route_R1=" | + | static_routes=" |
- | sysrc ipv6_route_R6=" | + | |
+ | route_DST2=" | ||
+ | route_SRC1="-net 10.0.12.0/ | ||
+ | | ||
+ | ipv6_static_routes=" | ||
+ | | ||
+ | ipv6_route_MPATH2=" | ||
+ | ipv6_route_SRC1=" | ||
+ | | ||
service hostname restart | service hostname restart | ||
service netif restart | service netif restart | ||
Line 156: | Line 136: | ||
config save | config save | ||
</ | </ | ||
+ | ==== Server ==== | ||
- | ==== VM 4 (router) ==== | + | A simple |
- | + | ||
- | VM 4 is like VM 3, a simple | + | |
< | < | ||
- | sysrc hostname=R4 | + | sysrc hostname=server \ |
- | sysrc ifconfig_vtnet1="inet 10.0.24.4/ | + | |
- | sysrc ifconfig_vtnet3="inet 10.0.45.4/ | + | ipv6_gateway_enable=NO \ |
- | sysrc ifconfig_vtnet1_ipv6=" | + | |
- | sysrc ifconfig_vtnet3_ipv6="inet6 2001: | + | |
- | sysrc static_routes="R6 R1" | + | |
- | sysrc route_R1="-net 10.0.12.0/24 10.0.24.2" | + | |
- | sysrc route_R6="-net 10.0.56.0/24 10.0.45.5" | + | |
- | sysrc ipv6_static_routes=" | + | |
- | sysrc ipv6_route_R1=" | + | ipv6_static_routes=" |
- | sysrc ipv6_route_R6=" | + | |
+ | ipv6_route_231=" | ||
+ | | ||
service hostname restart | service hostname restart | ||
service netif restart | service netif restart | ||
Line 179: | Line 160: | ||
</ | </ | ||
- | ==== VM 5 (ECMP router) | + | ===== FRR Multipath setup ===== |
- | VM 5 is like VM 2, a router | + | Replacing static routes by FRR (OSPF) compiled |
+ | |||
+ | ==== R1 (ECMP router) ==== | ||
+ | |||
+ | In place of static | ||
< | < | ||
- | sysrc hostname=R5 | + | sysrc frr_vtysh_boot="YES" |
- | sysrc ifconfig_vtnet2="inet 10.0.35.5/ | + | |
- | sysrc ifconfig_vtnet3="inet 10.0.45.5/ | + | |
- | sysrc ifconfig_vtnet4="inet 10.0.56.5/ | + | |
- | sysrc ifconfig_vtnet2_ipv6="inet6 2001: | + | |
- | sysrc ifconfig_vtnet3_ipv6="inet6 2001: | + | |
- | sysrc ifconfig_vtnet4_ipv6=" | + | cat > / |
- | sysrc static_routes=" | + | frr version 8.4.1 |
- | sysrc route_R3R1=" | + | frr defaults traditional |
- | sysrc route_R3R2=" | + | hostname R1 |
- | sysrc route_R4R1=" | + | ! |
- | sysrc route_R4R2=" | + | interface igb0 |
- | sysrc ipv6_static_routes=" | + | ip ospf passive |
- | sysrc ipv6_route_R3R1=" | + | ipv6 ospf6 area 0.0.0.0 |
- | sysrc ipv6_route_R3R2=" | + | ipv6 ospf6 passive |
- | sysrc ipv6_route_R4R1=" | + | exit |
- | sysrc ipv6_route_R4R2=" | + | ! |
- | service hostname restart | + | interface igb1 |
- | service | + | ipv6 ospf6 area 0.0.0.0 |
- | service | + | exit |
- | config save | + | ! |
+ | interface igb2 | ||
+ | ipv6 ospf6 area 0.0.0.0 | ||
+ | exit | ||
+ | ! | ||
+ | router ospf | ||
+ | ospf router-id 1.1.1.1 | ||
+ | network | ||
+ | | ||
+ | network | ||
+ | exit | ||
+ | ! | ||
+ | router ospf6 | ||
+ | exit | ||
+ | ! | ||
+ | EOF | ||
+ | service | ||
+ | service | ||
</ | </ | ||
- | ==== Router 6 (server) ==== | + | ==== R2 (ECMP router) ==== |
- | R6, like R1 is configured | + | Same as R1 with OSPF and FRR: |
+ | < | ||
+ | sysrc frr_vtysh_boot=" | ||
+ | frr_enable=" | ||
+ | frr_daemons=" | ||
+ | watchfrr_flags=" | ||
+ | watchfrr_enable=" | ||
+ | cat > / | ||
+ | frr version 8.4.1 | ||
+ | frr defaults traditional | ||
+ | hostname R2 | ||
+ | ! | ||
+ | ip route 10.0.0.0/16 10.0.34.4 | ||
+ | ipv6 route 2001: | ||
+ | ! | ||
+ | interface igb0 | ||
+ | ip ospf passive | ||
+ | ipv6 ospf6 area 0.0.0.0 | ||
+ | ipv6 ospf6 passive | ||
+ | exit | ||
+ | ! | ||
+ | interface igb1 | ||
+ | ipv6 ospf6 area 0.0.0.0 | ||
+ | exit | ||
+ | ! | ||
+ | interface igb2 | ||
+ | ipv6 ospf6 area 0.0.0.0 | ||
+ | exit | ||
+ | ! | ||
+ | router ospf | ||
+ | ospf router-id 2.2.2.2 | ||
+ | | ||
+ | | ||
+ | | ||
+ | | ||
+ | exit | ||
+ | ! | ||
+ | router ospf6 | ||
+ | | ||
+ | exit | ||
+ | ! | ||
+ | EOF | ||
+ | service frr start | ||
+ | service watchfrr start | ||
+ | </ | ||
+ | |||
+ | ==== Checking routes installed ==== | ||
+ | |||
+ | On R1: | ||
< | < | ||
- | sysrc hostname=R6 | + | root@R1:~ # vtysh |
- | sysrc gateway_enable=NO | + | Hello, this is FRRouting (version 8.4.1). |
- | sysrc ipv6_gateway_enable=NO | + | Copyright 1996-2005 Kunihiro Ishiguro, et al. |
- | sysrc ifconfig_vtnet4=" | + | |
- | sysrc ifconfig_vtnet4_alias1=" | + | R1# sh ip route 10.0.0.0 |
- | sysrc ifconfig_vtnet4_alias2=" | + | Routing entry for 10.0.0.0/16 |
- | sysrc ifconfig_vtnet4_alias3=" | + | Known via "ospf", distance 110, metric 20, best |
- | sysrc ifconfig_vtnet4_alias4=" | + | Last update 00:02:26 ago |
- | sysrc ifconfig_vtnet4_alias5=" | + | * 10.0.231.3, via igb1, weight 1 |
- | sysrc ifconfig_vtnet4_ipv6=" | + | |
- | sysrc ifconfig_vtnet4_alias61=" | + | |
- | sysrc ifconfig_vtnet4_alias62="inet6 2001: | + | R1# sh ipv6 route 2001: |
- | sysrc ifconfig_vtnet4_alias63=" | + | Routing entry for 2001:db8::/32 |
- | sysrc ifconfig_vtnet4_alias64=" | + | Known via "ospf6", distance 110, metric 20, best |
- | sysrc ifconfig_vtnet4_alias65=" | + | Last update 00:02:39 ago |
- | sysrc defaultrouter=" | + | * fe80::20d:b9ff:fe45:7ad5, via igb1, weight 1 |
- | sysrc ipv6_defaultrouter=" | + | * fe80::20d:b9ff:fe45:7ad6, via igb2, weight 1 |
- | service hostname restart | + | |
- | service netif restart | + | |
- | service routing restart | + | |
- | config save | + | |
</ | </ | ||
- | ==== Testing | + | ===== Test Load balancing |
- | Flows from R1 to R6 should be load-balanced by R2 toward R3 and R4. Let's check using multiple sources and destination IP addresses: | + | Flows from the client |
- | ^ Source ^ Destination ^ Ongoing path selected by R2 ^ Returning path selected by R5^ | + | On server: |
- | | 10.0.12.1 | 10.0.56.6 | R3 | R3 | | + | < |
- | | 10.0.12.11 | 10.0.56.16 | R3 | R3 | | + | root@server: |
- | | 10.0.12.21 | 10.0.56.26 | R3 | R3 | | + | </ |
- | | 10.0.12.31 | 10.0.56.36 | R3 | R3 | | + | |
- | | 10.0.12.41 | 10.0.56.46 | R3 | R3 | | + | On client: |
- | | 10.0.12.51 | 10.0.56.56 | R3 | R3 | | + | < |
- | | 2001: | + | root@client: |
- | | 2001: | + | </ |
- | | 2001: | + | |
- | | 2001: | + | On R1: |
- | | 2001: | + | < |
- | | 2001: | + | systat -ifstat -match igb0, |
- | | 10.0.12.1 | 10.0.45.5 | R4 | R3 | | + | |
+ | /0 / | ||
+ | Load Average | ||
+ | |||
+ | Interface | ||
+ | | ||
+ | | ||
+ | |||
+ | | ||
+ | out 9.770 Kp/s 9.771 Kp/s 80.341 Mp | ||
+ | |||
+ | | ||
+ | out 0.000 Kp/s | ||
+ | |||
+ | </ | ||
+ | |||
+ | => We confirm that 20 Kps entering igb0 and are equally split by exiting by igb1 and igb2 | ||
+ | |||
+ | On R2: | ||
+ | < | ||
+ | systat -ifstat -match igb0, | ||
+ | |||
+ | /0 /1 /2 / | ||
+ | Load Average | ||
+ | |||
+ | | ||
+ | igb2 in 9.768 Kp/s 9.771 Kp/s 300.830 Kp | ||
+ | | ||
+ | |||
+ | igb1 in 9.763 Kp/s 9.768 Kp/s 300.785 Kp | ||
+ | | ||
+ | |||
+ | | ||
+ | | ||
+ | </ | ||
- | <note>There is no flow load-balancing with RADIX MPATH on FreeBSD 11, [[https:// | + | => R2 has no choice than receiving packets from igb1 and igb2, and forwarding them through igb0. |
documentation/examples/ecmp.txt · Last modified: 2024/03/28 23:19 by olivier