Differences

This shows you the differences between two versions of the page.

--- documentation:examples:ecmp [2019/11/01 08:49] – [VM 2 (ECMP router)] olivier
+++ documentation:examples:ecmp [2025/01/17 18:53] (current) – [R2 (ECMP router)] olivier
@@ Line 1: / Line 1: @@
 ====== Equal-cost multi-path routing (ECMP) ======
-This lab show an ECMP (feature called RADIX_MPATH on FreeBSD) routing example.
 ===== Presentation =====
+<note warning>Bhyve doesn't support emulating multiqueue NIC, so RSS flow-id could not be tested using a bhyve based lab: Had to use physical lab</note>
 ==== Network diagram ====
 Here is the logical and physical view:
-{{:documentation:examples:labs.examples.ecmp.png|}}
+{{:documentation:examples:bsdrp-lab-mpath.png|}}
 ===== Setting-up the lab =====
@@ Line 15: / Line 14: @@
 ==== Downloading BSD Router Project images ====
-Download BSDRP serial image (prevent to have to use an X display) on Sourceforge.
+Download BSDRP serial image on Sourceforge and upload them to the 2 ECMP routers.
-==== Download Lab scripts =====
-More information on these BSDRP lab scripts available on [[documentation:examples:How to build a BSDRP router lab]].
-Start the lab with 6 routers.
-With bhyve under FreeBSD:
-<code>
-user:~ # BSDRP-lab-bhyve.sh -i BSDRP.amd64/BSDRP-1.71-full-amd64-serial.img.xz -n 6
-BSD Router Project (http://bsdrp.net) - bhyve full-meshed lab script
-Setting-up a virtual lab with 6 VM(s):
-- Working directory: /tmp/BSDRP
-- Each VM have 1 core(s) and 256M RAM
-- Switch mode: bridge + tap
-- 0 LAN(s) between all VM
-- Full mesh Ethernet links between each VM
-VM 1 have the following NIC:
-- vtnet0 connected to VM 2.
-- vtnet1 connected to VM 3.
-- vtnet2 connected to VM 4.
-- vtnet3 connected to VM 5.
-- vtnet4 connected to VM 6.
-VM 2 have the following NIC:
-- vtnet0 connected to VM 1.
-- vtnet1 connected to VM 3.
-- vtnet2 connected to VM 4.
-- vtnet3 connected to VM 5.
-- vtnet4 connected to VM 6.
-VM 3 have the following NIC:
-- vtnet0 connected to VM 1.
-- vtnet1 connected to VM 2.
-- vtnet2 connected to VM 4.
-- vtnet3 connected to VM 5.
-- vtnet4 connected to VM 6.
-VM 4 have the following NIC:
-- vtnet0 connected to VM 1.
-- vtnet1 connected to VM 2.
-- vtnet2 connected to VM 3.
-- vtnet3 connected to VM 5.
-- vtnet4 connected to VM 6.
-VM 5 have the following NIC:
-- vtnet0 connected to VM 1.
-- vtnet1 connected to VM 2.
-- vtnet2 connected to VM 3.
-- vtnet3 connected to VM 4.
-- vtnet4 connected to VM 6.
-VM 6 have the following NIC:
-- vtnet0 connected to VM 1.
-- vtnet1 connected to VM 2.
-- vtnet2 connected to VM 3.
-- vtnet3 connected to VM 4.
-- vtnet4 connected to VM 5.
-For connecting to VM'serial console, you can use:
-- VM 1 : cu -l /dev/nmdm1B
-- VM 2 : cu -l /dev/nmdm2B
-- VM 3 : cu -l /dev/nmdm3B
-- VM 4 : cu -l /dev/nmdm4B
-- VM 5 : cu -l /dev/nmdm5B
-- VM 6 : cu -l /dev/nmdm6B
-</code>
 ===== Static routing setup =====
-==== VM 1 (client) ====
+==== Client ====
-VM1 is configured as a simple host with multiples aliases:
+A simple host with static routes:
 <code>
-sysrc hostname=VM1
+sysrc hostname=client \
-sysrc gateway_enable=NO
+  gateway_enable=NO \
-sysrc ipv6_gateway_enable=NO
+  ipv6_gateway_enable=NO \
-sysrc ifconfig_vtnet0="inet 10.0.12.1/24"
+  ifconfig_igb1="inet 10.0.12.1/24" \
-sysrc ifconfig_vtnet0_alias1="inet 10.0.12.11/32"
+  ifconfig_igb1_ipv6="inet6 2001:db8:12::3 prefixlen 64" \
-sysrc ifconfig_vtnet0_alias2="inet 10.0.12.21/32"
+  route_LAB="-net 10.0.0.0/16 10.0.12.2" \
-sysrc ifconfig_vtnet0_alias3="inet 10.0.12.31/32"
+  static_routes="LAB" \
-sysrc ifconfig_vtnet0_alias4="inet 10.0.12.41/32"
+  ipv6_static_routes="LAB" \
-sysrc ifconfig_vtnet0_alias5="inet 10.0.12.51/32"
+  ipv6_route_LAB="2001:db8:: -prefixlen 32 2001:db8:12::2"
-sysrc ifconfig_vtnet0_ipv6="inet6 2001:db8:12::1 prefixlen 64"
-sysrc ifconfig_vtnet0_alias61="inet6 2001:db8:12::11 prefixlen 64"
-sysrc ifconfig_vtnet0_alias62="inet6 2001:db8:12::21 prefixlen 64"
-sysrc ifconfig_vtnet0_alias63="inet6 2001:db8:12::31 prefixlen 64"
-sysrc ifconfig_vtnet0_alias64="inet6 2001:db8:12::41 prefixlen 64"
-sysrc ifconfig_vtnet0_alias65="inet6 2001:db8:12::51 prefixlen 64"
-sysrc defaultrouter="10.0.12.2"
-sysrc ipv6_defaultrouter="2001:db8:12::2"
 service hostname restart
 service netif restart
@@ Line 107: / Line 39: @@
 </code>
-==== VM 2 (ECMP router) ====
+==== R1 (ECMP router) ====
-VM 2 is a router with ECMP: 2 static routes toward the same destination but using 2 different next-hop.
+R1 is a router with ECMP: 2 static routes toward the same destination but using 2 different next-hop.
-Need to disable fastforward by re-enabling ICMP redirect (disabled on BSDRP by default).
 <code>
-sysrc hostname=VM2
-sysrc ifconfig_vtnet0="inet 10.0.12.2/24"
+sysrc hostname=R1 \
-sysrc ifconfig_vtnet1="inet 10.0.23.2/24"
+  gateway_enable=YES \
-sysrc ifconfig_vtnet2="inet 10.0.24.2/24"
+  ipv6_gateway_enable=YES \
-sysrc ifconfig_vtnet0_ipv6="inet6 2001:db8:12::2 prefixlen 64"
+  ifconfig_igb0="inet 10.0.12.2/24" \
-sysrc ifconfig_vtnet1_ipv6="inet6 2001:db8:23::2 prefixlen 64"
+  ifconfig_igb0_ipv6="inet6 2001:db8:12::2 prefixlen 64" \
-sysrc ifconfig_vtnet2_ipv6="inet6 2001:db8:24::2 prefixlen 64"
+  ifconfig_igb1="inet 10.0.231.2/24" \
-sysrc static_routes="VM3VM6 VM3VM5 VM4VM6 VM4VM5"
+  ifconfig_igb1_ipv6="inet6 2001:db8:231::2 prefixlen 64" \
-sysrc route_VM3VM6="-net 10.0.56.0/24 10.0.23.3"
+  ifconfig_igb2="inet 10.0.232.2/24" \
-sysrc route_VM3VM5="-net 10.0.35.0/24 10.0.23.3"
+  ifconfig_igb2_ipv6="inet6 2001:db8:232::2 prefixlen 64" \
-sysrc route_VM4VM6="-net 10.0.56.0/24 10.0.24.4"
+  static_routes="MPATH1 MPATH2" \
-sysrc route_VM4VM5="-net 10.0.45.0/24 10.0.24.4"
+  route_MPATH1="-net 10.0.0.0/16 10.0.231.3" \
-sysrc ipv6_static_routes="VM3VM6 VM3VM5 VM4VM6 VM4VM5"
+  route_MPATH2="-net 10.0.0.0/16 10.0.232.3" \
-sysrc ipv6_route_VM3VM6="2001:db8:56:: -prefixlen 64 2001:db8:23::3"
+  ipv6_static_routes="MPATH1 MPATH2" \
-sysrc ipv6_route_VM3VM5="2001:db8:35:: -prefixlen 64 2001:db8:23::3"
+  ipv6_route_MPATH1="2001:db8:: -prefixlen 32 2001:db8:231::3" \
-sysrc ipv6_route_VM4VM6="2001:db8:56:: -prefixlen 64 2001:db8:24::4"
+  ipv6_route_MPATH2="2001:db8:: -prefixlen 32 2001:db8:232::3"
-sysrc ipv6_route_VM4VM5="2001:db8:45:: -prefixlen 64 2001:db8:24::4"
-sed -i '' -e '/net.inet.ip.redirect=0/d' /etc/sysctl.conf
-sed -i '' -e '/net.inet6.ip6.redirect=0/d' /etc/sysctl.conf
-sysctl net.inet.ip.redirect=1
-sysctl net.inet6.ip6.redirect=1
 service hostname restart
 service netif restart
@@ Line 140: / Line 66: @@
 </code>
-And first problem using static route:
+Checking static route with multiple next-hop:
 <code>
-add net 10.0.56.0: gateway 10.0.23.3 fib 0
+root@R1:~ # netstat -rn4 | grep 10.0.0.0/16
-add net 10.0.35.0: gateway 10.0.23.3 fib 0
+.0.0.0/16        10.0.231.3         UGS        igb1
-add net 10.0.56.0: gateway 10.0.24.4 fib 0: route already in table
+.0.0.0/16        10.0.232.3         UGS        igb2
-add net 10.0.45.0: gateway 10.0.24.4 fib 0
-add net 2001:db8:56::: gateway 2001:db8:23::3 fib 0
+root@R1:~ # netstat -4onW
-add net 2001:db8:35::: gateway 2001:db8:23::3 fib 0
+Nexthop data
-add net 2001:db8:56::: gateway 2001:db8:24::4 fib 0: route already in table
-add net 2001:db8:45::: gateway 2001:db8:24::4 fib 0
+Internet:
-[root@VM2]~# route get 10.0.56.0/24
+Idx   Type         IFA                Gateway             Flags      Use Mtu         Netif     Addrif Refcnt Prepend
-   route to: 10.0.56.0
+       v4/resolve 127.0.0.1          lo0/resolve        H             0  16384        lo0               2
-destination: 10.0.56.0
+       v4/resolve 10.0.12.2          igb0/resolve                     0   1500       igb0               2
-       mask: 255.255.255.0
+       v4/resolve 127.0.0.1          lo0/resolve        HS            0  16384        lo0      igb0     2
-    gateway: 10.0.23.3
+       v4/resolve 10.0.231.2         igb1/resolve                     0   1500       igb1               2
-        fib: 0
+       v4/resolve 127.0.0.1          lo0/resolve        HS            0  16384        lo0      igb1     2
-  interface: vtnet1
+       v4/resolve 10.0.232.2         igb2/resolve                     0   1500       igb2               2
-      flags: <UP,GATEWAY,DONE,STATIC>
+       v4/resolve 127.0.0.1          lo0/resolve        HS            0  16384        lo0      igb2     2
- recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
+            v4/gw 10.0.231.2         10.0.231.3         GS            0   1500       igb1               1
-         0         0         0      1500         1         0
+            v4/gw 10.0.232.2         10.0.232.3         GS            0   1500       igb2               1
+root@R1:~ # netstat -6onW
+Nexthop data
+Internet6:
+Idx   Type         IFA                           Gateway                        Flags      Use Mtu       Netif   Addrif Refcnt Prepend
+       v6/resolve ::1                           lo0/resolve                   HS            0  16384      lo0             2
+       v6/resolve fe80::1%lo0                   lo0/resolve                   HS            0  16384      lo0             2
+       v6/resolve fe80::1%lo0                   lo0/resolve                                 0  16384      lo0             2
+       v6/resolve ::1                           lo0/resolve                   HS            0  16384      lo0    igb0     3
+       v6/resolve fe80::20d:b9ff:fe41:ca3c%igb0 igb0/resolve                                0   1500     igb0             3
+       v6/resolve ::1                           lo0/resolve                   HS            0  16384      lo0    igb1     3
+       v6/resolve fe80::20d:b9ff:fe41:ca3d%igb1 igb1/resolve                                0   1500     igb1             3
+       v6/resolve ::1                           lo0/resolve                   HS            0  16384      lo0    igb2     3
+       v6/resolve fe80::20d:b9ff:fe41:ca3e%igb2 igb2/resolve                                0   1500     igb2             3
+           v6/gw ::1                           ::1                           GRS           0  16384      lo0             5
+           v6/gw 2001:db8:231::2               2001:db8:231::3               GS            0   1500     igb1             1
+           v6/gw 2001:db8:232::2               2001:db8:232::3               GS            0   1500     igb2             1
 </code>
-==== VM 3 (router) ====
+==== R2 (ECMP router) ====
-VM 3 is a simple router.
+R2 is like R1, a router with ECMP: 2 static routing toward the same destination but using 2 different next-hop..
 <code>
-sysrc hostname=VM3
+sysrc hostname=R2 \
-sysrc ifconfig_vtnet1="inet 10.0.23.3/24"
+  gateway_enable=YES \
-sysrc ifconfig_vtnet3="inet 10.0.35.3/24"
+  ipv6_gateway_enable=YES \
-sysrc ifconfig_vtnet1_ipv6="inet6 2001:db8:23::3 prefixlen 64"
+  ifconfig_igb0="inet 10.0.34.3/24" \
-sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:35::3 prefixlen 64"
+  ifconfig_igb0_ipv6="inet6 2001:db8:34::3 prefixlen 64" \
-sysrc static_routes="VM6 VM1"
+  ifconfig_igb1="inet 10.0.231.3/24" \
-sysrc route_VM1="-net 10.0.12.0/24 10.0.23.2"
+  ifconfig_igb1_ipv6="inet6 2001:db8:231::3 prefixlen 64" \
-sysrc route_VM6="-net 10.0.56.0/24 10.0.35.5"
+  ifconfig_igb2="inet 10.0.232.3/24" \
-sysrc ipv6_static_routes="VM6 VM1"
+  ifconfig_igb2_ipv6="inet6 2001:db8:232::3 prefixlen 64" \
-sysrc ipv6_route_VM1="2001:db8:12:: -prefixlen 64 2001:db8:23::2"
+  static_routes="MPATH1 MPATH2" \
-sysrc ipv6_route_VM6="2001:db8:56:: -prefixlen 64 2001:db8:35::5"
+  route_MPATH1="-net 10.0.0.0/16 10.0.231.4" \
+  route_MPATH2="-net 10.0.0.0/16 10.0.232.4" \
+  ipv6_static_routes="MPATH1 MPATH2" \
+  ipv6_route_MPATH1="2001:db8:: -prefixlen 32 2001:db8:231::4" \
+  ipv6_route_MPATH2="2001:db8:: -prefixlen 32 2001:db8:232::4"
 service hostname restart
 service netif restart
@@ Line 183: / Line 131: @@
 config save
 </code>
+==== Server ====
-==== VM 4 (router) ====
+A simple host with some static routes:
-VM 4 is like VM 3, a simple router.
 <code>
-sysrc hostname=VM4
+sysrc hostname=server \
-sysrc ifconfig_vtnet1="inet 10.0.24.4/24"
+  gateway_enable=NO \
-sysrc ifconfig_vtnet3="inet 10.0.45.4/24"
+  ipv6_gateway_enable=NO \
-sysrc ifconfig_vtnet1_ipv6="inet6 2001:db8:24::4 prefixlen 64"
+  ifconfig_igb1="inet 10.0.34.4/24" \
-sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:45::4 prefixlen 64"
+  ifconfig_igb1_ipv6="inet6 2001:db8:34::4 prefixlen 64" \
-sysrc static_routes="VM6 VM1"
+  static_routes="LAB" \
-sysrc route_VM1="-net 10.0.12.0/24 10.0.24.2"
+  route_LAB="-net 10.0.0.0/16 10.0.34.3" \
-sysrc route_VM6="-net 10.0.56.0/24 10.0.45.5"
+  ipv6_static_routes="LAB" \
-sysrc ipv6_static_routes="VM6 VM1"
+  ipv6_route_LAB="2001:db8:: -prefixlen 32 2001:db8:34::3"
-sysrc ipv6_route_VM1="2001:db8:12:: -prefixlen 64 2001:db8:24::2"
-sysrc ipv6_route_VM6="2001:db8:56:: -prefixlen 64 2001:db8:45::5"
 service hostname restart
 service netif restart
@@ Line 206: / Line 151: @@
 </code>
-==== VM 5 (ECMP router) ====
+===== FRR Multipath setup =====
-VM 5 is like VM 2, a router with ECMP: 2 static routing toward the same destination but using 2 different next-hop..
+Replacing static routes by FRR (OSPF) compiled with MULTIPATH option.
+==== R1 (ECMP router) ====
+In place of static routes, OSPF with FRR is used:
 <code>
-sysrc hostname=VM5
+sysrc frr_vtysh_boot="YES" \
-sysrc ifconfig_vtnet2="inet 10.0.35.5/24"
+  frr_enable="YES" \
-sysrc ifconfig_vtnet3="inet 10.0.45.5/24"
+  frr_daemons="zebra ospfd ospf6d" \
-sysrc ifconfig_vtnet4="inet 10.0.56.5/24"
+  watchfrr_flags=" -d -r /usr/sbin/servicebBfrrbBrestartbB%s -s /usr/sbin/servicebBfrrbBstartbB%s -k /usr/sbin/servicebBfrrbBstopbB%s -b bB -t 30 zebra ospfd ospf6d" \
-sysrc ifconfig_vtnet2_ipv6="inet6 2001:db8:35::5 prefixlen 64"
+  watchfrr_enable="YES"
-sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:45::5 prefixlen 64"
-sysrc ifconfig_vtnet4_ipv6="inet6 2001:db8:56::5 prefixlen 64"
+cat > /usr/local/etc/frr/frr.conf <EOF
-sysrc static_routes="VM3VM1 VM3VM2 VM4VM1 VM4VM2"
+frr version 8.4.1
-sysrc route_VM3VM1="-net 10.0.12.0/24 10.0.35.3"
+frr defaults traditional
-sysrc route_VM3VM2="-net 10.0.23.0/24 10.0.35.3"
+hostname R1
-sysrc route_VM4VM1="-net 10.0.12.0/24 10.0.45.4"
+!
-sysrc route_VM4VM2="-net 10.0.24.0/24 10.0.45.4"
+interface igb0
-sysrc ipv6_static_routes="VM3VM1 VM3VM2 VM4VM1 VM4VM2"
+ ip ospf passive
-sysrc ipv6_route_VM3VM1="2001:db8:12:: -prefixlen 64 2001:db8:35::3"
+ ipv6 ospf6 area 0.0.0.0
-sysrc ipv6_route_VM3VM2="2001:db8:23:: -prefixlen 64 2001:db8:35::3"
+ ipv6 ospf6 passive
-sysrc ipv6_route_VM4VM1="2001:db8:12:: -prefixlen 64 2001:db8:45::4"
+exit
-sysrc ipv6_route_VM4VM2="2001:db8:24:: -prefixlen 64 2001:db8:45::4"
+!
-sed -i '' -e '/net.inet.ip.redirect=0/d' /etc/sysctl.conf
+interface igb1
-sed -i '' -e '/net.inet6.ip6.redirect=0/d' /etc/sysctl.conf
+ ipv6 ospf6 area 0.0.0.0
-sysctl net.inet.ip.redirect=1
+exit
-sysctl net.inet6.ip6.redirect=1
+!
-service hostname restart
+interface igb2
-service netif restart
+ ipv6 ospf6 area 0.0.0.0
-service routing restart
+exit
-config save
+!
+router ospf
+ ospf router-id 1.1.1.1
+ network 10.0.12.0/24 area 0
+ network 10.0.231.0/24 area 0
+ network 10.0.232.0/24 area 0
+exit
+!
+router ospf6
+exit
+!
+EOF
+service frr start
+service watchfrr start
 </code>
-==== VM 6 (server) ====
+==== R2 (ECMP router) ====
-VM 6, like VM 1 is configured as a simple host with multiple aliases:
+Same as R1 with OSPF and FRR:
+<code>
+sysrc frr_vtysh_boot="YES" \
+  frr_enable="YES" \
+  frr_daemons="zebra staticd ospfd ospf6d" \
+  watchfrr_flags=" -d -r /usr/sbin/servicebBfrrbBrestartbB%s -s /usr/sbin/servicebBfrrbBstartbB%s -k /usr/sbin/servicebBfrrbBstopbB%s -b bB -t 30 zebra ospfd ospf6d" \
+  watchfrr_enable="YES"
+cat > /usr/local/etc/frr/frr.conf <EOF
+frr version 8.4.1
+frr defaults traditional
+hostname R2
+!
+ip route 10.0.0.0/16 10.0.34.4
+ipv6 route 2001:db8::/32 2001:db8:231::4
+!
+interface igb0
+ ip ospf passive
+ ipv6 ospf6 area 0.0.0.0
+ ipv6 ospf6 passive
+exit
+!
+interface igb1
+ ipv6 ospf6 area 0.0.0.0
+exit
+!
+interface igb2
+ ipv6 ospf6 area 0.0.0.0
+exit
+!
+router ospf
+ ospf router-id 2.2.2.2
+ redistribute static
+ network 10.0.34.0/24 area 0
+ network 10.0.231.0/24 area 0
+ network 10.0.232.0/24 area 0
+exit
+!
+router ospf6
+ redistribute static
+exit
+!
+EOF
+service frr start
+service watchfrr start
+</code>
+==== Checking routes installed ====
+On R1:
 <code>
-sysrc hostname=VM6
+root@R1:~ # vtysh
-sysrc gateway_enable=NO
+Hello, this is FRRouting (version 8.4.1).
-sysrc ipv6_gateway_enable=NO
+Copyright 1996-2005 Kunihiro Ishiguro, et al.
-sysrc ifconfig_vtnet4="inet 10.0.56.6/24"
-sysrc ifconfig_vtnet4_alias1="inet 10.0.56.16/32"
+R1# sh ip route 10.0.0.0
-sysrc ifconfig_vtnet4_alias2="inet 10.0.56.26/32"
+Routing entry for 10.0.0.0/16
-sysrc ifconfig_vtnet4_alias3="inet 10.0.56.36/32"
+  Known via "ospf", distance 110, metric 20, best
-sysrc ifconfig_vtnet4_alias4="inet 10.0.56.46/32"
+  Last update 00:02:26 ago
-sysrc ifconfig_vtnet4_alias5="inet 10.0.56.56/32"
+  * 10.0.231.3, via igb1, weight 1
-sysrc ifconfig_vtnet4_ipv6="inet6 2001:db8:56::6 prefixlen 64"
+  * 10.0.232.3, via igb2, weight 1
-sysrc ifconfig_vtnet4_alias61="inet6 2001:db8:56::16 prefixlen 64"
-sysrc ifconfig_vtnet4_alias62="inet6 2001:db8:56::26 prefixlen 64"
+R1# sh ipv6 route 2001:db8::
-sysrc ifconfig_vtnet4_alias63="inet6 2001:db8:56::36 prefixlen 64"
+Routing entry for 2001:db8::/32
-sysrc ifconfig_vtnet4_alias64="inet6 2001:db8:56::46 prefixlen 64"
+  Known via "ospf6", distance 110, metric 20, best
-sysrc ifconfig_vtnet4_alias65="inet6 2001:db8:56::56 prefixlen 64"
+  Last update 00:02:39 ago
-sysrc defaultrouter="10.0.56.5"
+  * fe80::20d:b9ff:fe45:7ad5, via igb1, weight 1
-sysrc ipv6_defaultrouter="2001:db8:56::5"
+  * fe80::20d:b9ff:fe45:7ad6, via igb2, weight 1
-service hostname restart
-service netif restart
-service routing restart
-config save
 </code>
-==== Testing Load balancing ====
+===== Test Load balancing IP packets =====
-Flows from VM1 to VM6 should be load-balanced by VM2 toward VM3 and VM4. Let's check using multiple sources and destination IP addresses.
+Flows from the client to the server should be "flow-id shared" between the 2 paths. Let's check using multiple sources and destination IP addresses using pkt-gen on client and server, then using systat on R1 and R2 to check their load-distribution.
-Using ping record feature like with this example:
+On server:
 <code>
-[root@VM1]~# ping -c1 -R -S 10.0.12.21 10.0.56.26
+root@server:~ # pkt-gen -i igb1 -f rx
-PING 10.0.56.26 (10.0.56.26) from 10.0.12.21: 56 data bytes
+</code>
-bytes from 10.0.56.26: icmp_seq=0 ttl=61 time=0.358 ms
-RR:     10.0.23.2
-.0.35.3
-.0.56.5
-.0.56.26
-.0.35.5
-.0.23.3
-.0.12.2
-.0.12.21
---- 10.0.56.26 ping statistics ---
+On client:
-packets transmitted, 1 packets received, 0.0% packet loss
+<code>
-round-trip min/avg/max/stddev = 0.358/0.358/0.358/0.000 ms
+root@client:~ # pkt-gen -i igb1 -f tx -n 8000000 -l 60 -d 10.0.255.1:2000-10.0.255.254 -D 00:0d:b9:41:ca:3c -s 10.0.254.1:2000-10.0.254.254 -S 00:0d:b9:45:7f:b0 -w 4 -R 20000
 </code>
-^ Source ^ Destination ^ Ongoing path selected by VM2 ^ Returning path selected by VM5^
+On R1:
-| 10.0.12.1 | 10.0.56.6 | VM3 | VM3 |
+<code>
-| 10.0.12.11 | 10.0.56.16 | VM3 | VM3 |
+systat -ifstat -match igb0,igb1,igb2 -pps
-| 10.0.12.21 | 10.0.56.26 | VM3 | VM3 |
-| 10.0.12.31 | 10.0.56.36 | VM3 | VM3 |
+                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
-| 10.0.12.41 | 10.0.56.46 | VM3 | VM3 |
+     Load Average   |
-| 10.0.12.51 | 10.0.56.56 | VM3 | VM3 |
-| 2001:db8:12::1 | 2001:db8:56::6 | VM3 | VM3 |
+      Interface           Traffic               Peak                Total
-| 2001:db8:12::11 | 2001:db8:56::16 | VM3 | VM3 |
+           igb2  in      0.000 Kp/s          0.000 Kp/s           71.247 Mp
-| 2001:db8:12::21 | 2001:db8:56::26 | VM3 | VM3 |
+                 out     9.762 Kp/s          9.777 Kp/s           76.892 Mp
-| 2001:db8:12::31 | 2001:db8:56::36 | VM3 | VM3 |
-| 2001:db8:12::41 | 2001:db8:56::46 | VM3 | VM3 |
+           igb1  in      0.000 Kp/s          0.000 Kp/s           71.392 Mp
-| 2001:db8:12::51 | 2001:db8:56::56 | VM3 | VM3 |
+                 out     9.770 Kp/s          9.771 Kp/s           80.341 Mp
-| 10.0.12.1 | 10.0.45.5 | VM4 | VM3 |
+           igb0  in     19.533 Kp/s         19.534 Kp/s           90.007 Mp
+                 out     0.000 Kp/s          0.000 Kp/s            0.243 Kp
+</code>
+=> We confirm that 20 Kps entering igb0 and are equally split by exiting by igb1 and igb2
+On R2:
+<code>
+systat -ifstat -match igb0,igb1,igb2 -pps
+                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
+     Load Average   |
+      Interface           Traffic               Peak                Total
+           igb2  in      9.768 Kp/s          9.771 Kp/s          300.830 Kp
+                 out     0.000 Kp/s          0.000 Kp/s            0.000 Kp
+           igb1  in      9.763 Kp/s          9.768 Kp/s          300.785 Kp
+                 out     0.000 Kp/s          0.000 Kp/s            0.006 Kp
+           igb0  in      0.000 Kp/s          0.001 Kp/s            0.240 Kp
+                 out    19.530 Kp/s         19.531 Kp/s          601.615 Kp
+</code>
-<note>There is no flow load-balancing with RADIX MPATH since FreeBSD 11, [[https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225792|seems broken with tryforward]], put proposed patch disable tryforward(fastforward) if RADIX_MPATH is enabled.</note>
+=> R2 has no choice than receiving packets from igb1 and igb2, and forwarding them through igb0.