This lab shows an example of aggregating multiple independent ISP links with MLVPN.

Here is the concept:

And here is this lab detailed diagram:

This chapter will describe how to start each routers and configuring the 3 centrals routers.

More information on these BSDRP lab scripts available on How to build a BSDRP router lab.

Start the Virtual lab (example using bhyve):

# ./tools/BSDRP-lab-bhyve.sh -n 5
BSD Router Project (http://bsdrp.net) - bhyve full-meshed lab script
Setting-up a virtual lab with 5 VM(s):
- Working directory: /tmp/BSDRP
- Each VM have 1 core(s) and 256M RAM
- Switch mode: bridge + tap
- 0 LAN(s) between all VM
- Full mesh Ethernet links between each VM
VM 1 have the following NIC:
- vtnet0 connected to VM 2.
- vtnet1 connected to VM 3.
- vtnet2 connected to VM 4.
- vtnet3 connected to VM 5.
VM 2 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 3.
- vtnet2 connected to VM 4.
- vtnet3 connected to VM 5.
VM 3 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 4.
- vtnet3 connected to VM 5.
VM 4 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 3.
- vtnet3 connected to VM 5.
VM 5 have the following NIC:
- vtnet0 connected to VM 1.
- vtnet1 connected to VM 2.
- vtnet2 connected to VM 3.
- vtnet3 connected to VM 4.
For connecting to VM'serial console, you can use:
- VM 1 : cu -l /dev/nmdm1B
- VM 2 : cu -l /dev/nmdm2B
- VM 3 : cu -l /dev/nmdm3B
- VM 4 : cu -l /dev/nmdm4B
- VM 5 : cu -l /dev/nmdm5B

Router 2 is configured for rate-limiting traffic at 10 Mb/s on interface to/from R1.

sysrc hostname=R2
sysrc ifconfig_vtnet0="10.0.12.2/24"
sysrc ifconfig_vtnet2="10.0.24.2/24"
sysrc static_routes="R5"
sysrc route_R5="-net 10.0.45.0/24 10.0.24.4"
sysrc firewall_enable=YES
sysrc firewall_script="/etc/ipfw.rules"

cat > /etc/ipfw.rules <<'EOF'
#!/bin/sh
fwcmd="/sbin/ipfw"
kldstat -q -m dummynet || kldload dummynet
# Flush out the list before we begin.
${fwcmd} -f flush
#Create pipes (one for each direction)
${fwcmd} pipe 10 config bw 10Mbit/s
${fwcmd} pipe 20 config bw 10Mbit/s
#Traffic getting out vtnet0 is limited to 10Mbit/s
${fwcmd} add 1000 pipe 10 all from any to any out via vtnet0
#Traffic getting int vtnet0 is limited to 10Mbit/s
${fwcmd} add 2000 pipe 20 all from any to any in via vtnet0
#We don't want to block traffic, only shape some
${fwcmd} add 3000 allow ip from any to any
'EOF'

service netif restart
service routing restart
service ipfw start
config save

Router 3 is configured for rate-limiting traffic at 10 Mb/s on interface to/from R1.

sysrc hostname=R3
sysrc ifconfig_vtnet0="10.0.13.3/24"
sysrc ifconfig_vtnet2="10.0.34.3/24"
sysrc static_routes="R5"
sysrc route_R5="-net 10.0.45.0/24 10.0.34.4"
sysrc firewall_enable=YES
sysrc firewall_script="/etc/ipfw.rules"

cat > /etc/ipfw.rules <<'EOF'
#!/bin/sh
fwcmd="/sbin/ipfw"
kldstat -q -m dummynet || kldload dummynet
# Flush out the list before we begin.
${fwcmd} -f flush
#Create pipes (one for each direction)
${fwcmd} pipe 10 config bw 10Mbit/s
${fwcmd} pipe 20 config bw 10Mbit/s
#Traffic getting out vtnet0 is limited to 10Mbit/s
${fwcmd} add 1000 pipe 10 all from any to any out via vtnet0
#Traffic getting int vtnet0 is limited to 10Mbit/s
${fwcmd} add 2000 pipe 20 all from any to any in via vtnet0
#We don't want to block traffic, only shape some
${fwcmd} add 3000 allow ip from any to any
'EOF'

service netif restart
service routing restart
service ipfw start
config save

Router 4 is the aggregating server's default gateway.

sysrc hostname=R4
sysrc ifconfig_vtnet1="10.0.24.4/24"
sysrc ifconfig_vtnet2="10.0.34.4/24"
sysrc ifconfig_vtnet3="10.0.45.4/24"
sysrc static_routes="R2 R3"
sysrc route_R2="-net 10.0.12.0/24 10.0.24.2"
sysrc route_R3="-net 10.0.13.0/24 10.0.34.3"
service netif restart
service routing restart
config save

Router 1 is configured as a MLVPN client router connected to 3 different Internet links. The big difference with MLPPP: We can't use 3 differents IP addresses on our server, then can't simply install 3 differents static routes. We need a 3 default routes, then a minimum of 4 differents routing table.

sysrc hostname=R1
sysrc cloned_interfaces="lo1"
sysrc ifconfig_lo1="inet 10.1.1.1/32"
sysrc ifconfig_vtnet0="10.0.12.1/24 fib 2"
sysrc ifconfig_vtnet1="10.0.13.1/24 fib 3"
sysrc static_routes="ISP1 ISP2"
sysrc route_ISP1="-fib 2 default 10.0.12.2"
sysrc route_ISP2="-fib 3 default 10.0.13.3"
sysrc mlvpn_enable=YES

cat <<EOF > /usr/local/etc/mlvpn/mlvpn.conf
[general]
statuscommand = "/usr/local/etc/mlvpn/mlvpn_updown.sh"
mode = "client"
mtu = 1452
tuntap = "tun"
ip4 = "10.0.15.1/30"
ip4_gateway = "10.0.15.5"
ip4_routes = "10.5.5.5/32"
timeout = 30
password = "pleasechangeme!"
#reorder_buffer_size = 64
loss_tolerence = 10

[dsl2]
bindhost = "10.0.12.1"
bindport = 5082
bindfib = 2
remotehost = "10.0.45.5"
remoteport = 5082

[dsl3]
bindhost = "10.0.13.1"
bindport = 5083
bindfib = 3
remotehost = "10.0.45.5"
remoteport = 5083

EOF

service netif restart
service routing restart
service mlvpn start
config save

Router 5 is configured as a aggregating server.

sysrc hostname=R5
sysrc cloned_interfaces="lo1"
sysrc ifconfig_lo1="inet 10.5.5.5/32"
sysrc ifconfig_vtnet3="10.0.45.5/24"
sysrc defaultrouter=10.0.45.4
sysrc mlvpn_enable=YES

cat <<'EOF' > /usr/local/etc/mlvpn/mlvpn.conf
[general]
statuscommand = "/usr/local/etc/mlvpn/mlvpn_updown.sh"
tuntap = "tun"
mode = "server"
ip4 = "10.0.15.5/30"
ip4_gateway = "10.0.15.1"
ip4_routes = "10.1.1.1/32"
timeout = 30
password = "pleasechangeme!"
#reorder_buffer_size = 64
loss_tolerence = 10

[adsl2]
bindhost = "10.0.45.5"
bindport = 5082

[adsl3]
bindhost = "10.0.45.5"
bindport = 5083
'EOF'

service netif restart
service routing restart
service mlvpn start
config save

Start by checking that R5 is reacheable from each R1's fib (2, 3):

[root@VM1]~# setfib 2 ping -c 2 10.0.56.6
PING 10.0.56.6 (10.0.56.6): 56 data bytes
64 bytes from 10.0.56.6: icmp_seq=0 ttl=62 time=16.473 ms
64 bytes from 10.0.56.6: icmp_seq=1 ttl=62 time=20.017 ms

--- 10.0.56.6 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.473/18.245/20.017/1.772 ms
[root@VM1]~# setfib 3 ping -c 2 10.0.56.6
PING 10.0.56.6 (10.0.56.6): 56 data bytes
64 bytes from 10.0.56.6: icmp_seq=0 ttl=62 time=18.202 ms
64 bytes from 10.0.56.6: icmp_seq=1 ttl=62 time=11.193 ms

--- 10.0.56.6 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 11.193/14.698/18.202/3.504 ms
[root@VM1]~# setfib 4 ping -c 2 10.0.56.6
PING 10.0.56.6 (10.0.56.6): 56 data bytes
64 bytes from 10.0.56.6: icmp_seq=0 ttl=62 time=10.973 ms
64 bytes from 10.0.56.6: icmp_seq=1 ttl=62 time=14.465 ms

--- 10.0.56.6 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.973/12.719/14.465/1.746 ms

Test bandwidth of each link by starting an iperf on MLVPN server:

[root@VM6]# iperf3 -s

Then from the MLVPN client, test bandwidth for each ISP links:

[root@VM1]~# setfib 2 iperf -c 10.0.56.6
------------------------------------------------------------
Client connecting to 10.0.45.5, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 10.0.12.1 port 59888 connected with 10.0.45.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  11.8 MBytes  9.75 Mbits/sec

[root@R1]~# setfib 3 iperf -c 10.0.45.5
------------------------------------------------------------
Client connecting to 10.0.45.5, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 10.0.13.1 port 53380 connected with 10.0.45.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  11.8 MBytes  9.75 Mbits/sec

MLVPN can be started in debug mode:

[root@R1]# mlvpn --debug -n mlvpn -u mlvpn
2016-04-19T23:48:21 [INFO/config] new password set
2016-04-19T23:48:21 [INFO/config] dsl2 tunnel added
2016-04-19T23:48:21 [INFO/config] dsl3 tunnel added
2016-04-19T23:48:21 [INFO] created interface `tun0'
2016-04-19T23:48:21 [INFO] dsl2 bind to 10.0.12.1
2016-04-19T23:48:21 [INFO] dsl3 bind to 10.0.13.1
2016-04-19T23:48:21 [INFO/protocol] dsl3 authenticated
2016-04-19T23:48:21 [INFO/protocol] dsl2 authenticated

tun interface need to be check (correct IP address and non-1500 MTU):

[root@R1]# ifconfig tun0
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1452
        options=80000<LINKSTATE>
        inet6 fe80::5a9c:fcff:fe01:201%tun0 prefixlen 64 scopeid 0x7
        inet 10.0.15.1 --> 10.0.15.5 netmask 0xfffffffc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 2326

And static route(s) needs to be installed (10.5.5.5/32 in this example):

[root@R1]~# netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
10.0.12.0/24       link#1             U        vtnet0
10.0.13.0/24       link#2             U        vtnet1
10.0.15.1          link#7             UHS         lo0
10.0.15.5          link#7             UH         tun0
10.5.5.5/32        10.0.15.5          UGS        tun0
127.0.0.1          link#6             UH          lo0

Check that aggregated bandwitdh is 10+10 = 20Mbit/s on this lab.

[root@R1]# iperf --bind 10.1.1.1 -c 10.5.5.5 -t 60
------------------------------------------------------------
Client connecting to 10.5.5.5, TCP port 5001
Binding to local address 10.1.1.1
TCP window size: 32.3 KByte (default)
------------------------------------------------------------
[  3] local 10.1.1.1 port 5001 connected with 10.5.5.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   129 MBytes  18.1 Mbits/sec