User Tools

Site Tools


documentation:examples:aggregating_multiple_isp_links

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

documentation:examples:aggregating_multiple_isp_links [2018/11/12 06:09] (current)
Line 1: Line 1:
 +====== Aggregating multiple ISP links with ML PPP ======
  
 +This lab shows an example of aggregating multiple independent ISP links with [[http://​mpd.sourceforge.net/​ | MPD5]].
 +
 +===== Network diagram ====
 +
 +Here is the concept:
 +
 +{{:​documentation:​examples:​bsdrp.lab.aggregate.isp.links.concept.png|}}
 +
 +And here is this lab detailed diagram:
 +
 +{{:​documentation:​examples:​bsdrp.lab.aggregate.isp.links.detail.png|}}
 +
 +===== Virtual Lab =====
 +
 +This chapter will describe how to start each routers and configuring the 3 central routers.
 +
 +More information on these BSDRP lab scripts available on [[documentation:​examples:​How to build a BSDRP router lab]].
 +
 +Start the Virtual lab (example using bhyve):
 +
 +<​code>​
 +# ./​tools/​BSDRP-lab-bhyve.sh -n 6
 +Setting-up a virtual lab with 6 VM(s):
 +- Working directory: /tmp/BSDRP
 +- Each VM has 1 core(s) and 512M RAM
 +- Emulated NIC: virtio-net
 +- Switch mode: bridge + tap
 +- 0 LAN(s) between all VM
 +- Full mesh Ethernet links between each VM
 +VM 1 has the following NIC:
 +- vtnet0 connected to VM 2
 +- vtnet1 connected to VM 3
 +- vtnet2 connected to VM 4
 +- vtnet3 connected to VM 5
 +- vtnet4 connected to VM 6
 +VM 2 has the following NIC:
 +- vtnet0 connected to VM 1
 +- vtnet1 connected to VM 3
 +- vtnet2 connected to VM 4
 +- vtnet3 connected to VM 5
 +- vtnet4 connected to VM 6
 +VM 3 has the following NIC:
 +- vtnet0 connected to VM 1
 +- vtnet1 connected to VM 2
 +- vtnet2 connected to VM 4
 +- vtnet3 connected to VM 5
 +- vtnet4 connected to VM 6
 +VM 4 has the following NIC:
 +- vtnet0 connected to VM 1
 +- vtnet1 connected to VM 2
 +- vtnet2 connected to VM 3
 +- vtnet3 connected to VM 5
 +- vtnet4 connected to VM 6
 +VM 5 has the following NIC:
 +- vtnet0 connected to VM 1
 +- vtnet1 connected to VM 2
 +- vtnet2 connected to VM 3
 +- vtnet3 connected to VM 4
 +- vtnet4 connected to VM 6
 +VM 6 has the following NIC:
 +- vtnet0 connected to VM 1
 +- vtnet1 connected to VM 2
 +- vtnet2 connected to VM 3
 +- vtnet3 connected to VM 4
 +- vtnet4 connected to VM 5
 +For connecting to VM'​serial console, you can use:
 +- VM 1 : cu -l /dev/nmdm1B
 +- VM 2 : cu -l /dev/nmdm2B
 +- VM 4 : cu -l /dev/nmdm4B
 +- VM 3 : cu -l /dev/nmdm3B
 +- VM 5 : cu -l /dev/nmdm5B
 +- VM 6 : cu -l /dev/nmdm6B
 +</​code>​
 +
 +==== Backbone routers configuration ====
 +
 +=== Router 2 ===
 +
 +Router 2 is configured for rate-limiting traffic at 1 Mb/s on interface to/from R1.
 +
 +<​code>​
 +sysrc hostname=R2
 +sysrc ifconfig_vtnet0="​inet 10.0.12.2/​24"​
 +sysrc ifconfig_vtnet3="​inet 10.0.25.2/​24"​
 +sysrc defaultrouter="​10.0.25.5"​
 +sysrc firewall_enable=YES
 +sysrc firewall_script="/​etc/​ipfw.rules"​
 +
 +cat > /​etc/​ipfw.rules <<EOF
 +#!/bin/sh
 +fwcmd="/​sbin/​ipfw"​
 +kldstat -q -m dummynet || kldload dummynet
 +# Flush out the list before we begin.
 +${fwcmd} -f flush
 +${fwcmd} pipe 10 config bw 1Mbit/s
 +${fwcmd} pipe 20 config bw 1Mbit/s
 +#Traffic getting out vtnet0 is limited to 1Mbit/s
 +${fwcmd} add 1000 pipe 10 all from any to any out via vtnet0
 +#Traffic getting int vtnet0 is limited to 1Mbit/s
 +${fwcmd} add 2000 pipe 20 all from any to any in via vtnet0
 +#We don't want to block traffic, only shape some
 +${fwcmd} add 3000 allow ip from any to any
 +EOF
 +
 +service netif restart
 +service routing restart
 +service ipfw start
 +hostname R2
 +config save
 +</​code>​
 +
 +=== Router 3 ===
 +
 +Router 3 is configured for rate-limiting traffic at 20 Mb/s on interface to/from R1.
 +
 +<​code>​
 +sysrc hostname=R3
 +sysrc ifconfig_vtnet0="​inet 10.0.13.3/​24"​
 +sysrc ifconfig_vtnet3="​inet 10.0.35.3/​24"​
 +sysrc defaultrouter="​10.0.35.5"​
 +sysrc firewall_enable=YES
 +sysrc firewall_script="/​etc/​ipfw.rules"​
 +
 +cat > /​etc/​ipfw.rules <<EOF
 +#!/bin/sh
 +fwcmd="/​sbin/​ipfw"​
 +kldstat -q -m dummynet || kldload dummynet
 +# Flush out the list before we begin.
 +${fwcmd} -f flush
 +${fwcmd} pipe 10 config bw 2Mbit/s
 +${fwcmd} pipe 20 config bw 2Mbit/s
 +#Traffic getting out vtnet0 is limited to 2Mbit/s
 +${fwcmd} add 1000 pipe 10 all from any to any out via vtnet0
 +#Traffic getting int vtnet0 is limited to 2Mbit/s
 +${fwcmd} add 2000 pipe 20 all from any to any in via vtnet0
 +#We don't want to block traffic, only shape some
 +${fwcmd} add 3000 allow ip from any to any
 +EOF
 +
 +service netif restart
 +service routing restart
 +service ipfw start
 +hostname R3
 +config save
 +
 +</​code>​
 +
 +=== Router 4 ===
 +
 +Router 4 is configured for rate-limiting traffic at 3 Mb/s on interface to/from R1.
 +
 +<​code>​
 +sysrc hostname=R4
 +sysrc ifconfig_vtnet0="​inet 10.0.14.4/​24"​
 +sysrc ifconfig_vtnet3="​inet 10.0.45.4/​24"​
 +sysrc defaultrouter="​10.0.45.5"​
 +sysrc firewall_enable=YES
 +sysrc firewall_script="/​etc/​ipfw.rules"​
 +
 +cat > /​etc/​ipfw.rules <<EOF
 +#!/bin/sh
 +fwcmd="/​sbin/​ipfw"​
 +kldstat -q -m dummynet || kldload dummynet
 +# Flush out the list before we begin.
 +${fwcmd} -f flush
 +${fwcmd} pipe 10 config bw 3Mbit/s
 +${fwcmd} pipe 20 config bw 3Mbit/s
 +#Traffic getting out vtnet0 is limited to 3Mbit/s
 +${fwcmd} add 1000 pipe 10 all from any to any out via vtnet0
 +#Traffic getting int vten0 is limited to 3Mbit/s
 +${fwcmd} add 2000 pipe 20 all from any to any in via vtnet0
 +#We don't want to block traffic, only shape some
 +${fwcmd} add 3000 allow ip from any to any
 +EOF
 +
 +service netif restart
 +service routing restart
 +service ipfw start
 +hostname R4
 +config save
 +</​code>​
 +
 +=== Router 5 ===
 +
 +Router 5 is the MLPPP server default gateway.
 +
 +<​code>​
 +sysrc hostname=R5
 +sysrc ifconfig_vtnet1="​inet 10.0.25.5/​24"​
 +sysrc ifconfig_vtnet2="​inet 10.0.35.5/​24"​
 +sysrc ifconfig_vtnet3="​inet 10.0.45.5/​24"​
 +sysrc ifconfig_vtnet4="​inet 10.0.56.5/​24"​
 +sysrc static_routes="​ISP1 ISP2 ISP3"
 +sysrc route_ISP1="​-host 10.0.12.1 10.0.25.2"​
 +sysrc route_ISP2="​-host 10.0.13.1 10.0.35.3"​
 +sysrc route_ISP3="​-host 10.0.14.1 10.0.45.4"​
 +service netif restart
 +service routing restart
 +hostname R5
 +config save
 +</​code>​
 +==== Router 6 : L2TP MLPPP server ====
 +
 +Router 6 is configured as a L2TP server.
 +
 +<​code>​
 +sysrc hostname=R6
 +sysrc cloned_interfaces="​lo1"​
 +sysrc ifconfig_lo1="​inet 10.6.6.6/​32"​
 +sysrc ifconfig_vtnet4="​inet 10.0.56.62/​24"​
 +sysrc ifconfig_vtnet4_alias1="​inet 10.0.56.63/​32"​
 +sysrc ifconfig_vtnet4_alias2="​inet 10.0.56.64/​32"​
 +sysrc defaultrouter="​10.0.56.5"​
 +sysrc mpd_enable=YES
 +sysrc mpd_flags="​-b -s ppp"
 +cat > /​usr/​local/​etc/​mpd5/​mpd.conf <<EOF
 +default:
 +        load l2tp_server
 +l2tp_server:​
 +        # IP Pool
 +        set ippool add pool1 10.0.16.10 10.0.16.100
 +        # Create bundle template named B
 +        create bundle template B
 +        # Enable IPv6
 +        set bundle enable ipv6cp
 +        # Configure interface
 +        set iface enable tcpmssfix
 +        # Handle IPCP configuration
 +        set ipcp yes vjcomp
 +        # Handle the IPCP configuration
 +        set ipcp ranges 10.0.16.1/​24 ippool pool1
 +        # Create clonable link template named adsl1
 +        create link template L l2tp
 +        set link action bundle B
 +        set link enable multilink
 +        set link keep-alive 10 30
 +        set link mtu 1460
 +        set l2tp secret blah
 +        # SDSL1
 +        create link static sdsl1 L
 +        set l2tp self 10.0.56.62
 +        # set DOWNLOAD bandwidth of ISP1
 +        set link bandwidth 1000000
 +        set link enable incoming
 +        # SDSL2
 +        create link static sdsl2 L
 +        set l2tp self 10.0.56.63
 +        # set DOWNLOAD bandwidth of ISP2
 +        set link bandwidth 2000000
 +        set link enable incoming
 +        # SDSL3
 +        create link static sdsl3 L
 +        set l2tp self 10.0.56.64
 +        # set DOWNLOAD bandwidth of ISP3
 +        set link bandwidth 3000000
 +        set link enable incoming
 +EOF
 +
 +service netif restart
 +service routing restart
 +service mpd5 start
 +hostname R6
 +config save
 +</​code>​
 +
 +==== Router 1 : L2TP MLPPP client ====
 +
 +Router 1 is configured as a simple L2TP MLPPP client router connected to 3 Internet link.
 +
 +<​code>​
 +sysrc hostname=R1
 +sysrc ifconfig_vtnet0="​inet 10.0.12.1/​24"​
 +sysrc ifconfig_vtnet1="​inet 10.0.13.1/​24"​
 +sysrc ifconfig_vtnet2="​inet 10.0.14.1/​24"​
 +sysrc static_routes="​ISP1 ISP2 ISP3"
 +sysrc route_ISP1="​-host 10.0.56.62 10.0.12.2"​
 +sysrc route_ISP2="​-host 10.0.56.63 10.0.13.3"​
 +sysrc route_ISP3="​-host 10.0.56.64 10.0.14.4"​
 +sysrc mpd_enable=YES
 +sysrc mpd_flags="​-b -s ppp"
 +cat > /​usr/​local/​etc/​mpd5/​mpd.conf <<EOF
 +default:
 +        load l2tp_client
 +l2tp_client:​
 +        # Create the bundle
 +        create bundle template B
 +        # Enable IPv6
 +        set bundle enable ipv6cp
 +        # Enable TCP MSS fix
 +        set iface enable tcpmssfix
 +        # Use this interface as default route
 +        set iface route default
 +        # Disable IPCP configuration for the iperf test
 +        #set ipcp yes vjcomp
 +        # Create clonable template link ADSL1
 +        create link template L l2tp
 +        set link action bundle B
 +        set link enable multilink
 +        set link keep-alive 10 30
 +        set link mtu 1460
 +        set l2tp secret blah
 +        # Retry indefinitly to redial
 +        set link max-redial 0
 +        # SDSL1
 +        create link static sdsl1 L
 +        set l2tp peer 10.0.56.62
 +        # Configure the UPLOAD bandwidth
 +        set link bandwidth 1000000
 +        open link
 +        # SDSL2
 +        create link static sdsl2 L
 +        set l2tp peer 10.0.56.63
 +        # Configure the UPLOAD bandwidth
 +        set link bandwidth 2000000
 +        open link
 +        # SDSL3
 +        create link static sdsl3 L
 +        set l2tp peer 10.0.56.64
 +        # Configure the UPLOAD bandwidth
 +        set link bandwidth 3000000
 +        open link
 +EOF
 +
 +service netif restart
 +service routing restart
 +service mpd5 start
 +hostname R1
 +config save
 +</​code>​
 +
 +===== Final testing =====
 +
 +==== Each ISP link bandwidth ====
 +
 +Start iperf in server mode on R6:
 +<​code>​
 +[root@R6]~# iperf3 -s
 +-----------------------------------------------------------
 +Server listening on 5201
 +-----------------------------------------------------------
 +</​code>​
 +
 +
 +Now check the correct limited bandwitdh for each different links:
 +  * Link to R6 across R2: 1Mb/s
 +  * Link to R6 across R3: 2Mb/s
 +  * Link to R6 across R4: 3Mb/s
 +
 +<​code>​
 +[root@R1]~# iperf3 -i 0 -c 10.0.56.62
 +Connecting to host 10.0.56.62, port 5201
 +[  5] local 10.0.12.1 port 30648 connected to 10.0.56.62 port 5201
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        ​Retr ​ Cwnd
 +[  5]   ​0.00-10.00 ​ sec  1.21 MBytes ​ 1.02 Mbits/​sec ​   0   65.1 KBytes
 +- - - - - - - - - - - - - - - - - - - - - - - - -
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        Retr
 +[  5]   ​0.00-10.00 ​ sec  1.21 MBytes ​ 1.02 Mbits/​sec ​   0             ​sender
 +[  5]   ​0.00-10.51 ​ sec  1.19 MBytes ​  953 Kbits/​sec ​                 receiver
 +
 +iperf Done.
 +[root@R1]~# iperf3 -i 0 -c 10.0.56.63
 +Connecting to host 10.0.56.63, port 5201
 +[  5] local 10.0.13.1 port 13090 connected to 10.0.56.63 port 5201
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        ​Retr ​ Cwnd
 +[  5]   ​0.00-10.00 ​ sec  2.35 MBytes ​ 1.97 Mbits/​sec ​   0   65.1 KBytes
 +- - - - - - - - - - - - - - - - - - - - - - - - -
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        Retr
 +[  5]   ​0.00-10.00 ​ sec  2.35 MBytes ​ 1.97 Mbits/​sec ​   0             ​sender
 +[  5]   ​0.00-10.26 ​ sec  2.33 MBytes ​ 1.91 Mbits/​sec ​                 receiver
 +
 +iperf Done.
 +[root@R1]~# iperf3 -i 0 -c 10.0.56.64
 +Connecting to host 10.0.56.64, port 5201
 +[  5] local 10.0.14.1 port 57319 connected to 10.0.56.64 port 5201
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        ​Retr ​ Cwnd
 +[  5]   ​0.00-10.00 ​ sec  3.48 MBytes ​ 2.92 Mbits/​sec ​   0   65.1 KBytes
 +- - - - - - - - - - - - - - - - - - - - - - - - -
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        Retr
 +[  5]   ​0.00-10.00 ​ sec  3.48 MBytes ​ 2.92 Mbits/​sec ​   0             ​sender
 +[  5]   ​0.00-10.16 ​ sec  3.46 MBytes ​ 2.86 Mbits/​sec ​                 receiver
 +
 +iperf Done.
 +
 +</​code>​
 +==== Aggregated ISP link bandwidth ====
 +
 +The aggregated link bandwidth should be negotiated to 6Mb/s (1+2+3):
 +
 +<​code>​
 +[root@R1]~# grep Bundle /​var/​log/​ppp.log
 +Nov 12 06:04:54 router ppp[87823]: [B-1] Bundle: Interface ng0 created
 +Nov 12 06:04:54 router ppp[87823]: [B-1] Bundle: Status update: up 1 link, total bandwidth 2000000 bps
 +Nov 12 06:04:54 router ppp[87823]: [B-1] Bundle: Status update: up 2 links, total bandwidth 3000000 bps
 +Nov 12 06:04:54 router ppp[87823]: [B-1] Bundle: Status update: up 3 links, total bandwidth 6000000 bps
 +</​code>​
 +
 +and iperf measurement close to 6Mb/s:
 +<​code>​
 +[root@R1]~# iperf3 -i 0 -c 10.6.6.6
 +Connecting to host 10.6.6.6, port 5201
 +[  5] local 10.0.16.10 port 51350 connected to 10.6.6.6 port 5201
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        ​Retr ​ Cwnd
 +[  5]   ​0.00-10.00 ​ sec  6.42 MBytes ​ 5.38 Mbits/​sec ​   0   65.1 KBytes
 +- - - - - - - - - - - - - - - - - - - - - - - - -
 +[ ID] Interval ​          ​Transfer ​    ​Bitrate ​        Retr
 +[  5]   ​0.00-10.00 ​ sec  6.42 MBytes ​ 5.38 Mbits/​sec ​   0             ​sender
 +[  5]   ​0.00-10.09 ​ sec  6.40 MBytes ​ 5.32 Mbits/​sec ​                 receiver
 +
 +iperf Done.
 +</​code>​
 +
 +In the same time, if you start a "​netstat -ihw 1" on R2, R3 and R4, you should see distributed traffic between them.
 +    
 +
 +
 +
 +===== Performance lab =====
 +
 +This lab will test mpd5 performance by aggregating 2 gigabit links.
 +
 +Here is the concept:
 +
 +{{:​documentation:​examples:​bsdrp.lab.mpd5.performance.png|}}
 +
 +This lab use 3 [[IBM System x3550 M3]] with **quad** cores (Intel Xeon L5630 2.13GHz, hyper-threading disabled), quad NIC 82580 connected to the PCI-Express Bus and dual port Intel 10-Gigabit X540-AT2 connected to the PCI-Express Bus.
 +
 +==== Router 1 ====
 +
 +Router 1 is configured a a simple end-point.
 +
 +Set the base parameters:
 +<​code>​
 +sysrc hostname=R1
 +sysrc ifconfig_ix0="​1.1.1.1/​24"​
 +sysrc ifconfig_ix0_ipv6="​inet6 2001:​db8:​1::​1/​64"​
 +sysrc defaultrouter="​1.1.1.2"​
 +sysrc ipv6_defaultrouter="​2001:​db8:​1::​2"​
 +service netif restart
 +service routing restart
 +config save
 +</​code>​
 +
 +==== Router 2 ====
 +
 +Router 2 is configured as a L2TP MLPPP client router.
 +
 +Configure global parameters:
 +<​code>​
 +sysrc hostname=R2
 +sysrc ifconfig_ix0="​1.1.1.2/​24"​
 +sysrc ifconfig_ix0_ipv6="​inet6 2001:​db8:​1::​2/​64"​
 +sysrc ifconfig_igb2="​10.0.23.2/​24"​
 +sysrc ifconfig_igb3="​10.1.23.2/​24"​
 +sysrc mpd_enable=YES
 +sysrc mpd_flags="​-b -s ppp"
 +</​code>​
 +
 +Configure mpd:
 +
 +<​code>​
 +cat > /​usr/​local/​etc/​mpd5/​mpd.conf <<'​EOF'​
 +default:
 +        load l2tp_client
 +l2tp_client:​
 +        # Create the bundle
 +        create bundle template B
 +        # Enable IPv6
 +        set bundle enable ipv6cp
 +        # Disable compression (for iperf test)
 +        #set bundle enable compression
 +        #set ccp yes deflate
 +        # Enable TCP MSS fix
 +        set iface enable tcpmssfix
 +        # Use this interface as default route
 +        set iface route default
 +        # Disable IPCP configuration for the iperf test
 +        #set ipcp yes vjcomp
 +        # Create clonable template link
 +        create link template L l2tp
 +        set link action bundle B
 +        set link enable multilink
 +        set link keep-alive 10 30
 +        set link mtu 1460
 +        set l2tp secret blah
 +        set link max-redial 0
 +        # LINK1
 +        create link static link1 L
 +        set l2tp peer 10.0.23.3
 +        open link
 +        # LINK2
 +        create link static link2 L
 +        set l2tp peer 10.1.23.3
 +        open link
 +'​EOF'​
 +</​code>​
 +
 +And apply your changes:
 +
 +<​code>​
 +service netif restart
 +service routing restart
 +service mpd5 start
 +config save
 +</​code>​
 +
 +==== Router 3 ====
 +
 +Router 3 is configured as a (L2TP server).
 +
 +Set the global parameters:
 +<​code>​
 +sysrc hostname=R3
 +sysrc cloned_interfaces="​lo1"​
 +sysrc ifconfig_lo1="​inet 3.3.3.3/​32"​
 +sysrc ifconfig_lo1_ipv6="​inet6 2001:​db8:​3::​3/​64"​
 +sysrc ifconfig_igb2="​10.0.23.3/​24"​
 +sysrc ifconfig_igb3="​10.1.23.3/​24"​
 +sysrc mpd_enable=YES
 +sysrc mpd_flags="​-b -s ppp"
 +</​code>​
 +
 +Configure mpd5:
 +
 +<​code>​
 +cat > /​usr/​local/​etc/​mpd5/​mpd.conf <<'​EOF'​
 +default:
 +        load l2tp_server
 +l2tp_server:​
 +        # IP Pool
 +        set ippool add pool1 10.3.23.10 10.3.23.100
 +        # Create bundle template named B
 +        create bundle template B
 +        # Enable compression (disabled on the client)
 +        set bundle enable compression
 +        set ccp yes deflate
 +        # Enable IPv6
 +        set bundle enable ipv6cp
 +        # Configure interface
 +        set iface enable tcpmssfix
 +        # Handle IPCP configuration
 +        set ipcp yes vjcomp
 +        # Handle the IPCP configuration
 +        set ipcp ranges 10.3.23.1/​24 ippool pool1
 +        # Create clonable link template
 +        create link template L l2tp
 +        set link action bundle B
 +        set link enable multilink
 +        set link keep-alive 10 30
 +        set link mtu 1460
 +        set l2tp secret blah
 +        # LINK1
 +        create link static link1 L
 +        set l2tp self 10.0.23.3
 +        set link enable incoming
 +        # LINK2
 +        create link static link2 L
 +        set l2tp self 10.1.23.3
 +        set link enable incoming
 +'​EOF'​
 +</​code>​
 +
 +if-up script (for installing routes to R1 subnet):
 +
 +<​code>​
 +cat > /​usr/​local/​etc/​mpd5/​if-up.sh <<'​EOF'​
 +#!/bin/sh
 +#mpd5 call script with options:
 +#interface proto local-ip remote-ip authname [ dns1 server-ip ] [ dns2 server-ip ] peer-address
 +#Examples
 +#command "/​usr/​local/​etc/​mpd5/​if-up.sh ng0 inet 10.3.23.1/​32 10.3.23.10 '​-'​ ''​ ''​ '​10.1.23.2'"​
 +#command "/​usr/​local/​etc/​mpd5/​if-up.sh ng0 inet6 fe80::​5ef3:​fcff:​fee5:​a4c0%ng0 fe80::​5ef3:​fcff:​fee5:​7338%ng0 '​-'​ '​10.1.23.2'"​
 +#mpd5 wait for 0 as successful
 +set -e
 +logger "$0 called with parameters: $@"
 +remote_inet="​1.1.1.0/​24"​
 +remote_inet6="​2001:​db8:​1::​ -prefixlen 64"
 +eval "
 +        if route get -net -\$2 \${remote_$2};​ then
 +                logger \"​route \${remote_$2} already present\"​
 +                return 0
 +        else
 +                cmd=\"​route add -\$2 \${remote_$2} \$4\"
 +        fi
 +"
 +if $cmd; then
 +        logger "$0: $cmd successfull"​
 +        return 0
 +else
 +        logger "$0: $cmd failed"​
 +        return 1
 +fi
 +'​EOF'​
 +chmod +x /​usr/​local/​etc/​mpd5/​if-up.sh
 +</​code>​
 +
 +Then the if-down script:
 +
 +<​code>​
 +cat > /​usr/​local/​etc/​mpd5/​if-down.sh <<'​EOF'​
 +#!/bin/sh
 +#mpd5 call script with options:
 +#interface proto local-ip remote-ip authname peer-address
 +#example:
 +#command "/​urs/​local/​etc/​mpd5/​if-down.sh ng0 inet 10.3.23.1/​32 10.3.23.10 '​-'​ '​10.0.23.2'"​
 +logger "$0 called with parameters: $@"
 +remote_inet="​1.1.1.0/​24"​
 +remote_net6="​2001:​db8:​1::​1 -prefixlen 64"
 +eval "
 +        if ! route get -net -\$2 ${remote_$2};​ then
 +                logger "Route ${remote_inet} not in table"
 +                return 0
 +        else
 +                cmd=\"​route del \${remote_$2} \$4\"
 +        fi
 +"
 +if $cmd; then
 +        logger "​if-down:​ ${cmd} succesfull"​
 +        return 0
 +else
 +        logger "​if-down:​ ${cmd} failed"​
 +        return 1
 +fi
 +
 +'​EOF'​
 +chmod +x /​usr/​local/​etc/​mpd5/​if-down.sh
 +</​code>​
 +And apply your changes:
 +
 +<​code>​
 +service netif restart
 +service routing restart
 +service mpd5 start
 +config save
 +</​code>​
 +
 +==== Performance tests ====
 +
 +=== Checking perf tool ===
 +
 +== Direct tests between R1 and R2 ===
 +
 +<​code>​
 +[root@bsdrp1]~#​ iperf -c 1.1.1.2 -t 60
 +------------------------------------------------------------
 +Client connecting to 1.1.1.2, TCP port 5001
 +TCP window size: 32.5 KByte (default)
 +------------------------------------------------------------
 +[  3] local 1.1.1.1 port 57149 connected with 1.1.1.2 port 5001
 +[ ID] Interval ​      ​Transfer ​    ​Bandwidth
 +[  3]  0.0-60.0 sec  24.5 GBytes ​ 3.50 Gbits/sec
 +</​code>​
 +
 +==== Direct tests between R2 and R3 ====
 +We start by testing each Gigabit links between R2 and R3 for measuring iperf value on standard gigabit link:
 +
 +<​code>​
 +[root@R2]~# iperf -c 10.0.23.3 -t 60
 +------------------------------------------------------------
 +Client connecting to 10.0.23.3, TCP port 5001
 +TCP window size: 32.5 KByte (default)
 +------------------------------------------------------------
 +[  3] local 10.0.23.2 port 21046 connected with 10.0.23.3 port 5001
 +[ ID] Interval ​      ​Transfer ​    ​Bandwidth
 +[  3]  0.0-60.0 sec  6.54 GBytes ​  936 Mbits/sec
 +[root@R2]~# iperf -c 10.1.23.3 -t 60
 +------------------------------------------------------------
 +Client connecting to 10.1.23.3, TCP port 5001
 +TCP window size: 32.5 KByte (default)
 +------------------------------------------------------------
 +[  3] local 10.1.23.2 port 50717 connected with 10.1.23.3 port 5001
 +[ ID] Interval ​      ​Transfer ​    ​Bandwidth
 +[  3]  0.0-60.0 sec  6.55 GBytes ​  937 Mbits/sec
 +</​code>​
 +==== mpd5 perfs ===
 +
 +=== between R2 and R3 ==
 +
 +Iperf will use the MPPP tunnel extremity:
 +<​code>​
 +[root@R2]~# set DEST=`ifconfig ng0 | grep 'inet ' | cut -d ' ' -f 4`
 +[root@R2]~# iperf -c $DEST -t 60
 +------------------------------------------------------------
 +Client connecting to 10.3.23.1, TCP port 5001
 +TCP window size: 32.5 KByte (default)
 +------------------------------------------------------------
 +[  3] local 10.3.23.10 port 19383 connected with 10.3.23.1 port 5001
 +[ ID] Interval ​      ​Transfer ​    ​Bandwidth
 +[  3]  0.0-60.1 sec  6.14 GBytes ​  878 Mbits/sec
 +</​code>​
 +
 +
 +
 +The value is almost the same than without MLPPP aggregated link, but correctly load-balanced across each link.
 +iR2 stats during this test:
 +
 +<​code>​
 +                    /0   /​1 ​  /​2 ​  /​3 ​  /​4 ​  /​5 ​  /​6 ​  /​7 ​  /​8 ​  /​9 ​  /10
 +     Load Average ​  ||||
 +
 +      Interface ​          ​Traffic ​              ​Peak ​               Total
 +            ng0  in     ​17.271 Mb/s         ​18.694 Mb/s            3.743 GB
 +                 ​out ​  ​974.656 Mb/s       ​1055.503 Mb/s            7.324 GB
 +
 +           ​igb3 ​ in     ​11.405 Mb/s         ​12.261 Mb/s            3.456 GB
 +                 ​out ​  ​507.970 Mb/s        550.255 Mb/s           ​27.449 GB
 +
 +           ​igb2 ​ in     ​11.389 Mb/s         ​12.274 Mb/s            3.422 GB
 +                 ​out ​  ​508.061 Mb/s        550.328 Mb/s           ​18.673 GB
 +</​code>​
 +
 +And the load show:
 +<​code>​
 +[root@R2]~# top -nCHSIzs1
 +last pid: 14152; ​ load averages: ​ 0.94,  0.48,  0.21  up 0+23:​54:​56 ​   12:40:23
 +155 processes: 5 running, 99 sleeping, 51 waiting
 +
 +Mem: 3564K Active, 26M Inact, 384M Wired, 256K Cache, 17M Buf, 15G Free
 +Swap:
 +
 +
 +  PID USERNAME ​  PRI NICE   ​SIZE ​   RES STATE   ​C ​  ​TIME ​    CPU COMMAND
 + 8524 root       ​-16 ​   -     ​0K ​   64K sleep   ​0 ​  ​2:​04 ​ 25.68% ng_queue{ng_queue2}
 + 8524 root       ​-16 ​   -     ​0K ​   64K sleep   ​1 ​  ​1:​47 ​ 25.49% ng_queue{ng_queue1}
 + 8524 root       ​-16 ​   -     ​0K ​   64K sleep   ​3 ​  ​2:​03 ​ 22.36% ng_queue{ng_queue0}
 +14149 root        36    0 32136K ​ 3092K sbwait ​ 1   ​0:​06 ​ 21.88% iperf{iperf}
 + 8524 root       ​-16 ​   -     ​0K ​   64K sleep   ​2 ​  ​1:​56 ​ 20.26% ng_queue{ng_queue3}
 +   11 root       ​-92 ​   -     ​0K ​  816K WAIT    3   ​0:​20 ​  7.57% intr{irq286:​ igb3:que}
 +   11 root       ​-92 ​   -     ​0K ​  816K WAIT    1   ​0:​17 ​  5.96% intr{irq279:​ igb2:que}
 +   11 root       ​-92 ​   -     ​0K ​  816K WAIT    3   ​0:​09 ​  0.78% intr{irq281:​ igb2:que}
 +   11 root       ​-92 ​   -     ​0K ​  816K WAIT    2   ​0:​05 ​  0.59% intr{irq280:​ igb2:que}
 +   11 root       ​-92 ​   -     ​0K ​  816K WAIT    0   ​0:​06 ​  0.39% intr{irq278:​ igb2:que}
 +    0 root       ​-92 ​   0     ​0K ​  560K -       ​1 ​  ​0:​00 ​  0.10% kernel{igb3 que}
 +</​code>​
 +
 +
 +For information,​ using netblast (UDP packet generator) disturb the links:
 +<​code>​
 +netblast $DEST 9090 1470 30 `sysctl -n hw.ncpu`
 +</​code>​
 +
 +and measure the bandwidth received on R3:
 +<​code>​
 +         0 pps    0.000 Mbps - 0 pkts in 0.522753682 ns
 +      2149 pps   ​25.277 Mbps - 1079 pkts in 0.501997052 ns
 +         0 pps    0.000 Mbps - 0 pkts in 0.501999569 ns
 +        45 pps    0.539 Mbps - 23 pkts in 0.502000238 ns
 +         0 pps    0.000 Mbps - 0 pkts in 0.502000362 ns
 +       713 pps    8.387 Mbps - 358 pkts in 0.501999604 ns
 +      2107 pps   ​24.786 Mbps - 1077 pkts in 0.511002670 ns
 +         0 pps    0.000 Mbps - 0 pkts in 0.501998712 ns
 +         0 pps    0.000 Mbps - 0 pkts in 0.501998967 ns
 +        21 pps    0.255 Mbps - 11 pkts in 0.508000385 ns
 +         0 pps    0.000 Mbps - 0 pkts in 0.501998785 ns
 +</​code>​
 +
 +The packet generator prevent to manage keepalive:
 +
 +<​code>​
 +Jan 27 10:50:38 R2 ppp: [link1] LCP: no reply to 1 echo request(s)
 +Jan 27 10:50:38 R2 ppp: [link2] LCP: no reply to 1 echo request(s)
 +Jan 27 10:50:48 R2 ppp: [link2] LCP: no reply to 2 echo request(s)
 +Jan 27 10:50:48 R2 ppp: [link2] LCP: peer not responding to echo requests
 +Jan 27 10:50:48 R2 ppp: [link2] LCP: state change Opened --> Stopping
 +Jan 27 10:50:48 R2 ppp: [link2] Link: Leave bundle "​B-1"​
 +</​code>​
documentation/examples/aggregating_multiple_isp_links.txt · Last modified: 2018/11/12 06:09 (external edit)