Skip to content

Branch office ECMP over WireGuard

This lab connects a branch office to its headquarters through two ISPs at the same time, without mpd5 or MLVPN:

  • one WireGuard tunnel per ISP, so the ISPs only carry encrypted IPv4 between the two public addresses, and IPv6 travels inside the tunnels;
  • OSPF and OSPFv3 over both tunnels with the same cost, so FRR installs equal-cost multi-path (ECMP) routes and the flows share both links;
  • BFD on every adjacency: a tunnel interface stays up when its ISP fails, so BFD is what notices the silence and removes the path in under a second.

Unlike link aggregation (ML-PPP, MLVPN), ECMP keeps each flow on one path: no packet reordering, but a single TCP connection never goes faster than one link. Many flows, as from a whole office, use both.

Overview

Network diagram

flowchart LR
    PC(("<b>pc</b>"))
    BR(("<b>branch</b><br/>FRR"))
    IA(("<b>ispA</b>"))
    IB(("<b>ispB</b>"))
    HQ(("<b>hq</b><br/>FRR<br/>10.0.5.5"))

    PC ---|"10.0.12.0/24"| BR
    BR ---|"10.0.23.0/24"| IA
    BR ---|"10.0.24.0/24"| IB
    IA ---|"10.0.35.0/24"| HQ
    IB ---|"10.0.45.0/24"| HQ
    BR -.-|"<b>wg0</b>, OSPF, BFD<br/>10.9.1.0/24"| HQ
    BR -.-|"<b>wg1</b>, OSPF, BFD<br/>10.9.2.0/24"| HQ

    classDef frr fill:#fde2c4,stroke:#d9480f,stroke-width:2px,color:#222
    classDef router fill:#ffe3e3,stroke:#c92a2a,stroke-width:2px,color:#222
    classDef host fill:#e9ecef,stroke:#6c757d,stroke-width:2px,color:#222
    class BR,HQ frr
    class IA,IB router
    class PC host

    linkStyle 0,1,2,3,4 stroke:#868e96,stroke-width:3px
    linkStyle 5,6 stroke:#0b7285,stroke-width:4px

The PC LAN 10.0.12.0/24 has an IPv6 twin 2001:db8:12::/64, and the tunnels 2001:db8:91::/64 and 2001:db8:92::/64. HQ's services sit on its lo1: 10.0.5.5 and 2001:db8:5::5. The ISP links are IPv4 only.

Router Role
pc Branch workstation, default route via the branch router
branch Branch router: wg0 over ISP-A, wg1 over ISP-B, OSPF, OSPFv3 and BFD on both
ispA, ispB Two ISPs, plain IPv4 forwarding
hq Headquarters router, same tunnels and routing as the branch

Setting up the lab

Downloading BSD Router Project images

Download the BSDRP serial image from SourceForge.

Download lab scripts

More information on these BSDRP lab scripts is available on How to build a BSDRP router lab.

Start the lab with five routers, configured automatically by labconfig:

tools/BSDRP-lab-bhyve.sh -i BSDRP-full-amd64.img -n 5 -r branch

The VMs are fully meshed; this lab only uses the links drawn above.

Router configuration

Each block below is what labconfig branch_vm<N> applies. Replace em by vtnet in the interface names on bhyve (labconfig does it for you). The WireGuard keys are lab keys: generate your own with wg genkey.

PC (VM1)

sysrc hostname=pc \
 gateway_enable=NO \
 ipv6_gateway_enable=NO \
 ifconfig_em0="inet 10.0.12.1/24" \
 ifconfig_em0_ipv6="inet6 2001:db8:12::1 prefixlen 64" \
 defaultrouter=10.0.12.2 \
 ipv6_defaultrouter=2001:db8:12::2
config save
service hostname restart
service netif restart
service routing restart

Branch router (VM2)

Interfaces, the two tunnels, and one host route per ISP to HQ's public address on that ISP, so each tunnel's encrypted packets leave through its own ISP:

sysrc hostname=branch \
 ifconfig_em0="inet 10.0.12.2/24" \
 ifconfig_em0_ipv6="inet6 2001:db8:12::2 prefixlen 64" \
 ifconfig_em1="inet 10.0.23.2/24" \
 ifconfig_em2="inet 10.0.24.2/24" \
 cloned_interfaces="wg0 wg1" \
 ifconfig_wg0="inet 10.9.1.2/24" \
 ifconfig_wg0_ipv6="inet6 2001:db8:91::2 prefixlen 64 auto_linklocal" \
 ifconfig_wg1="inet 10.9.2.2/24" \
 ifconfig_wg1_ipv6="inet6 2001:db8:92::2 prefixlen 64 auto_linklocal" \
 static_routes="ispa ispb" \
 route_ispa="-host 10.0.35.5 10.0.23.3" \
 route_ispb="-host 10.0.45.5 10.0.24.4" \
 frr_enable=YES \
 frr_daemons="mgmtd zebra staticd bfdd ospfd ospf6d"

if_wg gets no IPv6 link-local address by default, and OSPFv3 needs one: hence auto_linklocal.

The tunnels, one per ISP. AllowedIPs accepts everything, as OSPF decides what goes through which tunnel. The rc.d netif script sources /etc/start_if.<interface> before configuring an interface, which is the place to load the keys:

mkdir -p /usr/local/etc/wireguard
cat > /usr/local/etc/wireguard/wg0.conf <<EOF
[Interface]
PrivateKey = wKzufFch9l+SpPS417DGvSgUbhII6wcTB3ul/tjP7XQ=
ListenPort = 51820

[Peer]
PublicKey = 46I9Wa1Neq3NxsMnQh+1sKjrh0np8EdLhrCoFk78pSg=
Endpoint = 10.0.35.5:51820
AllowedIPs = 0.0.0.0/0, ::/0
PersistentKeepalive = 10
EOF
cat > /usr/local/etc/wireguard/wg1.conf <<EOF
[Interface]
PrivateKey = wKzufFch9l+SpPS417DGvSgUbhII6wcTB3ul/tjP7XQ=
ListenPort = 51821

[Peer]
PublicKey = 46I9Wa1Neq3NxsMnQh+1sKjrh0np8EdLhrCoFk78pSg=
Endpoint = 10.0.45.5:51821
AllowedIPs = 0.0.0.0/0, ::/0
PersistentKeepalive = 10
EOF
chmod 600 /usr/local/etc/wireguard/wg*.conf
echo "wg setconf wg0 /usr/local/etc/wireguard/wg0.conf" > /etc/start_if.wg0
echo "wg setconf wg1 /usr/local/etc/wireguard/wg1.conf" > /etc/start_if.wg1

Flow hashing for forwarded packets (see Why net.route.hash_forward):

echo "net.route.hash_forward=1" >> /etc/sysctl.conf
sysctl net.route.hash_forward=1

FRR: OSPF and OSPFv3 on both tunnels, BFD on every adjacency, the LAN announced as passive:

cat > /usr/local/etc/frr/frr.conf <<EOF
frr defaults traditional
hostname branch
log syslog
!
interface wg0
 ip ospf network point-to-point
 ip ospf bfd
 ipv6 ospf6 area 0.0.0.0
 ipv6 ospf6 network point-to-point
 ipv6 ospf6 bfd
exit
!
interface wg1
 ip ospf network point-to-point
 ip ospf bfd
 ipv6 ospf6 area 0.0.0.0
 ipv6 ospf6 network point-to-point
 ipv6 ospf6 bfd
exit
!
interface em0
 ipv6 ospf6 area 0.0.0.0
 ipv6 ospf6 passive
exit
!
router ospf
 ospf router-id 10.0.0.2
 passive-interface em0
 network 10.9.0.0/16 area 0.0.0.0
 network 10.0.12.0/24 area 0.0.0.0
exit
!
router ospf6
 ospf6 router-id 10.0.0.2
exit
!
EOF
config save
service hostname restart
service netif restart
service routing restart
service frr start

The underlay subnets stay out of OSPF: if HQ's public addresses were learned through the tunnels, the tunnels would try to run over themselves.

ISP routers (VM3 and VM4)

ISP-A is shown; ISP-B uses ispB, 10.0.24.4/24 and 10.0.45.4/24:

sysrc hostname=ispA \
 gateway_enable=YES \
 ipv6_gateway_enable=NO \
 ifconfig_em1="inet 10.0.23.3/24" \
 ifconfig_em3="inet 10.0.35.3/24"
config save
service hostname restart
service netif restart
service routing restart

HQ router (VM5)

The mirror of the branch: its services on lo1, the ISP links on em2 and em3, the peers' addresses swapped:

sysrc hostname=hq \
 ifconfig_em2="inet 10.0.35.5/24" \
 ifconfig_em3="inet 10.0.45.5/24" \
 cloned_interfaces="lo1 wg0 wg1" \
 ifconfig_lo1="inet 10.0.5.5/32" \
 ifconfig_lo1_ipv6="inet6 2001:db8:5::5 prefixlen 128" \
 ifconfig_wg0="inet 10.9.1.5/24" \
 ifconfig_wg0_ipv6="inet6 2001:db8:91::5 prefixlen 64 auto_linklocal" \
 ifconfig_wg1="inet 10.9.2.5/24" \
 ifconfig_wg1_ipv6="inet6 2001:db8:92::5 prefixlen 64 auto_linklocal" \
 static_routes="ispa ispb" \
 route_ispa="-host 10.0.23.2 10.0.35.3" \
 route_ispb="-host 10.0.24.2 10.0.45.4" \
 frr_enable=YES \
 frr_daemons="mgmtd zebra staticd bfdd ospfd ospf6d"

The WireGuard files use HQ's private key (GHMb2PUmcrykraEdpeMPAoZMGqxyipzKCBEUue/Xp3U=), the branch's public key (2qxqCFgTol1zqt14XTppz83eG/aLbF0O+kM1PwCkWwk=) and the endpoints 10.0.23.2:51820 and 10.0.24.2:51821. In frr.conf, the router-id is 10.0.0.5, lo1 replaces em0 and the network is 10.0.5.5/32. HQ also sets net.route.hash_forward=1, for the traffic it forwards to the branch.

Why net.route.hash_forward

With several equal-cost routes, FreeBSD picks the next-hop from the packet's flow hash (flowid). A router does not compute it: the receiving NIC does, with RSS. Here no packet has one:

  • bhyve's virtio-net has a single queue and stamps every packet with the same queue number;
  • if_wg clears the hash of every packet it decrypts, so even on RSS-capable hardware the traffic coming out of a tunnel has none.

Without a hash, every flow takes the first next-hop and the second ISP stays idle. net.route.hash_forward=1 (BSDRP 2.4 and later) makes the router compute a hash of the addresses, protocol and ports of each forwarded packet that arrives without one; packets that already carry an RSS hash keep it. It defaults to 0 because it costs a hash per forwarded packet.

Testing

ECMP routes

Each router has two OSPF paths to the other side, one per tunnel:

[root@branch]~# vtysh -c "show ip route 10.0.5.5"
Routing entry for 10.0.5.5/32
  Known via "ospf", distance 110, metric 10, best
  Last update 00:01:01 ago
  Flags: Selected
  Status: Installed
  * 10.9.1.5, via wg0, weight 1
  * 10.9.2.5, via wg1, weight 1

[root@branch]~# vtysh -c "show bfd peers brief"
Session count: 4
SessionId  LocalAddress                             PeerAddress                             Status          Profile
=========  ============                             ===========                             ======          =======
367124539  10.9.1.2                                 10.9.1.5                                up              -
4091054727 10.9.2.2                                 10.9.2.5                                up              -
214490437  fe80::5a9c:fcff:fe01:202                 fe80::5a9c:fcff:fe01:505                up              -
2521932787 fe80::5a9c:fcff:fe01:202                 fe80::5a9c:fcff:fe01:505                up              -

The kernel holds both next-hops:

[root@branch]~# netstat -rn | grep 10.0.5.5
10.0.5.5           10.9.1.5           UGH1            wg0
10.0.5.5           10.9.2.5           UGH1            wg1

Flows spread over both ISPs

The PC sends 256 UDP flows to HQ, one per source port, and the branch counts the packets leaving through each tunnel:

[root@pc]~# for p in $(jot 256 20000); do echo x | nc -u -w0 -p $p 10.0.5.5 9; done
net.route.hash_forward wg0 (ISP-A) wg1 (ISP-B)
0 256 0
1 128 128

IPv6 gives the same result. With only a few flows the split is less even: eight iperf3 TCP streams sent 54 MB through wg0 and 29 MB through wg1.

ISP-A fails

Bring down ISP-A's link to HQ while the PC pings HQ every 100 ms:

[root@ispA]~# ifconfig vtnet3 down
[root@pc]~# ping -i 0.1 -c 150 10.0.5.5
...
150 packets transmitted, 142 packets received, 5.3% packet loss

8 lost packets: about 0.8 s, the BFD detection time (3 x 300 ms). wg0 is still up as an interface, but BFD drops its adjacencies and the branch keeps one path:

[root@branch]~# netstat -rn | grep 10.0.5.5
10.0.5.5           10.9.2.5           UGH1            wg1

Only the flows that were using wg0 notice the failure. When ISP-A comes back, OSPF restores the second path and the flows spread again.

Automated test

tools/regression-test.sh <image> branch runs the checks above: the tunnels, OSPF, OSPFv3 and BFD state, the ECMP routes on both routers, the 256-flow spread over IPv4 and IPv6, and the ISP-A failure and recovery.