Table of Contents
Multi-tenant HA pf firewalls
Network Diagram
Starting the lab
More information on these BSDRP lab scripts available on How to build a BSDRP router lab.
Example with the bhyve lab script:
# tools/BSDRP-lab-bhyve.sh -i BSDRP.amd64/BSDRP-1.80-full-amd64-serial.img.xz -n 5 -l 2 BSD Router Project (http://bsdrp.net) - bhyve full-meshed lab script Setting-up a virtual lab with 5 VM(s): - Working directory: /tmp/BSDRP - Each VM have 1 core(s) and 256M RAM - Emulated NIC: virtio-net - Switch mode: bridge + tap - 3 LAN(s) between all VM - Full mesh Ethernet links between each VM VM 1 have the following NIC: - vtnet0 connected to VM 2 - vtnet1 connected to VM 3 - vtnet2 connected to VM 4 - vtnet3 connected to VM 5 - vtnet4 connected to LAN number 1 - vtnet5 connected to LAN number 2 VM 2 have the following NIC: - vtnet0 connected to VM 1 - vtnet1 connected to VM 3 - vtnet2 connected to VM 4 - vtnet3 connected to VM 5 - vtnet4 connected to LAN number 1 - vtnet5 connected to LAN number 2 VM 3 have the following NIC: - vtnet0 connected to VM 1 - vtnet1 connected to VM 2 - vtnet2 connected to VM 4 - vtnet3 connected to VM 5 - vtnet4 connected to LAN number 1 - vtnet5 connected to LAN number 2 VM 4 have the following NIC: - vtnet0 connected to VM 1 - vtnet1 connected to VM 2 - vtnet2 connected to VM 3 - vtnet3 connected to VM 5 - vtnet4 connected to LAN number 1 - vtnet5 connected to LAN number 2 VM 5 have the following NIC: - vtnet0 connected to VM 1 - vtnet1 connected to VM 2 - vtnet2 connected to VM 3 - vtnet3 connected to VM 4 - vtnet4 connected to LAN number 1 - vtnet5 connected to LAN number 2 For connecting to VM'serial console, you can use: - VM 1 : cu -l /dev/nmdm-BSDRP.1B - VM 2 : cu -l /dev/nmdm-BSDRP.2B - VM 3 : cu -l /dev/nmdm-BSDRP.3B - VM 4 : cu -l /dev/nmdm-BSDRP.4B - VM 5 : cu -l /dev/nmdm-BSDRP.5B
Configuring Routers
With BSDRP's labconfig
All these routers can be rapidly configured with BSDRP's labconfig tool (use it only on a lab, because it will replace your current running configuration). Notice this example is using another BSDRP'shell script to simplify nullfs jail creation on a nanobsd.
labconfig jailpf_vm[VM-NUMBER]
Or you can do it step-by-step like described.
Public server (VM3)
sysrc hostname=VM3 hostname VM3 sysrc ifconfig_vtnet4="inet 2.2.2.3/24" sysrc -x gateway_enable sysrc -x ipv6_gateway_enable sysrc inetd_enable=YES sed -i -e 's/#echo/echo/g' /etc/inetd.conf service netif restart service routing restart service inetd start config save
Customer 1 workstation (VM4)
sysrc hostname=VM4 hostname VM4 sysrc ifconfig_vtnet5="up" sysrc vlans_vtnet5="1" sysrc ifconfig_vtnet5_1="inet 10.0.0.4/24" sysrc defaultrouter="10.0.0.254" sysrc -x gateway_enable sysrc -x ipv6_gateway_enable service netif restart service routing restart config save
Customer 2 workstation (VM5)
sysrc hostname=VM5 hostname VM5 sysrc ifconfig_vtnet5="up" sysrc vlans_vtnet5="2" sysrc ifconfig_vtnet5_2="inet 10.0.0.5/24" sysrc defaultrouter="10.0.0.254" sysrc -x gateway_enable sysrc -x ipv6_gateway_enable service netif restart service routing restart config save
First Multi-tenant firewall (VM1)
sysrc hostname=VM1 sysrc cloned_interfaces="bridge0" sysrc ifconfig_bridge0="addm vtnet4" sysrc ifconfig_vtnet4="up" sysrc ifconfig_vtnet5="up" sysrc vlans_vtnet5="1 2" sysrc ifconfig_vtnet5_1="up" sysrc ifconfig_vtnet5_2="up" sysrc kld_list+=" carp pf pflog pfsync" cat > /etc/devfs.rules <<'EOF' [devfsrules_jailpf=4] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path 'pf' unhide add path 'pflog*' unhide add path 'bpf*' unhide 'EOF' hostname VM1 service devfs restart service netif restart service kld start
Jails needs to use different CARP vid on the public shared LAN for avoid carp MAC address conflict
Customer 1 Firewall1 (jail11)
tenant -c -j jail11 -i bridge0,vtnet5.1 cat > /etc/jails/jail11/rc.conf <<EOF hostname="jail11" sshd_enable=YES gateway_enable=YES ipv6_gateway_enable=YES ifconfig_vtnet5_1="inet 10.0.0.252/24" ifconfig_vtnet5_1_alias0="inet 10.0.0.254/32 vhid 1 advskew 100 pass customer1i" ifconfig_epair10b="inet 2.2.2.11/24" ifconfig_epair10b_alias0="inet 2.2.2.1/32 vhid 2 advskew 100 pass customer1e" pf_enable=YES pflog_enable=YES pfsync_enable=YES pfsync_syncdev=vtnet5.1 EOF echo "net.inet.carp.preempt=1" >> /etc/jails/jail11/sysctl.conf cat > /etc/jails/jail11/pf.conf <<EOF nat on epair10b from vtnet5.1:network to any -> 2.2.2.1 block pass quick on vtnet5.1 proto pfsync keep state (no-sync) pass quick on epair10b proto carp keep state (no-sync) pass quick on vtnet5.1 proto carp keep state (no-sync) pass log from vtnet5.1:network to epair10b:network pass log from self to any EOF
Customer 2 Firewall1 (jail12)
tenant -c -j jail12 -i bridge0,vtnet5.2 cat > /etc/jails/jail12/rc.conf <<EOF hostname=jail12 sshd_enable=YES gateway_enable=YES ipv6_gateway_enable=YES ifconfig_vtnet5_2="inet 10.0.0.252/24" ifconfig_vtnet5_2_alias0="inet 10.0.0.254/32 vhid 3 advskew 200 pass customer2i" ifconfig_epair20b="inet 2.2.2.12/24" ifconfig_epair20b_alias0="inet 2.2.2.2/32 vhid 4 advskew 200 pass customer2e" pf_enable=YES pflog_enable=YES pfsync_enable=YES pfsync_syncdev=vtnet5.2 EOF echo "net.inet.carp.preempt=1" >> /etc/jails/jail12/sysctl.conf cat > /etc/jails/jail12/pf.conf <<EOF nat on epair20b from vtnet5.2:network to any -> 2.2.2.2 block pass quick on vtnet5.2 proto pfsync keep state (no-sync) pass quick on epair20b proto carp keep state (no-sync) pass quick on vtnet5.2 proto carp keep state (no-sync) pass log from vtnet5.2:network to epair20b:network pass log from self to any EOF
Starting customer firewall
service jail start
Second Multi-tenant firewall (VM2)
sysrc hostname=VM2 sysrc cloned_interfaces="bridge0" sysrc ifconfig_bridge0="addm vtnet4" sysrc ifconfig_vtnet4="up" sysrc ifconfig_vtnet5="up" sysrc vlans_vtnet5="1 2" sysrc ifconfig_vtnet5_1="up" sysrc ifconfig_vtnet5_2="up" sysrc kld_list+=" carp pf pflog pfsync" cat > /etc/devfs.rules <<'EOF' [devfsrules_jailpf=4] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path 'pf' unhide add path 'pflog*' unhide add path 'bpf*' unhide 'EOF' hostname VM2 service devfs restart service netif restart service kld start
Jails needs to use different CARP vid on the public shared LAN for avoid carp MAC address conflict
Customer 1 Firewall2 (jail21)
tenant -c -j jail21 -i bridge0,vtnet5.1 cat > /etc/jails/jail21/rc.conf <<EOF hostname=jail21 sshd_enable=YES gateway_enable=YES ipv6_gateway_enable=YES ifconfig_vtnet5_1="inet 10.0.0.253/24" ifconfig_vtnet5_1_alias0="inet 10.0.0.254/32 vhid 1 advskew 200 pass customer1i" ifconfig_epair10b="inet 2.2.2.21/24" ifconfig_epair10b_alias0="inet 2.2.2.1/32 vhid 2 advskew 200 pass customer1e" pf_enable=YES pflog_enable=YES pfsync_enable=YES pfsync_syncdev=vtnet5.1 EOF echo "net.inet.carp.preempt=1" >> /etc/jails/jail21/sysctl.conf cat > /etc/jails/jail21/pf.conf <<EOF nat on epair10b from vtnet5.1:network to any -> 2.2.2.1 block pass quick on vtnet5.1 proto pfsync keep state (no-sync) pass quick on epair10b proto carp keep state (no-sync) pass quick on vtnet5.1 proto carp keep state (no-sync) pass log from vtnet5.1:network to epair10b:network pass log from self to any EOF
Customer 2 Firewall2 (jail22)
tenant -c -j jail22 -i bridge0,vtnet5.2 cat > /etc/jails/jail22/rc.conf <<EOF hostname=jail22 sshd_enable=YES gateway_enable=YES ipv6_gateway_enable=YES ifconfig_vtnet5_2="inet 10.0.0.253/24" ifconfig_vtnet5_2_alias0="inet 10.0.0.254/32 vhid 3 advskew 100 pass customer2i" ifconfig_epair20b="inet 2.2.2.22/24" ifconfig_epair20b_alias0="inet 2.2.2.2/32 vhid 4 advskew 100 pass customer2e" pf_enable=YES pflog_enable=YES pfsync_enable=YES pfsync_syncdev=vtnet5.2 EOF echo "net.inet.carp.preempt=1" >> /etc/jails/jail22/sysctl.conf cat > /etc/jails/jail22/pf.conf <<EOF nat on epair20b from vtnet5.2:network to any -> 2.2.2.2 block pass quick on vtnet5.2 proto pfsync keep state (no-sync) pass quick on epair20b proto carp keep state (no-sync) pass quick on vtnet5.2 proto carp keep state (no-sync) pass log from vtnet5.2:network to epair20b:network pass log from self to any EOF
Starting customer firewall
service jail start
Checking firewalls status
carp state
Customer 1
Check that on VM1 jail11 is in carp master state:
[root@VM1]~# jexec jail11 ifconfig | grep carp
carp: MASTER vhid 2 advbase 1 advskew 100
carp: MASTER vhid 1 advbase 1 advskew 100
And on VM2 that jail21 in backup state:
[root@VM2]~# jexec jail21 ifconfig | grep carp
carp: BACKUP vhid 2 advbase 1 advskew 200
carp: BACKUP vhid 1 advbase 1 advskew 200
Customer 2
Check on VM1 that jail12 is in carp backup state:
[root@VM1]~# jexec jail12 ifconfig | grep carp
carp: BACKUP vhid 2 advbase 1 advskew 200
carp: BACKUP vhid 1 advbase 1 advskew 200
And on VM2 that jail22 in master state:
[root@VM2]~# jexec jail22 ifconfig | grep carp
carp: MASTER vhid 2 advbase 1 advskew 100
carp: MASTER vhid 1 advbase 1 advskew 100
pfsync
Customer 1
Generate a flow from VM4 (customer 1 workstation) to VM3 (Internet server):
[root@VM4]~# telnet 2.2.2.3 7 Trying 2.2.2.3... Connected to 2.2.2.3. Escape character is '^]'. echo echo
And still connected, check state tables on jail11:
[root@VM1]~# jexec jail11 pfctl -ss all carp 10.0.0.252 -> 224.0.0.18 SINGLE:NO_TRAFFIC all carp 2.2.2.11 -> 224.0.0.18 SINGLE:NO_TRAFFIC all carp 224.0.0.18 <- 2.2.2.22 NO_TRAFFIC:SINGLE all tcp 2.2.2.3:7 <- 10.0.0.4:22829 ESTABLISHED:ESTABLISHED all tcp 2.2.2.1:64414 (10.0.0.4:22829) -> 2.2.2.3:7 ESTABLISHED:ESTABLISHED
And state table should be synced on jail21 too:
[root@VM2]~# jexec jail21 pfctl -ss all tcp 2.2.2.3:7 <- 10.0.0.4:22829 ESTABLISHED:ESTABLISHED all tcp 2.2.2.1:64414 (10.0.0.4:22829) -> 2.2.2.3:7 ESTABLISHED:ESTABLISHED all carp 224.0.0.18 <- 10.0.0.252 NO_TRAFFIC:SINGLE all carp 224.0.0.18 <- 2.2.2.11 NO_TRAFFIC:SINGLE all carp 224.0.0.18 <- 2.2.2.22 NO_TRAFFIC:SINGLE
Customer 2
Generate a flow from VM5 (customer 2 workstation) to VM3 (Internet server):
[root@VM5]~# telnet 2.2.2.3 7 Trying 2.2.2.3... Connected to 2.2.2.3. Escape character is '^]'. echo echo
And still connected, check state tables on VM2/jail22:
[root@VM2]~# jexec jail22 pfctl -ss all carp 224.0.0.18 <- 2.2.2.11 NO_TRAFFIC:SINGLE all carp 10.0.0.253 -> 224.0.0.18 SINGLE:NO_TRAFFIC all carp 2.2.2.22 -> 224.0.0.18 SINGLE:NO_TRAFFIC all tcp 2.2.2.3:7 <- 10.0.0.5:48257 ESTABLISHED:ESTABLISHED all tcp 2.2.2.2:55134 (10.0.0.5:48257) -> 2.2.2.3:7 ESTABLISHED:ESTABLISHED
And state table should be synced on VM1/jail12 too:
[root@VM1]~# jexec jail12 pfctl -ss all carp 224.0.0.18 <- 2.2.2.11 NO_TRAFFIC:SINGLE all carp 224.0.0.18 <- 10.0.0.253 NO_TRAFFIC:SINGLE all carp 224.0.0.18 <- 2.2.2.22 NO_TRAFFIC:SINGLE all pfsync 224.0.0.240 <- 10.0.0.253 NO_TRAFFIC:SINGLE all tcp 2.2.2.3:7 <- 10.0.0.5:29481 ESTABLISHED:ESTABLISHED all tcp 2.2.2.2:58530 (10.0.0.5:29481) -> 2.2.2.3:7 ESTABLISHED:ESTABLISHED
pflog
Customer 1
Check log file on customer 1 master firewall (after default 60 seconds timer flush):
[root@jail11]~# tcpdump -n -e -ttt -i pflog0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 262144 bytes 00:00:00.000000 rule 4/0(match): pass in on vtnet5.1: 10.0.0.4 > 2.2.2.3: ICMP echo request, id 59656, seq 0, length 64 00:00:00.000016 rule 7/0(match): pass out on epair1b: 2.2.2.1 > 2.2.2.3: ICMP echo request, id 59599, seq 0, length 64 00:00:03.602014 rule 4/0(match): pass in on vtnet5.1: 10.0.0.4.40843 > 2.2.2.3.7: Flags [S], seq 3135529453, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 24986171 ecr 0], length 0 00:00:00.000009 rule 7/0(match): pass out on epair1b: 2.2.2.1.53035 > 2.2.2.3.7: Flags [S], seq 3135529453, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 24986171 ecr 0], length 0 ^C 4 packets captured 4 packets received by filter 0 packets dropped by kernel
Customer 2
[root@jail22]~# tcpdump -n -e -ttt -i pflog0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 262144 bytes 00:00:00.000000 rule 4/0(match): pass in on vtnet5.2: 10.0.0.5 > 2.2.2.3: ICMP echo request, id 64776, seq 0, length 64 00:00:00.000008 rule 7/0(match): pass out on epair2b: 2.2.2.2 > 2.2.2.3: ICMP echo request, id 11316, seq 0, length 64 00:00:02.654863 rule 4/0(match): pass in on vtnet5.2: 10.0.0.5.20739 > 2.2.2.3.7: Flags [S], seq 2458333239, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 24913037 ecr 0], length 0 00:00:00.000008 rule 7/0(match): pass out on epair2b: 2.2.2.2.56545 > 2.2.2.3.7: Flags [S], seq 2458333239, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 24913037 ecr 0], length 0 ^C 4 packets captured 4 packets received by filter 0 packets dropped by kernel
pflogd
Customer 1
Check log file on customer 1 master firewall jail11 (after default 60 seconds timer flush):
[root@jail11]~# tcpdump -r /var/log/pflog reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file)
Nothing??
[root@jail11]~# service pflogd status pflogd does not exist in /etc/rc.d or the local startup directories (/usr/local/etc/rc.d), or is not executable [root@jail11]~# service pflog status pflog is running as pid 2267.
Customer 2
Check log file on customer 2 master firewall jail22 (after default 60 seconds timer flush):
[root@jail22]~# tcpdump -r /var/log/pflog reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file) [root@jail22]~# service pflog status pflog is running as pid 2261.
pflogd seems to have a problem running inside a jail
Trying to stop pflogd for forcing a flush, but can't stop it:
[root@jail11]~# ps -auxww | grep pflogd _pflogd 2269 49.6 1.3 12304 2872 - RJ 09:13 10:06.89 pflogd: [running] -s 116 -i pflog0 -f /var/log/pflog (pflogd) root 2267 0.0 1.3 12236 2868 - IsJ 09:13 0:00.00 pflogd: [priv] (pflogd) root 3020 0.0 0.1 420 300 0 R+J 09:33 0:00.00 grep pflogd [root@jail11]~# service pflog stop Stopping pflog. Waiting for PIDS: 2267 load: 2.00 cmd: pwait 3032 [kqread] 28.06r 0.00u 0.00s 0% 1488k