documentation:examples:pf_and_carp_lab
- en
- fr
This is an old revision of the document!
Table of Contents
pf, pfsync, pflog and carp lab
Network Diagram
Starting the lab
More information on these BSDRP lab scripts available on How to build a BSDRP router lab.
Example with the bhyve lab script (notice that FreeBSD 10.1 virtIO drivers had a bug with carp):
# ./BSDRP-lab-bhyve.sh -i /usr/obj/BSDRP.amd64/BSDRP-1.54-full-amd64-vga.img -n 4 -l 2 BSD Router Project (http://bsdrp.net) - bhyve full-meshed lab script Setting-up a virtual lab with 4 VM(s): - Working directory: /root/BSDRP-VMs - Each VM has a total of 1 (1 cores and 1 threads) and 512M RAM - Emulated NIC: virtio-net - Switch mode: bridge + tap - 2 LAN(s) between all VM - Full mesh Ethernet links between each VM VM 1 has the following NIC: - vtnet0 connected to VM 2 - vtnet1 connected to VM 3 - vtnet2 connected to VM 4 - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 VM 2 has the following NIC: - vtnet0 connected to VM 1 - vtnet1 connected to VM 3 - vtnet2 connected to VM 4 - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 VM 3 has the following NIC: - vtnet0 connected to VM 1 - vtnet1 connected to VM 2 - vtnet2 connected to VM 4 - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 VM 4 has the following NIC: - vtnet0 connected to VM 1 - vtnet1 connected to VM 2 - vtnet2 connected to VM 3 - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 To connect VM'serial console, you can use: - VM 1 : cu -l /dev/nmdm-BSDRP.1B - VM 2 : cu -l /dev/nmdm-BSDRP.2B - VM 3 : cu -l /dev/nmdm-BSDRP.3B - VM 4 : cu -l /dev/nmdm-BSDRP.4B
Configuring Routers
Inside host (VM1)
sysrc hostname=VM1 sysrc ifconfig_vtnet3="inet 192.168.10.1/24" sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:10::1 prefixlen 64" sysrc defaultrouter="192.168.10.254" sysrc ipv6_defaultrouter="2001:db8:10::fe" sysrc gateway_enable=NO sysrc ipv6_gateway_enable=NO config save hostname VM1 service netif restart service routing restart
Master Firewall (VM2)
sysrc hostname=VM2 sysrc ifconfig_vtnet1="inet 192.168.23.2/24" sysrc ifconfig_vtnet3="inet 192.168.10.2/30" sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:10::2 prefixlen 64" sysrc ifconfig_vtnet3_alias0="inet 192.168.10.254/32 vhid 1 advskew 100 pass testpass41" sysrc ifconfig_vtnet3_alias1="inet6 2001:db8:10::fe prefixlen 128 vhid 2 advskew 100 pass testpass61" sysrc ifconfig_vtnet4="inet 2.2.2.2/24" sysrc ifconfig_vtnet4_ipv6="inet6 2001:db8:2:2:2::2 prefixlen 64" sysrc ifconfig_vtnet4_alias0="inet 2.2.2.254/32 vhid 3 advskew 100 pass testpass42" sysrc ifconfig_vtnet4_alias1="inet6 2001:db8:2:2:2::fe prefixlen 128 vhid 4 advskew 100 pass testpass62" sysrc pf_enable=YES sysrc pfsync_enable=YES sysrc pflog_enable=YES sysrc pfsync_syncdev=vtnet1 sysrc kld_list="carp" echo "net.inet.carp.preempt=1" >> /etc/sysctl.conf cat > /etc/pf.conf <<EOF ExtIf="vtnet4" IntIf="vtnet3" SyncIf="vtnet1" # Default block all block # Don't filter on loopback set skip on lo0 # Don't sync carp and pfsync pass quick on \$SyncIf proto pfsync keep state (no-sync) pass quick on \$ExtIf proto carp keep state (no-sync) pass quick on \$IntIf proto carp keep state (no-sync) # Don't block icmpv6 (don't use this large rule in production!) pass proto ipv6-icmp from any to any # Allow traffic from inside to outside pass log from \$IntIf:network to any # Allow traffic from self to any pass log from self to any EOF config save hostname VM2 kldload carp sysctl net.inet.carp.preempt=1 service netif restart service pf start service pfsync start service pflog start
Backup Firewall (VM3)
sysrc hostname=VM3 sysrc ifconfig_vtnet1="inet 192.168.23.3/24" sysrc ifconfig_vtnet3="inet 192.168.10.3/30" sysrc ifconfig_vtnet3_ipv6="inet6 2001:db8:10::3 prefixlen 64" sysrc ifconfig_vtnet3_alias0="inet 192.168.10.254/32 vhid 1 advskew 200 pass testpass41" sysrc ifconfig_vtnet3_alias1="inet6 2001:db8:10::fe prefixlen 128 vhid 2 advskew 200 pass testpass61" sysrc ifconfig_vtnet4="inet 2.2.2.3/24" sysrc ifconfig_vtnet4_ipv6="inet6 2001:db8:2:2:2::3 prefixlen 64" sysrc ifconfig_vtnet4_alias0="inet 2.2.2.254/32 vhid 3 advskew 200 pass testpass42" sysrc ifconfig_vtnet4_alias1="inet6 2001:db8:2:2:2::fe prefixlen 128 vhid 4 advskew 200 pass testpass62" sysrc pf_enable=YES sysrc pfsync_enable=YES sysrc pflog_enable=YES sysrc pfsync_syncdev=vtnet1 sysrc kld_list="carp" echo "net.inet.carp.preempt=1" >> /etc/sysctl.conf cat > /etc/pf.conf <<EOF ExtIf="vtnet4" IntIf="vtnet3" SyncIf="vtnet1" # Default block all block # Don't filter on loopback set skip on lo0 # Don't sync carp and pfsync pass quick on \$SyncIf proto pfsync keep state (no-sync) pass quick on \$ExtIf proto carp keep state (no-sync) pass quick on \$IntIf proto carp keep state (no-sync) # Don't block icmpv6 (don't use this large rule in production!) pass proto ipv6-icmp from any to any # Allow traffic from inside to outside pass log from \$IntIf:network to any # Allow traffic from self to any pass log from self to any EOF config save hostname VM3 kldload carp sysctl net.inet.carp.preempt=1 service netif restart service pf start service pfsync start service pflog start
Outside host (VM4)
sysrc hostname=VM4 sysrc ifconfig_vtnet4="inet 2.2.2.4/24" sysrc ifconfig_vtnet4_ipv6="inet6 2001:db8:2:2:2::4 prefixlen 64" sysrc defaultrouter="2.2.2.254" sysrc ipv6_defaultrouter="2001:db8:2:2:2::fe" sysrc gateway_enable=NO sysrc ipv6_gateway_enable=NO sysrc inetd_enable=YES sed -i -e 's/#echo/echo/g' /etc/inetd.conf hostname VM4 service netif restart service routing restart service inetd start config save
Checking configuration
carp state
Check that VM2 is in carp master state:
[root@VM2]~# ifconfig vtnet3 vtnet3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 58:9c:fc:02:00:02 inet 192.168.10.2 netmask 0xfffffffc broadcast 192.168.10.3 inet 192.168.10.254 netmask 0xffffffff broadcast 192.168.10.254 vhid 1 inet6 fe80::5a9c:fcff:fe02:2%vtnet3 prefixlen 64 scopeid 0x4 inet6 2001:db8:10::2 prefixlen 64 inet6 2001:db8:10::fe prefixlen 128 vhid 2 carp: MASTER vhid 1 advbase 1 advskew 100 carp: MASTER vhid 2 advbase 1 advskew 100 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> [root@VM2]~# ifconfig vtnet4 vtnet4: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 58:9c:fc:02:00:02 inet 2.2.2.2 netmask 0xffffff00 broadcast 2.2.2.255 inet 2.2.2.254 netmask 0xffffffff broadcast 2.2.2.254 vhid 3 inet6 fe80::5a9c:fcff:fe02:2%vtnet4 prefixlen 64 scopeid 0x5 inet6 2001:db8:2:2:2::2 prefixlen 64 inet6 2001:db8:2:2:2::fe prefixlen 128 vhid 4 carp: MASTER vhid 3 advbase 1 advskew 100 carp: MASTER vhid 4 advbase 1 advskew 100 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
And VM3 in backup state:
[root@VM3]~# ifconfig vtnet3 vtnet3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 58:9c:fc:03:00:03 inet 192.168.10.3 netmask 0xfffffffc broadcast 192.168.10.3 inet 192.168.10.254 netmask 0xffffffff broadcast 192.168.10.254 vhid 1 inet6 fe80::5a9c:fcff:fe03:3%vtnet3 prefixlen 64 scopeid 0x4 inet6 2001:db8:10::3 prefixlen 64 inet6 2001:db8:10::fe prefixlen 128 vhid 2 carp: BACKUP vhid 1 advbase 1 advskew 200 carp: BACKUP vhid 2 advbase 1 advskew 200 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> [root@VM3]~# ifconfig vtnet4 vtnet4: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE> ether 58:9c:fc:03:00:03 inet 2.2.2.3 netmask 0xffffff00 broadcast 2.2.2.255 inet 2.2.2.254 netmask 0xffffffff broadcast 2.2.2.254 vhid 3 inet6 fe80::5a9c:fcff:fe03:3%vtnet4 prefixlen 64 scopeid 0x5 inet6 2001:db8:2:2:2::3 prefixlen 64 inet6 2001:db8:2:2:2::fe prefixlen 128 vhid 4 carp: BACKUP vhid 3 advbase 1 advskew 200 carp: BACKUP vhid 4 advbase 1 advskew 200 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pf state
Check the current rules applied:
[root@VM2]~# pfctl -sr block drop all pass quick on vtnet4 proto carp all keep state (no-sync) pass quick on vtnet3 proto carp all keep state (no-sync) pass quick on vtnet1 proto pfsync all keep state (no-sync) pass proto ipv6-icmp all keep state pass log on vtnet3 inet6 from fe80::5a9c:fcff:fe02:2 to any flags S/SA keep state pass log on vtnet4 inet6 from fe80::5a9c:fcff:fe02:2 to any flags S/SA keep state pass log inet6 from 2001:db8:10::/64 to any flags S/SA keep state pass log on vtnet1 inet6 from fe80::5a9c:fcff:fe02:302 to any flags S/SA keep state pass log inet6 from 2001:db8:2:2:2::2 to any flags S/SA keep state pass log inet6 from 2001:db8:2:2:2::fe to any flags S/SA keep state pass log inet6 from ::1 to any flags S/SA keep state pass log on lo0 inet6 from fe80::1 to any flags S/SA keep state pass log inet from <__automatic_8a1ff95a_0> to any flags S/SA keep state
Creating 2 flows from VM1 to VM4
Open a tmux session on R1 and generate 2 flows:
- A continous ping: ping 10.0.0.4
- A echo session: telnet 10.0.0.4 7
pf synchronisation
Now check there are 4 news states (one for each direction) on the Master firewall:
[root@VM3]~# pfctl -ss all icmp 10.0.0.4:267 <- 192.168.10.1:267 0:0 all icmp 192.168.10.1:267 -> 10.0.0.4:267 0:0 all tcp 10.0.0.4:7 <- 192.168.10.1:31058 ESTABLISHED:ESTABLISHED all tcp 192.168.10.1:31058 -> 10.0.0.4:7 ESTABLISHED:ESTABLISHED all carp 224.0.0.18 <- 192.168.10.2 NO_TRAFFIC:SINGLE all carp 224.0.0.18 <- 10.0.0.2 NO_TRAFFIC:SINGLE all pfsync 224.0.0.240 <- 192.168.23.2 NO_TRAFFIC:SINGLE
And these entries are synced to backup firewall:
[root@VM3]~# pfctl -ss all icmp 10.0.0.4:39946 <- 192.168.10.1:39946 0:0 all icmp 192.168.10.1:39946 -> 10.0.0.4:39946 0:0 all tcp 10.0.0.4:22 <- 192.168.10.1:46911 ESTABLISHED:ESTABLISHED all tcp 192.168.10.1:46911 -> 10.0.0.4:22 ESTABLISHED:ESTABLISHED all carp 224.0.0.18 <- 10.0.0.2 NO_TRAFFIC:SINGLE all carp 224.0.0.18 <- 192.168.10.2 NO_TRAFFIC:SINGLE all pfsync 224.0.0.240 <- 192.168.23.2 NO_TRAFFIC:SINGLE
pf log
Wait for the default 60seconds flush timer of pflogd on the MASTER carp firewall, then check log file:
[root@VM2]~# tcpdump -r /var/log/pflog reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file) 17:22:44.851364 IP6 2001:db8:10::1 > 2001:db8:2:2:2::4: ICMP6, echo request, seq 0, length 16 17:22:44.851373 IP6 2001:db8:10::1 > 2001:db8:2:2:2::4: ICMP6, echo request, seq 0, length 16 17:22:44.851385 IP6 fe80::5a9c:fcff:fe02:2 > ff02::1:ff00:4: ICMP6, neighbor solicitation[|icmp6] 17:22:44.851577 IP6 fe80::5a9c:fcff:fe02:2 > ff02::1:ff00:1: ICMP6, neighbor solicitation[|icmp6]
Testing failover
Halt master firewall and check:
- on VM1: no ping lost neither TCP echo session
- on VM3: It became carp master
documentation/examples/pf_and_carp_lab.1637845242.txt.gz · Last modified: 2021/11/25 14:00 by olivier