documentation:examples:pf_and_carp_lab
no way to compare when less than two revisions
Differences
This shows you the differences between two versions of the page.
Previous revision | |||
— | documentation:examples:pf_and_carp_lab [2021/11/25 14:04] (current) – [pf synchronisation] olivier | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | ====== pf, pfsync, pflog and carp lab ====== | ||
+ | ===== Network Diagram ===== | ||
+ | |||
+ | {{: | ||
+ | |||
+ | ===== Starting the lab ===== | ||
+ | |||
+ | More information on these BSDRP lab scripts available on [[documentation: | ||
+ | |||
+ | Example with the bhyve lab script (notice that FreeBSD 10.1 virtIO drivers had a bug with carp): | ||
+ | |||
+ | < | ||
+ | # ./ | ||
+ | BSD Router Project (http:// | ||
+ | Setting-up a virtual lab with 4 VM(s): | ||
+ | - Working directory: / | ||
+ | - Each VM has a total of 1 (1 cores and 1 threads) and 512M RAM | ||
+ | - Emulated NIC: virtio-net | ||
+ | - Switch mode: bridge + tap | ||
+ | - 2 LAN(s) between all VM | ||
+ | - Full mesh Ethernet links between each VM | ||
+ | VM 1 has the following NIC: | ||
+ | - vtnet0 connected to VM 2 | ||
+ | - vtnet1 connected to VM 3 | ||
+ | - vtnet2 connected to VM 4 | ||
+ | - vtnet3 connected to LAN number 1 | ||
+ | - vtnet4 connected to LAN number 2 | ||
+ | VM 2 has the following NIC: | ||
+ | - vtnet0 connected to VM 1 | ||
+ | - vtnet1 connected to VM 3 | ||
+ | - vtnet2 connected to VM 4 | ||
+ | - vtnet3 connected to LAN number 1 | ||
+ | - vtnet4 connected to LAN number 2 | ||
+ | VM 3 has the following NIC: | ||
+ | - vtnet0 connected to VM 1 | ||
+ | - vtnet1 connected to VM 2 | ||
+ | - vtnet2 connected to VM 4 | ||
+ | - vtnet3 connected to LAN number 1 | ||
+ | - vtnet4 connected to LAN number 2 | ||
+ | VM 4 has the following NIC: | ||
+ | - vtnet0 connected to VM 1 | ||
+ | - vtnet1 connected to VM 2 | ||
+ | - vtnet2 connected to VM 3 | ||
+ | - vtnet3 connected to LAN number 1 | ||
+ | - vtnet4 connected to LAN number 2 | ||
+ | To connect VM' | ||
+ | - VM 1 : cu -l / | ||
+ | - VM 2 : cu -l / | ||
+ | - VM 3 : cu -l / | ||
+ | - VM 4 : cu -l / | ||
+ | </ | ||
+ | |||
+ | ===== Configuring Routers ===== | ||
+ | |||
+ | ==== Inside host (VM1) ==== | ||
+ | |||
+ | < | ||
+ | sysrc hostname=VM1 | ||
+ | sysrc ifconfig_vtnet3=" | ||
+ | sysrc ifconfig_vtnet3_ipv6=" | ||
+ | sysrc defaultrouter=" | ||
+ | sysrc ipv6_defaultrouter=" | ||
+ | sysrc gateway_enable=NO | ||
+ | sysrc ipv6_gateway_enable=NO | ||
+ | config save | ||
+ | hostname VM1 | ||
+ | service netif restart | ||
+ | service routing restart | ||
+ | </ | ||
+ | |||
+ | ==== Master Firewall (VM2) ==== | ||
+ | |||
+ | < | ||
+ | sysrc hostname=VM2 | ||
+ | sysrc ifconfig_vtnet1=" | ||
+ | sysrc ifconfig_vtnet3=" | ||
+ | sysrc ifconfig_vtnet3_ipv6=" | ||
+ | sysrc ifconfig_vtnet3_alias0=" | ||
+ | sysrc ifconfig_vtnet3_alias1=" | ||
+ | sysrc ifconfig_vtnet4=" | ||
+ | sysrc ifconfig_vtnet4_ipv6=" | ||
+ | sysrc ifconfig_vtnet4_alias0=" | ||
+ | sysrc ifconfig_vtnet4_alias1=" | ||
+ | sysrc pf_enable=YES | ||
+ | sysrc pfsync_enable=YES | ||
+ | sysrc pflog_enable=YES | ||
+ | sysrc pfsync_syncdev=vtnet1 | ||
+ | sysrc kld_list=" | ||
+ | echo " | ||
+ | |||
+ | cat > / | ||
+ | ExtIf=" | ||
+ | IntIf=" | ||
+ | SyncIf=" | ||
+ | # Default block all | ||
+ | block | ||
+ | # Don't filter on loopback | ||
+ | set skip on lo0 | ||
+ | # Don't sync carp and pfsync | ||
+ | pass quick on \$SyncIf proto pfsync keep state (no-sync) | ||
+ | pass quick on \$ExtIf | ||
+ | pass quick on \$IntIf | ||
+ | # Don't block icmpv6 (don't use this large rule in production!) | ||
+ | pass proto ipv6-icmp from any to any | ||
+ | # Allow traffic from inside to outside | ||
+ | pass log from \$IntIf: | ||
+ | # Allow traffic from self to any | ||
+ | pass log from self to any | ||
+ | EOF | ||
+ | |||
+ | config save | ||
+ | hostname VM2 | ||
+ | kldload carp | ||
+ | sysctl net.inet.carp.preempt=1 | ||
+ | service netif restart | ||
+ | service pf start | ||
+ | service pfsync start | ||
+ | service pflog start | ||
+ | </ | ||
+ | |||
+ | ==== Backup Firewall (VM3) ==== | ||
+ | |||
+ | < | ||
+ | sysrc hostname=VM3 | ||
+ | sysrc ifconfig_vtnet1=" | ||
+ | sysrc ifconfig_vtnet3=" | ||
+ | sysrc ifconfig_vtnet3_ipv6=" | ||
+ | sysrc ifconfig_vtnet3_alias0=" | ||
+ | sysrc ifconfig_vtnet3_alias1=" | ||
+ | sysrc ifconfig_vtnet4=" | ||
+ | sysrc ifconfig_vtnet4_ipv6=" | ||
+ | sysrc ifconfig_vtnet4_alias0=" | ||
+ | sysrc ifconfig_vtnet4_alias1=" | ||
+ | sysrc pf_enable=YES | ||
+ | sysrc pfsync_enable=YES | ||
+ | sysrc pflog_enable=YES | ||
+ | sysrc pfsync_syncdev=vtnet1 | ||
+ | sysrc kld_list=" | ||
+ | echo " | ||
+ | |||
+ | cat > / | ||
+ | ExtIf=" | ||
+ | IntIf=" | ||
+ | SyncIf=" | ||
+ | # Default block all | ||
+ | block | ||
+ | # Don't filter on loopback | ||
+ | set skip on lo0 | ||
+ | # Don't sync carp and pfsync | ||
+ | pass quick on \$SyncIf proto pfsync keep state (no-sync) | ||
+ | pass quick on \$ExtIf | ||
+ | pass quick on \$IntIf | ||
+ | # Don't block icmpv6 (don't use this large rule in production!) | ||
+ | pass proto ipv6-icmp from any to any | ||
+ | # Allow traffic from inside to outside | ||
+ | pass log from \$IntIf: | ||
+ | # Allow traffic from self to any | ||
+ | pass log from self to any | ||
+ | EOF | ||
+ | |||
+ | config save | ||
+ | hostname VM3 | ||
+ | kldload carp | ||
+ | sysctl net.inet.carp.preempt=1 | ||
+ | service netif restart | ||
+ | service pf start | ||
+ | service pfsync start | ||
+ | service pflog start | ||
+ | </ | ||
+ | ==== Outside host (VM4) ==== | ||
+ | |||
+ | < | ||
+ | sysrc hostname=VM4 | ||
+ | sysrc ifconfig_vtnet4=" | ||
+ | sysrc ifconfig_vtnet4_ipv6=" | ||
+ | sysrc defaultrouter=" | ||
+ | sysrc ipv6_defaultrouter=" | ||
+ | sysrc gateway_enable=NO | ||
+ | sysrc ipv6_gateway_enable=NO | ||
+ | sysrc inetd_enable=YES | ||
+ | sed -i -e ' | ||
+ | hostname VM4 | ||
+ | service netif restart | ||
+ | service routing restart | ||
+ | service inetd start | ||
+ | config save | ||
+ | </ | ||
+ | |||
+ | ===== Checking configuration ===== | ||
+ | |||
+ | ==== carp state ==== | ||
+ | |||
+ | Check that VM2 is in carp master state: | ||
+ | |||
+ | < | ||
+ | [root@VM2]~# | ||
+ | vtnet3: flags=8863< | ||
+ | options=80028< | ||
+ | ether 58: | ||
+ | inet 192.168.10.2 netmask 0xfffffffc broadcast 192.168.10.3 | ||
+ | inet 192.168.10.254 netmask 0xffffffff broadcast 192.168.10.254 vhid 1 | ||
+ | inet6 fe80:: | ||
+ | inet6 2001: | ||
+ | inet6 2001: | ||
+ | carp: MASTER vhid 1 advbase 1 advskew 100 | ||
+ | carp: MASTER vhid 2 advbase 1 advskew 100 | ||
+ | media: Ethernet autoselect (10Gbase-T < | ||
+ | status: active | ||
+ | nd6 options=21< | ||
+ | [root@VM2]~# | ||
+ | vtnet4: flags=8863< | ||
+ | options=80028< | ||
+ | ether 58: | ||
+ | inet 2.2.2.2 netmask 0xffffff00 broadcast 2.2.2.255 | ||
+ | inet 2.2.2.254 netmask 0xffffffff broadcast 2.2.2.254 vhid 3 | ||
+ | inet6 fe80:: | ||
+ | inet6 2001: | ||
+ | inet6 2001: | ||
+ | carp: MASTER vhid 3 advbase 1 advskew 100 | ||
+ | carp: MASTER vhid 4 advbase 1 advskew 100 | ||
+ | media: Ethernet autoselect (10Gbase-T < | ||
+ | status: active | ||
+ | nd6 options=21< | ||
+ | </ | ||
+ | |||
+ | And VM3 in backup state: | ||
+ | |||
+ | < | ||
+ | [root@VM3]~# | ||
+ | vtnet3: flags=8863< | ||
+ | options=80028< | ||
+ | ether 58: | ||
+ | inet 192.168.10.3 netmask 0xfffffffc broadcast 192.168.10.3 | ||
+ | inet 192.168.10.254 netmask 0xffffffff broadcast 192.168.10.254 vhid 1 | ||
+ | inet6 fe80:: | ||
+ | inet6 2001: | ||
+ | inet6 2001: | ||
+ | carp: BACKUP vhid 1 advbase 1 advskew 200 | ||
+ | carp: BACKUP vhid 2 advbase 1 advskew 200 | ||
+ | media: Ethernet autoselect (10Gbase-T < | ||
+ | status: active | ||
+ | nd6 options=21< | ||
+ | [root@VM3]~# | ||
+ | vtnet4: flags=8863< | ||
+ | options=80028< | ||
+ | ether 58: | ||
+ | inet 2.2.2.3 netmask 0xffffff00 broadcast 2.2.2.255 | ||
+ | inet 2.2.2.254 netmask 0xffffffff broadcast 2.2.2.254 vhid 3 | ||
+ | inet6 fe80:: | ||
+ | inet6 2001: | ||
+ | inet6 2001: | ||
+ | carp: BACKUP vhid 3 advbase 1 advskew 200 | ||
+ | carp: BACKUP vhid 4 advbase 1 advskew 200 | ||
+ | media: Ethernet autoselect (10Gbase-T < | ||
+ | status: active | ||
+ | nd6 options=21< | ||
+ | </ | ||
+ | ==== pf state ==== | ||
+ | |||
+ | Check the current rules applied: | ||
+ | |||
+ | < | ||
+ | [root@VM2]~# | ||
+ | block drop all | ||
+ | pass quick on vtnet4 proto carp all keep state (no-sync) | ||
+ | pass quick on vtnet3 proto carp all keep state (no-sync) | ||
+ | pass quick on vtnet1 proto pfsync all keep state (no-sync) | ||
+ | pass proto ipv6-icmp all keep state | ||
+ | pass log on vtnet3 inet6 from fe80:: | ||
+ | pass log on vtnet4 inet6 from fe80:: | ||
+ | pass log inet6 from 2001: | ||
+ | pass log on vtnet1 inet6 from fe80:: | ||
+ | pass log inet6 from 2001: | ||
+ | pass log inet6 from 2001: | ||
+ | pass log inet6 from ::1 to any flags S/SA keep state | ||
+ | pass log on lo0 inet6 from fe80::1 to any flags S/SA keep state | ||
+ | pass log inet from < | ||
+ | </ | ||
+ | ===== Creating 2 flows from VM1 to VM4 ===== | ||
+ | |||
+ | Open a tmux session on R1 and generate 2 flows: | ||
+ | - A continous ping: ping 2.2.2.4 | ||
+ | - A echo session: telnet 2.2.2.4 7 | ||
+ | |||
+ | ==== pf synchronisation ==== | ||
+ | |||
+ | Now check there are 4 news states (one for each direction) on the Master firewall: | ||
+ | |||
+ | < | ||
+ | [root@VM2]~# | ||
+ | all carp fe80:: | ||
+ | all carp 2.2.2.2 -> 224.0.0.18 | ||
+ | all carp 192.168.10.2 -> 224.0.0.18 | ||
+ | all pfsync 192.168.23.2 -> 224.0.0.240 | ||
+ | all icmp 2.2.2.4: | ||
+ | all icmp 192.168.10.1: | ||
+ | all tcp 2.2.2.4:7 <- 192.168.10.1: | ||
+ | all tcp 192.168.10.1: | ||
+ | </ | ||
+ | |||
+ | And these entries are synced to backup firewall: | ||
+ | |||
+ | < | ||
+ | [root@VM3]~# | ||
+ | all carp 224.0.0.18 <- 192.168.10.2 | ||
+ | all carp 224.0.0.18 <- 2.2.2.2 | ||
+ | all carp ff02::12 <- fe80:: | ||
+ | all pfsync 192.168.23.3 -> 224.0.0.240 | ||
+ | all pfsync 224.0.0.240 <- 192.168.23.2 | ||
+ | all icmp 2.2.2.4: | ||
+ | all icmp 192.168.10.1: | ||
+ | all tcp 2.2.2.4:7 <- 192.168.10.1: | ||
+ | all tcp 192.168.10.1: | ||
+ | </ | ||
+ | |||
+ | ==== pf log ==== | ||
+ | |||
+ | Wait for the default 60seconds flush timer of pflogd on the MASTER carp firewall, then check log file: | ||
+ | < | ||
+ | [root@VM2]~# | ||
+ | reading from file / | ||
+ | 17: | ||
+ | 17: | ||
+ | 17: | ||
+ | 17: | ||
+ | </ | ||
+ | |||
+ | |||
+ | ==== Testing failover ==== | ||
+ | |||
+ | Halt master firewall and check: | ||
+ | - on VM1: no ping lost neither TCP echo session | ||
+ | - on VM3: It became carp master |
documentation/examples/pf_and_carp_lab.txt · Last modified: 2021/11/25 14:04 by olivier