User Tools

Site Tools


documentation:examples:pf_and_carp_lab

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

documentation:examples:pf_and_carp_lab [2017/07/07 00:52] (current)
Line 1: Line 1:
 +====== pf, pfsync, pflog and carp lab  ======
  
 +===== Network Diagram =====
 +
 +{{:​documentation:​examples:​bsdrp.labs.pf-carp.png}}
 +
 +===== Starting the lab =====
 +
 +More information on these BSDRP lab scripts available on [[documentation:​examples:​How to build a BSDRP router lab]].
 +
 +Example with the bhyve lab script (notice that FreeBSD 10.1 virtIO drivers had a bug with carp):
 +
 +<​code>​
 +# ./​BSDRP-lab-bhyve.sh -i /​usr/​obj/​BSDRP.amd64/​BSDRP-1.54-full-amd64-vga.img -n 4 -l 2
 +BSD Router Project (http://​bsdrp.net) - bhyve full-meshed lab script
 +Setting-up a virtual lab with 4 VM(s):
 +- Working directory: /tmp/BSDRP
 +- Each VM have 1 core(s) and 256M RAM
 +- 2 LAN(s) between all VM
 +- Full mesh Ethernet links between each VM
 +VM 1 have the following NIC:
 +- vtnet0 connected to VM 2.
 +- vtnet1 connected to VM 3.
 +- vtnet2 connected to VM 4.
 +- vtnet3 connected to LAN number 1
 +- vtnet4 connected to LAN number 2
 +VM 2 have the following NIC:
 +- vtnet0 connected to VM 1.
 +- vtnet1 connected to VM 3.
 +- vtnet2 connected to VM 4.
 +- vtnet3 connected to LAN number 1
 +- vtnet4 connected to LAN number 2
 +VM 3 have the following NIC:
 +- vtnet0 connected to VM 1.
 +- vtnet1 connected to VM 2.
 +- vtnet2 connected to VM 4.
 +- vtnet3 connected to LAN number 1
 +- vtnet4 connected to LAN number 2
 +VM 4 have the following NIC:
 +- vtnet0 connected to VM 1.
 +- vtnet1 connected to VM 2.
 +- vtnet2 connected to VM 3.
 +- vtnet3 connected to LAN number 1
 +- vtnet4 connected to LAN number 2
 +For connecting to VM'​serial console, you can use:
 +- VM 1 : cu -l /dev/nmdm1B
 +- VM 2 : cu -l /dev/nmdm2B
 +- VM 3 : cu -l /dev/nmdm3B
 +- VM 4 : cu -l /dev/nmdm4B
 +</​code>​
 +
 +===== Configuring Routers =====
 +
 +==== Inside host (VM1) ====
 +
 +<​code>​
 +sysrc hostname=VM1
 +sysrc ifconfig_vtnet3="​inet 192.168.10.1/​24"​
 +sysrc ifconfig_vtnet3_ipv6="​inet6 2001:​db8:​10::​1 prefixlen 64"
 +sysrc defaultrouter="​192.168.10.254"​
 +sysrc ipv6_defaultrouter="​2001:​db8:​10::​fe"​
 +sysrc gateway_enable=NO
 +sysrc ipv6_gateway_enable=NO
 +config save
 +hostname VM1
 +service netif restart
 +service routing restart
 +</​code>​
 +
 +==== Master Firewall (VM2) ====
 +
 +<​code>​
 +sysrc hostname=VM2
 +sysrc ifconfig_vtnet1="​inet 192.168.23.2/​24"​
 +sysrc ifconfig_vtnet3="​inet 192.168.10.2/​30"​
 +sysrc ifconfig_vtnet3_ipv6="​inet6 2001:​db8:​10::​2 prefixlen 64"
 +sysrc ifconfig_vtnet3_alias0="​inet 192.168.10.254/​32 vhid 1 advskew 100 pass testpass41"​
 +sysrc ifconfig_vtnet3_alias1="​inet6 2001:​db8:​10::​fe prefixlen 128 vhid 2 advskew 100 pass testpass61"​
 +sysrc ifconfig_vtnet4="​inet 2.2.2.2/​24"​
 +sysrc ifconfig_vtnet4_ipv6="​inet6 2001:​db8:​2:​2:​2::​2 prefixlen 64"
 +sysrc ifconfig_vtnet4_alias0="​inet 2.2.2.254/​32 vhid 3 advskew 100 pass testpass42"​
 +sysrc ifconfig_vtnet4_alias1="​inet6 2001:​db8:​2:​2:​2::​fe prefixlen 128 vhid 4 advskew 100 pass testpass62"​
 +sysrc pf_enable=YES
 +sysrc pfsync_enable=YES
 +sysrc pflog_enable=YES
 +sysrc pfsync_syncdev=vtnet1
 +mount -uw /
 +echo "​carp_load="​YES""​ >> /​boot/​loader.conf.local
 +mount -ur /
 +echo "​net.inet.carp.preempt=1"​ >> /​etc/​sysctl.conf
 +
 +cat > /​etc/​pf.conf <<EOF
 +ExtIf="​vtnet4"​
 +IntIf="​vtnet3"​
 +SyncIf="​vtnet1"​
 +# Default block all
 +block
 +# Don't filter on loopback
 +set skip on lo0
 +# Don't sync carp and pfsync
 +pass quick on \$SyncIf proto pfsync keep state (no-sync)
 +pass quick on \$ExtIf ​ proto carp   keep state (no-sync)
 +pass quick on \$IntIf ​ proto carp   keep state (no-sync)
 +# Don't block icmpv6 (don't use this large rule in production!)
 +pass proto ipv6-icmp from any to any
 +# Allow traffic from inside to outside
 +pass log from \$IntIf:​network to any
 +# Allow traffic from self to any
 +pass log from self to any
 +EOF
 +
 +config save
 +hostname VM2
 +kldload carp
 +sysctl net.inet.carp.preempt=1
 +service netif restart
 +service pf start
 +service pfsync start
 +service pflog start
 +</​code>​
 +
 +==== Backup Firewall (VM3) ====
 +
 +<​code>​
 +sysrc hostname=VM3
 +sysrc ifconfig_vtnet1="​inet 192.168.23.3/​24"​
 +sysrc ifconfig_vtnet3="​inet 192.168.10.3/​30"​
 +sysrc ifconfig_vtnet3_ipv6="​inet6 2001:​db8:​10::​3 prefixlen 64"
 +sysrc ifconfig_vtnet3_alias0="​inet 192.168.10.254/​32 vhid 1 advskew 200 pass testpass41"​
 +sysrc ifconfig_vtnet3_alias1="​inet6 2001:​db8:​10::​fe prefixlen 128 vhid 2 advskew 200 pass testpass61"​
 +sysrc ifconfig_vtnet4="​inet 2.2.2.3/​24"​
 +sysrc ifconfig_vtnet4_ipv6="​inet6 2001:​db8:​2:​2:​2::​3 prefixlen 64"
 +sysrc ifconfig_vtnet4_alias0="​inet 10.0.0.254/​32 vhid 3 advskew 200 pass testpass42"​
 +sysrc ifconfig_vtnet4_alias1="​inet6 2001:​db8:​2:​2:​2::​fe prefixlen 128 vhid 4 advskew 200 pass testpass62"​
 +sysrc pf_enable=YES
 +sysrc pfsync_enable=YES
 +sysrc pflog_enable=YES
 +sysrc pfsync_syncdev=vtnet1
 +mount -uw /
 +echo "​carp_load="​YES""​ >> /​boot/​loader.conf.local
 +mount -ur /
 +echo "​net.inet.carp.preempt=1"​ >> /​etc/​sysctl.conf
 +
 +cat > /​etc/​pf.conf <<EOF
 +ExtIf="​vtnet4"​
 +IntIf="​vtnet3"​
 +SyncIf="​vtnet1"​
 +# Default block all
 +block
 +# Don't filter on loopback
 +set skip on lo0
 +# Don't sync carp and pfsync
 +pass quick on \$SyncIf proto pfsync keep state (no-sync)
 +pass quick on \$ExtIf ​ proto carp   keep state (no-sync)
 +pass quick on \$IntIf ​ proto carp   keep state (no-sync)
 +# Don't block icmpv6 (don't use this large rule in production!)
 +pass proto ipv6-icmp from any to any
 +# Allow traffic from inside to outside
 +pass log from \$IntIf:​network to any
 +# Allow traffic from self to any
 +pass log from self to any
 +EOF
 +
 +config save
 +hostname VM3
 +kldload carp
 +sysctl net.inet.carp.preempt=1
 +service netif restart
 +service pf start
 +service pfsync start
 +service pflog start
 +</​code>​
 +==== Outside host (VM4) ====
 +
 +<​code>​
 +sysrc hostname=VM4
 +sysrc ifconfig_vtnet4="​inet 2.2.2.4/​24"​
 +sysrc ifconfig_vtnet4_ipv6="​inet6 2001:​db8:​2:​2:​2::​4 prefixlen 64"
 +sysrc defaultrouter="​2.2.2.254"​
 +sysrc ipv6_defaultrouter="​2001:​db8:​2:​2:​2::​fe"​
 +sysrc gateway_enable=NO
 +sysrc ipv6_gateway_enable=NO
 +sysrc inetd_enable=YES
 +sed -i -e '​s/#​echo/​echo/​g'​ /​etc/​inetd.conf
 +hostname VM4
 +service netif restart
 +service routing restart
 +service inetd start
 +config save
 +</​code>​
 +
 +===== Checking configuration =====
 +
 +==== carp state ====
 +
 +Check that VM2 is in carp master state:
 +
 +<​code>​
 +[root@VM2]~#​ ifconfig vtnet3
 +vtnet3: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ metric 0 mtu 1500
 +        options=80028<​VLAN_MTU,​JUMBO_MTU,​LINKSTATE>​
 +        ether 58:​9c:​fc:​02:​00:​02
 +        inet 192.168.10.2 netmask 0xfffffffc broadcast 192.168.10.3
 +        inet 192.168.10.254 netmask 0xffffffff broadcast 192.168.10.254 vhid 1
 +        inet6 fe80::​5a9c:​fcff:​fe02:​2%vtnet3 prefixlen 64 scopeid 0x4
 +        nd6 options=21<​PERFORMNUD,​AUTO_LINKLOCAL>​
 +        media: Ethernet 10Gbase-T <​full-duplex>​
 +        status: active
 +        carp: MASTER vhid 1 advbase 1 advskew 100
 +[root@VM2]~#​ ifconfig vtnet4
 +vtnet4: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ metric 0 mtu 1500
 +        options=80028<​VLAN_MTU,​JUMBO_MTU,​LINKSTATE>​
 +        ether 58:​9c:​fc:​02:​00:​02
 +        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
 +        inet 10.0.0.254 netmask 0xffffffff broadcast 10.0.0.254 vhid 2
 +        inet6 fe80::​5a9c:​fcff:​fe02:​2%vtnet4 prefixlen 64 scopeid 0x5
 +        nd6 options=21<​PERFORMNUD,​AUTO_LINKLOCAL>​
 +        media: Ethernet 10Gbase-T <​full-duplex>​
 +        status: active
 +        carp: MASTER vhid 2 advbase 1 advskew 100
 +</​code>​
 +
 +And VM3 in backup state:
 +
 +<​code>​
 +[root@VM3]~#​ ifconfig vtnet3
 +vtnet3: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ metric 0 mtu 1500
 +        options=80028<​VLAN_MTU,​JUMBO_MTU,​LINKSTATE>​
 +        ether 58:​9c:​fc:​03:​00:​03
 +        inet 192.168.10.3 netmask 0xfffffffc broadcast 192.168.10.3
 +        inet 192.168.10.254 netmask 0xffffffff broadcast 192.168.10.254 vhid 1
 +        inet6 fe80::​5a9c:​fcff:​fe03:​3%vtnet3 prefixlen 64 scopeid 0x4
 +        nd6 options=21<​PERFORMNUD,​AUTO_LINKLOCAL>​
 +        media: Ethernet 10Gbase-T <​full-duplex>​
 +        status: active
 +        carp: BACKUP vhid 1 advbase 1 advskew 200
 +[root@VM3]~#​ ifconfig vtnet4
 +vtnet4: flags=8943<​UP,​BROADCAST,​RUNNING,​PROMISC,​SIMPLEX,​MULTICAST>​ metric 0 mtu 1500
 +        options=80028<​VLAN_MTU,​JUMBO_MTU,​LINKSTATE>​
 +        ether 58:​9c:​fc:​03:​00:​03
 +        inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255
 +        inet 10.0.0.254 netmask 0xffffffff broadcast 10.0.0.254 vhid 2
 +        inet6 fe80::​5a9c:​fcff:​fe03:​3%vtnet4 prefixlen 64 scopeid 0x5
 +        nd6 options=21<​PERFORMNUD,​AUTO_LINKLOCAL>​
 +        media: Ethernet 10Gbase-T <​full-duplex>​
 +        status: active
 +        carp: BACKUP vhid 2 advbase 1 advskew 200
 +</​code>​
 +==== pf state ====
 +
 +Check the current rules applied:
 +
 +<​code>​
 +[root@VM2]~#​ pfctl -sr
 +block drop all
 +pass quick on vtnet4 proto carp all keep state (no-sync)
 +pass quick on vtnet3 proto carp all keep state (no-sync)
 +pass quick on vtnet1 proto pfsync all keep state (no-sync)
 +pass proto ipv6-icmp all keep state
 +pass log on vtnet3 inet6 from fe80::​5a9c:​fcff:​fe02:​2 to any flags S/SA keep state
 +pass log on vtnet4 inet6 from fe80::​5a9c:​fcff:​fe02:​2 to any flags S/SA keep state
 +pass log inet6 from 2001:​db8:​10::/​64 to any flags S/SA keep state
 +pass log on vtnet1 inet6 from fe80::​5a9c:​fcff:​fe02:​302 to any flags S/SA keep state
 +pass log inet6 from 2001:​db8:​2:​2:​2::​2 to any flags S/SA keep state
 +pass log inet6 from 2001:​db8:​2:​2:​2::​fe to any flags S/SA keep state
 +pass log inet6 from ::1 to any flags S/SA keep state
 +pass log on lo0 inet6 from fe80::1 to any flags S/SA keep state
 +pass log inet from <​__automatic_8a1ff95a_0>​ to any flags S/SA keep state
 +</​code>​
 +===== Creating 2 flows from VM1 to VM4 =====
 +
 +Open a tmux session on R1 and generate 2 flows:
 +  -  A continous ping: ping 10.0.0.4
 +  -  A echo session: telnet 10.0.0.4 7
 +
 +==== pf synchronisation ====
 +
 +Now check there are 4 news states (one for each direction) on the Master firewall:
 +
 +<​code>​
 +[root@VM3]~#​ pfctl -ss
 +all icmp 10.0.0.4:​267 <- 192.168.10.1:​267 ​      0:0
 +all icmp 192.168.10.1:​267 -> 10.0.0.4:​267 ​      0:0
 +all tcp 10.0.0.4:7 <- 192.168.10.1:​31058 ​      ​ESTABLISHED:​ESTABLISHED
 +all tcp 192.168.10.1:​31058 -> 10.0.0.4:​7 ​      ​ESTABLISHED:​ESTABLISHED
 +all carp 224.0.0.18 <- 192.168.10.2 ​      ​NO_TRAFFIC:​SINGLE
 +all carp 224.0.0.18 <- 10.0.0.2 ​      ​NO_TRAFFIC:​SINGLE
 +all pfsync 224.0.0.240 <- 192.168.23.2 ​      ​NO_TRAFFIC:​SINGLE
 +</​code>​
 +
 +And these entries are synced to backup firewall:
 +
 +<​code>​
 +[root@VM3]~#​ pfctl -ss
 +all icmp 10.0.0.4:​39946 <- 192.168.10.1:​39946 ​      0:0
 +all icmp 192.168.10.1:​39946 -> 10.0.0.4:​39946 ​      0:0
 +all tcp 10.0.0.4:22 <- 192.168.10.1:​46911 ​      ​ESTABLISHED:​ESTABLISHED
 +all tcp 192.168.10.1:​46911 -> 10.0.0.4:​22 ​      ​ESTABLISHED:​ESTABLISHED
 +all carp 224.0.0.18 <- 10.0.0.2 ​      ​NO_TRAFFIC:​SINGLE
 +all carp 224.0.0.18 <- 192.168.10.2 ​      ​NO_TRAFFIC:​SINGLE
 +all pfsync 224.0.0.240 <- 192.168.23.2 ​      ​NO_TRAFFIC:​SINGLE
 +</​code>​
 +
 +==== pf log ====
 +
 +Wait for the default 60seconds flush timer of pflogd on the MASTER carp firewall, then check log file:
 +<​code>​
 +[root@VM2]~#​ tcpdump -r /​var/​log/​pflog
 +reading from file /​var/​log/​pflog,​ link-type PFLOG (OpenBSD pflog file)
 +17:​22:​44.851364 IP6 2001:​db8:​10::​1 > 2001:​db8:​2:​2:​2::​4:​ ICMP6, echo request, seq 0, length 16
 +17:​22:​44.851373 IP6 2001:​db8:​10::​1 > 2001:​db8:​2:​2:​2::​4:​ ICMP6, echo request, seq 0, length 16
 +17:​22:​44.851385 IP6 fe80::​5a9c:​fcff:​fe02:​2 > ff02::​1:​ff00:​4:​ ICMP6, neighbor solicitation[|icmp6]
 +17:​22:​44.851577 IP6 fe80::​5a9c:​fcff:​fe02:​2 > ff02::​1:​ff00:​1:​ ICMP6, neighbor solicitation[|icmp6]
 +</​code>​
 +
 +
 +==== Testing failover ====
 +
 +Halt master firewall and check:
 +  - on VM1: no ping lost neither TCP echo session
 +  - on VM3: It became carp master
documentation/examples/pf_and_carp_lab.txt · Last modified: 2017/07/07 00:52 (external edit)