User Tools

Site Tools


documentation:examples:dropping_packets_at_high_rate

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
documentation:examples:dropping_packets_at_high_rate [2020/02/07 09:01] – [Performance impact] olivierdocumentation:examples:dropping_packets_at_high_rate [2024/02/09 09:42] (current) – [NIC level configuration] olivier
Line 1: Line 1:
 ====== Dropping packets at high rate ====== ====== Dropping packets at high rate ======
  
-===== Goal =====+===== Objective =====
  
 {{:documentation:examples:labs.examples.ddos.png}} {{:documentation:examples:labs.examples.ddos.png}}
Line 10: Line 10:
  
 The configuration file of an IPFW in standard mode is this one: The configuration file of an IPFW in standard mode is this one:
-  First rule is a blacklist of denied IP addresses +  First rule is to deny a blacklist table (IP addresses) 
-  Second rule is to allow all the rest +  Second rule is to allow all the rest 
-  * Disabling the outgoing [[https://www.freebsd.org/cgi/man.cgi?query=pfil&apropos=0&sektion=0&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html|pfil(9)]] hook because we don't need to filter outgoing traffic+  - Disable the outgoing [[https://www.freebsd.org/cgi/man.cgi?query=pfil&apropos=0&sektion=0&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html|pfil(9)]] hook at IP level because we don't need to filter outgoing traffic in this case
  
 <code> <code>
Line 28: Line 28:
 </code> </code>
 ==== NIC level configuration ==== ==== NIC level configuration ====
-(A FreeBSD 13 / head) only feature. 
  
-Only [[https://svnweb.freebsd.org/changeset/base/346632|iflib]][[https://svnweb.freebsd.org/changeset/base/356613|vtnet]], [[https://svnweb.freebsd.org/changeset/base/346247|Mellanox]] and [[https://svnweb.freebsd.org/changeset/base/357483|Chelsio]] drivers are "[[https://svnweb.freebsd.org/changeset/base/343631|Pfil Memory Pointer Hooks]]" compliant.+Currently the [[https://svnweb.freebsd.org/changeset/base/343631|Pfil Memory Pointer Hooks]] feature is supported by [[https://svnweb.freebsd.org/changeset/base/346632|iflib]], [[https://svnweb.freebsd.org/changeset/base/356613|vtnet]][[https://svnweb.freebsd.org/changeset/base/346247|Mellanox]] and [[https://svnweb.freebsd.org/changeset/base/357483|Chelsio]] drivers.
  
 The configuration file of an IPFW-at-NIC-level is this one: The configuration file of an IPFW-at-NIC-level is this one:
-  First rule is a blacklist of denied IP addresses +  First rule is to deny a blacklist table (IP addresses) 
-  Second rule is to allow all the rest +  Second rule is to allow all the rest 
-  * Removing pfil(9) from the IP level (in & out+  - Enabling pfil(9) at NIC level (in only
-  * Enabling pfil(9) at NIC level (in only)+  - Removing pfil(9) from the IP level (in & out)
  
 <code> <code>
Line 56: Line 55:
 </code> </code>
  
-==== Performance impact ====+==== Performance benches ====
  
 Hardware: Hardware:
Line 64: Line 63:
   * FreeBSD 13.0-CURRENT r357572   * FreeBSD 13.0-CURRENT r357572
  
-Here is the rate of inet4 packets-per-second forwarded while dropping the 42Mpps of DDoS using the different configuration sets:+Here is the rate of inet4 (legitimate) packets-per-second forwarded while dropping 42Mpps of denied packets using the different configuration sets:
 <code> <code>
 x ipfw-standard x ipfw-standard
Line 85: Line 84:
         (Student's t, pooled s = 29598.4)         (Student's t, pooled s = 29598.4)
 </code> </code>
 +
 +On the 14Mpps of legitimate traffic, this generic (ie: supported by multi drivers) software firewall is still able to forward 12Mpps while droping 42Mpps of denied packets.
 ===== Using Chelsio's TCAM firewall ===== ===== Using Chelsio's TCAM firewall =====
  
-Chelsio NIC allows to configure hardware firewall with the use of cxgbetool(8).+Chelsio NIC allows to configure hardware firewall with the use of cxgbetool(8): The [[https://service.chelsio.com/beta/drivers/ChelsioUwire-3.1.0.0/Chelsio-UnifiedWire-Linux-UserGuide.pdf|linux user guide]] gives a lots more details than the [[https://service.chelsio.com/beta/drivers/ChelsioUwire-FBSD-3.3.0.1/Chelsio-UnifiedWire-FreeBSD-UserGuide.pdf|FreeBSD user guide]]. 
 A Chelsio NIC is defined by its family name + id and the port id (if it's a 4 ports, port 0 to 4). A Chelsio NIC is defined by its family name + id and the port id (if it's a 4 ports, port 0 to 4).
 +
 Example with only one Chelsio (t5nex0) with 2 ports (0 and 1) Example with only one Chelsio (t5nex0) with 2 ports (0 and 1)
 <code> <code>
-# grep t5nex /var/run/dmesg.boot+# grep t.nex /var/run/dmesg.boot
 t5nex0: <Chelsio T580-LP-CR> mem 0xf9300000-0xf937ffff,0xf8000000-0xf8ffffff,0xf9984000-0xf9985fff irq 40 at device 0.4 on pci4 t5nex0: <Chelsio T580-LP-CR> mem 0xf9300000-0xf937ffff,0xf8000000-0xf8ffffff,0xf9984000-0xf9985fff irq 40 at device 0.4 on pci4
 cxl0: <port 0> on t5nex0 cxl0: <port 0> on t5nex0
Line 98: Line 101:
 </code> </code>
  
-Here is the same rules translated for this usage: +Translating the firewall rule for the Chelsio
-  * cxl0 is then port 0 of t5nex0 +  * Add a filter to drop packets incoming from Chelsio NIC 0 (t5nex0) port 0 matching source IP range 198.18.2.0/24
-  * Add a filter to drop packets incoming from port 0 matching source IP range 198.18.2.0/24+
 <code> <code>
 # cxgbetool t5nex0 filter 0 iport 0 sip 198.18.2.0/24 action drop # cxgbetool t5nex0 filter 0 iport 0 sip 198.18.2.0/24 action drop
Line 108: Line 110:
 </code> </code>
  
-Checking the drop rate now by using a small script:+To check the packet dropping rate, this a small script will be used:
 <code> <code>
 #!/bin/sh #!/bin/sh
Line 115: Line 117:
         echo "Need Chelsio nexus name (examble: t5nex0)"         echo "Need Chelsio nexus name (examble: t5nex0)"
                 echo "List of Nexus detected:"                 echo "List of Nexus detected:"
-                grep t5nex /var/run/dmesg.boot || true+                grep t.nex /var/run/dmesg.boot || true
         exit 1         exit 1
 fi fi
Line 141: Line 143:
 </code> </code>
  
-The NIC is filtering at a rate of 32Mpps: Where are the 10Mpps missing?+The script report an hardware dropping rate of 32Mpps: Where are the other 10Mpps ?
  
-Need to tune the TCAM for filtering usage: +Let's read the [[https://cgit.freebsd.org/src/tree/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt|Chelsio default firmware configuration file of our T5 family NIC]]: 
-  * Let only 2 occupied "regions" in the TCAM: route and filter+<code> 
 +        # TCAM has 8K cells; each region must start at multiple of 128 cell. 
 +        # Each entry in these categories takes 4 cells each.  nhash will use the 
 +        # TCAM iff there is room left (that is, the rest don't add up to 2048). 
 +        nroute = 32 
 +        nclip = 32 
 +        nfilter = 1008 
 +        nserver = 512 
 +        nhash = 524288 
 +</code>
  
-Here the default value:+And we can display the current value applied:
 <code> <code>
 #  sysctl -n dev.t5nex.0.misc.devlog | grep -w le #  sysctl -n dev.t5nex.0.misc.devlog | grep -w le
Line 152: Line 163:
         16           619796      INFO       RES  le initialization: nentries 2048 route 32 clip 32 filter 1440 server 416 active 128 hash 0 nserversram 0         16           619796      INFO       RES  le initialization: nentries 2048 route 32 clip 32 filter 1440 server 416 active 128 hash 0 nserversram 0
 </code> </code>
-For that we need to download a [[https://svnweb.freebsd.org/base/head/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt?view=co|default TCAM firmware configuration file for our T5 NIC]] to edit its parameters and load it into the NIC+ 
 +To improve the TCAM performance for a filtering usage, all unused "regions" will be disabled to kept only the route and filter (32 entries for route + 2016 for filter = 2048 total). 
 + 
 +For that we need to download a [[https://cgit.freebsd.org/src/tree/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt|default TCAM firmware configuration file for our T5 NIC]] to modify its parameters then load the modified configuration into the NIC flash and instruct the NIC to use the file from its flash.
 <code> <code>
-# fetch -o /etc/t5fw.txt 'https://svnweb.freebsd.org/base/head/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt?view=co'+# fetch -o /etc/t5fw.txt https://cgit.freebsd.org/src/plain/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt
 # sed -i "" -e "s/nclip.*/nclip = 0/" /etc/t5fw.txt # sed -i "" -e "s/nclip.*/nclip = 0/" /etc/t5fw.txt
-# sed -i "" -e "s/nfilter.*/nfilter = 2000/" /etc/t5fw.txt+# sed -i "" -e "s/nfilter.*/nfilter = 2016/" /etc/t5fw.txt
 # sed -i "" -e "s/nserver.*/nserver = 0/" /etc/t5fw.txt # sed -i "" -e "s/nserver.*/nserver = 0/" /etc/t5fw.txt
 # sed -i "" -e "s/nhash.*/nhash = 0/" /etc/t5fw.txt # sed -i "" -e "s/nhash.*/nhash = 0/" /etc/t5fw.txt
 # echo 'hw.cxgbe.config_file="flash"' >> /boot/loader.conf.local # echo 'hw.cxgbe.config_file="flash"' >> /boot/loader.conf.local
 # cxgbetool t5nex0 loadcfg /etc/t5fw.txt # cxgbetool t5nex0 loadcfg /etc/t5fw.txt
 +# reboot
 </code> </code>
  
Line 168: Line 183:
         12           690716      INFO       RES  le configuration: nentries 2048 route 32 clip 32 filter 1024 server 0 active 960 hash 0 nserversram 0         12           690716      INFO       RES  le configuration: nentries 2048 route 32 clip 32 filter 1024 server 0 active 960 hash 0 nserversram 0
 </code> </code>
-We confirm that server|hash are at 0,+We confirm that regions server and hash are at 0 (disabled): Notice that region clip is not disabled and filter didn't have the size we've instructedbut the filtering performance expectation are matched.
  
 Now the packet drop rate by the TCAM firewall match the generator's 42Mpps: Now the packet drop rate by the TCAM firewall match the generator's 42Mpps:
Line 184: Line 199:
 </code> </code>
  
-And the firewall able to forward all packets without being too busy in the same time:+And the firewall is now able to forward all packets **without being too busy** in the same time:
 <code> <code>
 [root@firewall]~# netstat -ihw 1 [root@firewall]~# netstat -ihw 1
Line 196: Line 211:
        46M       422       2.7G        14M           801M     0        46M       422       2.7G        14M           801M     0
        46M      3.9k       2.7G        14M           801M     0        46M      3.9k       2.7G        14M           801M     0
 +
 [root@firewall]~# nstat -I cxl0 [root@firewall]~# nstat -I cxl0
  InMpps  OMpps InGbs   OGbs err  TCP Est  %CPU syscalls    csw irq     GBfree  InMpps  OMpps InGbs   OGbs err  TCP Est  %CPU syscalls    csw irq     GBfree
documentation/examples/dropping_packets_at_high_rate.1581062507.txt.gz · Last modified: 2020/02/07 09:01 by olivier

Except where otherwise noted, content on this wiki is licensed under the following license: BSD 2-Clause
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki