Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision |
documentation:examples:dropping_packets_at_high_rate [2020/02/07 15:05] – [Using Chelsio's TCAM firewall] olivier | documentation:examples:dropping_packets_at_high_rate [2022/03/15 19:51] – [Using Chelsio's TCAM firewall] olivier |
---|
====== Dropping packets at high rate ====== | ====== Dropping packets at high rate ====== |
| |
===== Goal ===== | ===== Objective ===== |
| |
{{:documentation:examples:labs.examples.ddos.png}} | {{:documentation:examples:labs.examples.ddos.png}} |
| |
The configuration file of an IPFW in standard mode is this one: | The configuration file of an IPFW in standard mode is this one: |
* First rule is a blacklist of denied IP addresses | - First rule is to deny a blacklist table (IP addresses) |
* Second rule is to allow all the rest | - Second rule is to allow all the rest |
* Disabling the outgoing [[https://www.freebsd.org/cgi/man.cgi?query=pfil&apropos=0&sektion=0&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html|pfil(9)]] hook because we don't need to filter outgoing traffic | - Disable the outgoing [[https://www.freebsd.org/cgi/man.cgi?query=pfil&apropos=0&sektion=0&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html|pfil(9)]] hook at IP level because we don't need to filter outgoing traffic in this case |
| |
<code> | <code> |
</code> | </code> |
==== NIC level configuration ==== | ==== NIC level configuration ==== |
(A FreeBSD 13 / head) only feature. | ** A FreeBSD 13 (-head) only feature.** |
| |
Only [[https://svnweb.freebsd.org/changeset/base/346632|iflib]], [[https://svnweb.freebsd.org/changeset/base/356613|vtnet]], [[https://svnweb.freebsd.org/changeset/base/346247|Mellanox]] and [[https://svnweb.freebsd.org/changeset/base/357483|Chelsio]] drivers are "[[https://svnweb.freebsd.org/changeset/base/343631|Pfil Memory Pointer Hooks]]" compliant. | Currently the [[https://svnweb.freebsd.org/changeset/base/343631|Pfil Memory Pointer Hooks]] feature is supported by [[https://svnweb.freebsd.org/changeset/base/346632|iflib]], [[https://svnweb.freebsd.org/changeset/base/356613|vtnet]], [[https://svnweb.freebsd.org/changeset/base/346247|Mellanox]] and [[https://svnweb.freebsd.org/changeset/base/357483|Chelsio]] drivers. |
| |
The configuration file of an IPFW-at-NIC-level is this one: | The configuration file of an IPFW-at-NIC-level is this one: |
* First rule is a blacklist of denied IP addresses | - First rule is to deny a blacklist table (IP addresses) |
* Second rule is to allow all the rest | - Second rule is to allow all the rest |
* Removing pfil(9) from the IP level (in & out) | - Enabling pfil(9) at NIC level (in only) |
* Enabling pfil(9) at NIC level (in only) | - Removing pfil(9) from the IP level (in & out) |
| |
<code> | <code> |
</code> | </code> |
| |
==== Performance impact ==== | ==== Performance benches ==== |
| |
Hardware: | Hardware: |
* FreeBSD 13.0-CURRENT r357572 | * FreeBSD 13.0-CURRENT r357572 |
| |
Here is the rate of inet4 packets-per-second forwarded while dropping the 42Mpps of DDoS using the different configuration sets: | Here is the rate of inet4 (legitimate) packets-per-second forwarded while dropping 42Mpps of denied packets using the different configuration sets: |
<code> | <code> |
x ipfw-standard | x ipfw-standard |
(Student's t, pooled s = 29598.4) | (Student's t, pooled s = 29598.4) |
</code> | </code> |
| |
| On the 14Mpps of legitimate traffic, this generic (ie: supported by multi drivers) software firewall is still able to forward 12Mpps while droping 42Mpps of denied packets. |
===== Using Chelsio's TCAM firewall ===== | ===== Using Chelsio's TCAM firewall ===== |
| |
The script report an hardware dropping rate of 32Mpps: Where are the other 10Mpps ? | The script report an hardware dropping rate of 32Mpps: Where are the other 10Mpps ? |
| |
Let's read the [[https://svnweb.freebsd.org/base/head/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt?view=co|Chelsio default firmware configuration file of our T5 family NIC]]: | Let's read the [[https://cgit.freebsd.org/src/tree/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt|Chelsio default firmware configuration file of our T5 family NIC]]: |
<code> | <code> |
# TCAM has 8K cells; each region must start at a multiple of 128 cell. | # TCAM has 8K cells; each region must start at a multiple of 128 cell. |
</code> | </code> |
| |
To improve the TCAM performance for a filtering usage, all unused "regions" will be disabled to kept only the route and filter. | To improve the TCAM performance for a filtering usage, all unused "regions" will be disabled to kept only the route and filter (32 entries for route + 2016 for filter = 2048 total). |
| |
| For that we need to download a [[https://cgit.freebsd.org/src/tree/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt|default TCAM firmware configuration file for our T5 NIC]] to modify its parameters then load the modified configuration into the NIC flash and instruct the NIC to use the file from its flash. |
For that we need to download a [[https://svnweb.freebsd.org/base/head/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt?view=co|default TCAM firmware configuration file for our T5 NIC]] to edit its parameters and load it into the NIC | |
<code> | <code> |
# fetch -o /etc/t5fw.txt 'https://svnweb.freebsd.org/base/head/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt?view=co' | # fetch -o /etc/t5fw.txt 'https://svnweb.freebsd.org/base/head/sys/dev/cxgbe/firmware/t5fw_cfg_hashfilter.txt?view=co' |
# sed -i "" -e "s/nclip.*/nclip = 0/" /etc/t5fw.txt | # sed -i "" -e "s/nclip.*/nclip = 0/" /etc/t5fw.txt |
# sed -i "" -e "s/nfilter.*/nfilter = 2000/" /etc/t5fw.txt | # sed -i "" -e "s/nfilter.*/nfilter = 2016/" /etc/t5fw.txt |
# sed -i "" -e "s/nserver.*/nserver = 0/" /etc/t5fw.txt | # sed -i "" -e "s/nserver.*/nserver = 0/" /etc/t5fw.txt |
# sed -i "" -e "s/nhash.*/nhash = 0/" /etc/t5fw.txt | # sed -i "" -e "s/nhash.*/nhash = 0/" /etc/t5fw.txt |
# echo 'hw.cxgbe.config_file="flash"' >> /boot/loader.conf.local | # echo 'hw.cxgbe.config_file="flash"' >> /boot/loader.conf.local |
# cxgbetool t5nex0 loadcfg /etc/t5fw.txt | # cxgbetool t5nex0 loadcfg /etc/t5fw.txt |
| # reboot |
</code> | </code> |
| |
12 690716 INFO RES le configuration: nentries 2048 route 32 clip 32 filter 1024 server 0 active 960 hash 0 nserversram 0 | 12 690716 INFO RES le configuration: nentries 2048 route 32 clip 32 filter 1024 server 0 active 960 hash 0 nserversram 0 |
</code> | </code> |
We confirm that server|hash are at 0, | We confirm that regions server and hash are at 0 (disabled): Notice that region clip is not disabled and filter didn't have the size we've instructed, but the filtering performance expectation are matched. |
| |
Now the packet drop rate by the TCAM firewall match the generator's 42Mpps: | Now the packet drop rate by the TCAM firewall match the generator's 42Mpps: |
</code> | </code> |
| |
And the firewall able to forward all packets without being too busy in the same time: | And the firewall is now able to forward all packets **without being too busy** in the same time: |
<code> | <code> |
[root@firewall]~# netstat -ihw 1 | [root@firewall]~# netstat -ihw 1 |
46M 0 422 2.7G 14M 0 801M 0 | 46M 0 422 2.7G 14M 0 801M 0 |
46M 0 3.9k 2.7G 14M 0 801M 0 | 46M 0 3.9k 2.7G 14M 0 801M 0 |
| |
[root@firewall]~# nstat -I cxl0 | [root@firewall]~# nstat -I cxl0 |
InMpps OMpps InGbs OGbs err TCP Est %CPU syscalls csw irq GBfree | InMpps OMpps InGbs OGbs err TCP Est %CPU syscalls csw irq GBfree |