40 Gbps + Open Server Testing

[Jason_Carrillo]

We are getting ready to decommission some of our firewalls running on open hardware we are wanting to take a look at installing some 40 Gbps Intel network cards into the firewalls and do some testing.

I know that there aren't any 40 Gbps NICs on the HCL, but I am wondering... Has anyone done any testing with the higher throughput cards? 

Thanks!

[Amir_Arama]

did you had the chance to check it ?

thanks

[Timothy_Hall]

The 40Gb cards used on Check Point appliances are manufactured by Mellanox (driver name mlx5_core) so even if they aren't on the HCL for open hardware, they *should* work.

Also note that the limitation that kept SMT/Hyperthreading from being enabled when using the Mellanox cards (noted on page 402 of the second edition of my book) appears to have been lifted, or at least I can no longer find any reference to the SMT limitation here: sk116742: Installing and Configuring Dual 40GbE and Dual 25G/100G I/O card in 5000, 15000 and 23000 ...

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jason_Carrillo]

Hello Amir. We just now had the opportunity to get the cards installed and the firewalls setup. We are using Intel 40 Gbps cards and the firewall software was able to recognize them and their speeds.  We have yet to put any traffic through them though as the network team has some work to do on the routing. Will update when I have more information. 

[_Val_]

With all due respect to Timothy Hall, the official Check Point position on this is the following: please use a combination of platform and periphery from HLC only, if you are running open server system.

It is important to understand that each certified open server comes with its own list of supported hardware. Any other supported NICs that might work on other platforms, may not be supported on yours if it not on the list. Even if it might work, the configuration will still be no supported from TAC point of view.

I will be happy to assist you navigating HLC, if you like. Otherwise, please take care and make sure whatever you are using is there for the right version of software.

[Jason_Carrillo]

Interesting thing we've seen while testing. As soon as we tried to run tcpdump on an interface on the firewall, the interface would show down and the gateway would fail over to the other wall. Switch to the other wall, run tcpdump, and sure enough, it would fail back.  

Neat!

[JozkoMrkvicka]

What about fw monitor ?

[Jason_Carrillo]

Shoot forgot to include that in my previous post, fw monitor does not seem to cause a disruption. 

[Timothy_Hall]

What does ClusterXL log when the failover occurs as a result of running tcpdump?  Use the log filter "type:Control" to see the specific ClusterXL messages.  I'm assuming ClusterXL is reporting a failure of the interface against which the tcpdump is being run, that is most definitely not normal or expected.  Does including the -p option (don't use promiscuous mode) to tcpdump keep this ClusterXL issue from occurring?  It may have something to do with tcpdump/libpcap being a rather old version for kernel 2.6.18 while the Mellanox cards and their associated drivers are relatively new.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jason_Carrillo]

You assume correctly. When running tcpdump on eth0.

Cluster Information: (ClusterXL) member 2 (192.168.110.2) is down (Interface Active Check on member 2 (192.168.110.2) detected a problem (eth0 interface is down, 4 interfaces required, only 3 up).).

Description:(ClusterXL) member 2 (192.168.110.2) is down (Interface Active Check on member 2 (192.168.110.2) detected a problem (eth0 interface is down, 4 interfaces required, only 3 up).).

When running tcpdump with -p, it doesn't seem to happen.

We are using intel cards currently, but we are going to buy a Mellanox card for testing as well. Here is the driver info if interested:

[Expert@fw2]# ethtool -i eth0
driver: i40e
version: 1.4.25
firmware-version: 5.05 0x80002924 1.1313.0
bus-info: 0000:04:00.0

[Jason_Carrillo]

Update! So we've stood up an R80.10 cluster with our intel 40Gbps cards and sure enough the drivers on those cards (i40e) are not listed as supporting multi-queue. When we run perfsonar tests through the walls we end up topping out at around 11 Gbps and one of the cores goes 100% utilization.

Unfortunately, I don't know enough about multiqueue to know if that is the issue, but it appears that R80.20 will support multiqueue for with i40e drivers, and since 80.20 is now GA, we are going to upgrade it. 

[Timothy_Hall]

11Gbps sounds reasonable for a single SND/IRQ core emptying a busy interface's ring buffer without the aid of Multi-Queue, depending on the average size of frames being handled.  While using Multi-Queue is definitely the right solution to rectify this performance limit, should you be unable to move to R80.20 there are a few things you can try to squeeze a bit more performance out of it without Multi-Queue:

1) Make sure there are enough SND/IRQ cores allocated such that automatic interface affinity ends up assigning the busy interface its very own SND/IRQ core that is not being shared with any other interfaces (check this with sim affinity -l)

2) Increase the ring buffer size of the interface; always a last resort but may be appropriate in this situation

3) If you have a large percentage of fully-accelerated traffic (>50% Accelerated pkts as reported by fwaccel stats -s), disabling SMT/Hyperthreading may help

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jason_Carrillo]

We were able to get up to 37 Gbps on this firewall once we installed R80.20 and set up Multiqueue.  Unfortunately because of the issues we found with the management upgrade to R80.20, we've been unable to much more testing. However, we are pretty happy with that performance. No IPS or App control is enabled on the gateway though.

Since we had to roll back to R80.10 on management, we are looking at installing the hotfix for management to allow managing R80.20. Hopefully this doesn't break us again like the R80.20 upgrade did. Once it is installed we are going to be doing some additional testing.

It does beg the question though, when will Check Point support open hardware network cards of above 10 Gbps officially? Multiqueue support for i40e cards in R80.20 seems like they might be moving that way, unless there are appliance cards that run that driver class.