topic Re: CheckPoint 5900 VSX Cluster High CPU in Firewall and Security Management

CheckPoint 5900 VSX Cluster High CPU

Demith_Samaraw2 — Wed, 10 Oct 2018 01:21:12 GMT

Hi All

I have a strange issue, we have CP 5900 VSX VSLS cluster with 3 virtual firewalls, only one is active on node-1 and others are active node-2.

We have coreXL and SecureXL enabled with only IPS blade enabled, strangely on node 1 there is one firewall worker taking lot of CPU

Also strangely ~70% traffic takes F2F path without any explanation. If it would have being IPS it should take PXL path for the most of the traffic?.

Anyone has any idea what is wrong with this?

Re: CheckPoint 5900 VSX Cluster High CPU

Kaspars_Zibarts — Wed, 10 Oct 2018 07:16:10 GMT

You should see connections that are not accelerated with

fwaccel conns -f F

might help you to identify root cause

Re: CheckPoint 5900 VSX Cluster High CPU

Demith_Samaraw2 — Thu, 11 Oct 2018 05:25:05 GMT

Thanks Kaspars

I will have a look at that command

Re: CheckPoint 5900 VSX Cluster High CPU

Timothy_Hall — Fri, 12 Oct 2018 22:56:23 GMT

VSX is not my specialty but I'll take a shot here.

As far as the high F2F, try applying IPS profile "Optimized" to your gateway and see if it improves the situation with high F2F. If it does not, try running these commands in your VS:

ips off

fwaccel stats -r

(wait 60 seconds)

fwaccel stats -s

ips on

Did F2F go way down in "fwaccel stats -s"? If so it is definitely something in your IPS profile config, probably an active signature with a performance rating of "Critical" handling a lot of traffic. Make sure you run "ips on" at the end!

If F2F is still stubbornly high you could have fragmentation or some other kind of issue interfering with SecureXL. Please post the output of the following command to this thread:

fwaccel stats -p

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: CheckPoint 5900 VSX Cluster High CPU

Demith_Samaraw2 — Sun, 14 Oct 2018 23:47:14 GMT

Thanks Tim,

I will do this test tomorrow

Re: CheckPoint 5900 VSX Cluster High CPU

Demith_Samaraw2 — Thu, 18 Oct 2018 03:54:23 GMT

Hi Tim

Actually disabling IPS did not fix the issue much,

fwaccel stats -p gives this output

biggest culprits here are TCP conn is F2Fed, UDP miss conn, TCP state viol, and TCP-SYN miss conn

Any idea what kind of traffic is causing this,

Re: CheckPoint 5900 VSX Cluster High CPU

Kaspars_Zibarts — Thu, 18 Oct 2018 05:24:10 GMT

As said before, look at the actual traffic that's not being accelerated, might give some clues

fwaccel conns -f F

Re: CheckPoint 5900 VSX Cluster High CPU

Kaspars_Zibarts — Thu, 18 Oct 2018 05:32:01 GMT

Also I noticed that there's not a lot of traffic there - 40000 packets in 60secs.. That's ~700pps, almost nothing.

Are you looking at VS0 stats? It is quite normal to see 100% F2F on VS0 as most traffic will be either CP management (18192) or logs (257) and that cannot be accelerated as it originates from gateway itself

here's my VS0

And fwaccel conns -f F shows connections originating or terminating on GW itself

Re: CheckPoint 5900 VSX Cluster High CPU

Demith_Samaraw2 — Thu, 18 Oct 2018 05:38:07 GMT

Hi Kaspars

Nope, this is run on VS1, actually this is run very late in the night, when there were not much traffic, I guess I kind of have an idea what is causing this, I have done some packet captures on the day and based on the Wireshark, most of the traffic going through this firewall microsoft-ds/CIFS and I guess CP still send all of that traffic to F2F path, but I will get a fwaccel conns -f F output to compare the list of actuall connections.

Re: CheckPoint 5900 VSX Cluster High CPU

Kaspars_Zibarts — Thu, 18 Oct 2018 05:43:47 GMT

Great, we can rule that out. CIFS should take PXL not F2F. Check actual IPs and see if it leads somewhere

Re: CheckPoint 5900 VSX Cluster High CPU

Maarten_Sjouw — Thu, 18 Oct 2018 20:52:42 GMT

Check with cpview, advanced and network, this shows the heaviest connections and the path.