CheckPoint 5900 VSX Cluster High CPU

[Demith_Samaraw2]

Hi All

I have a strange issue, we have CP 5900 VSX VSLS cluster with 3 virtual firewalls, only one is active on node-1 and others are active node-2.

We have coreXL and SecureXL enabled with only IPS blade enabled, strangely on node 1 there is one firewall worker taking lot of CPU

Also strangely ~70% traffic takes F2F path without any explanation. If it would have being IPS it should take PXL path for the most of the traffic?.

Anyone has any idea what is wrong with this?

[Kaspars_Zibarts]

You should see connections that are not accelerated with 

fwaccel conns -f F

might help you to identify root cause

[Demith_Samaraw2]

Thanks Kaspars

I will have a look at that command

[Timothy_Hall]

VSX is not my specialty but I'll take a shot here.

As far as the high F2F, try applying IPS profile "Optimized" to your gateway and see if it improves the situation with high F2F.  If it does not, try running these commands in your VS:

ips off

fwaccel stats -r

(wait 60 seconds)

fwaccel stats -s

ips on

Did F2F go way down in "fwaccel stats -s"?  If so it is definitely something in your IPS profile config, probably an active signature with a performance rating of "Critical" handling a lot of traffic.  Make sure you run "ips on" at the end!

If F2F is still stubbornly high you could have fragmentation or some other kind of issue interfering with SecureXL.  Please post the output of the following command to this thread:

fwaccel stats -p

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Demith_Samaraw2]

Hi Tim

Actually disabling IPS did not fix the issue much,

fwaccel stats -p gives this output

biggest culprits here are TCP conn is F2Fed, UDP miss conn, TCP state viol, and TCP-SYN miss conn

Any idea what kind of traffic is causing this, 

[Kaspars_Zibarts]

As said before, look at the actual traffic that's not being accelerated, might give some clues

fwaccel conns -f F

[Kaspars_Zibarts]

Also I noticed that there's not a lot of traffic there - 40000 packets in 60secs.. That's ~700pps, almost nothing.

Are you looking at VS0 stats? It is quite normal to see 100% F2F on VS0 as most traffic will be either CP management (18192) or logs (257) and that cannot be accelerated as it originates from gateway itself

here's my VS0

And fwaccel conns -f F shows connections originating or terminating on GW itself

[Demith_Samaraw2]

Hi Kaspars

Nope, this is run on VS1, actually this is run very late in the night, when there were not much traffic, I guess I kind of have an idea what is causing this, I have done some packet captures on the day and based on the Wireshark, most of the traffic going through this firewall microsoft-ds/CIFS and I guess CP still send all of that traffic to F2F path, but I will get a fwaccel conns -f F output to compare the list of actuall connections.

[Kaspars_Zibarts]

Great, we can rule that out. CIFS should take PXL not F2F. Check actual IPs  and see if it leads somewhere

[Maarten_Sjouw]

Check with cpview, advanced and network, this shows the heaviest connections and the path.

[Demith_Samaraw2]

Thanks Tim,

I will do this test tomorrow