Checkpoint 6600 and UPPAK

[George_Sas]

Hi guys.

Since 1 month ago I am fighting with a high CPU load issue where my CPU is always over 70% when seen in cpview.
The load jumped from 30-40% to 70-100% over night without any MAJOR changes to our environment.

I have 6 firewalls , 2 x 6600 in a Cluster and 4 x Small appliances for remote offices.

So after a few days of troubleshooting on my own we have open a TAC with Checkpoint and things are going SLOOOOOW like never before with help from Checkpoint.

Finally today we got a remote session with checkpoint today. 

fwaccel stats show a pretty nasty TCP and UDP miss connections...

fwaccel stats -p


Checkpoint suggested to to switch from KPPAK to UPPAK mode.
In my opinion this would be only a bandaid to the real issue that is yet to be discovered.

Anyway , in cpconfig we have no option to change the cluster mode ?

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products

(11) Exit

We run 81.20 Take 120.

Is UPPAK really unavailable on 6600 appliances or are we missing something ?

Thanks in advance.

[Bob_Zimmerman]

I can confirm UPPAK is not available on the 6600 in R82 jumbo 60, so it's likely not expected to be available in R81.20. It also doesn't show up on my boxes with eight cores. I'd be a little surprised to see it on a box with fewer than 16.

[PhoneBoy]

KPPAK and UPPAK are not "clustering modes" but are relevant to where SecureXL operates (in Kernel or Userspace).
It appears UPPAK is not supported on the 6600 per: https://support.checkpoint.com/results/sk/sk153832

[George_Sas]

Grrrrrrr.... don't like this 🙂 
Well , planing to change to 9300 plus in a few months ... hopefully I won't have to change again in 3 years.

Thanks for confirmation.

[PhoneBoy]

Given that R82.10 only supports UPPAK, I think it's safe to say UPPAK isn't going anywhere.

[Timothy_Hall]

@George_Sas Please post the Super Seven and enabled_blades outputs, ideally taken when the system is slow.  We need to see the exact CPU execution type (us, si, sy, etc.) and on which core type (SND vs. Worker).  My initial guess would be a F2F/slowpath percentage well over 10%, but we'll see.  

https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/m-p/40...

[George_Sas]

I can see only 5% accelerated packets but what worries me is "fwaccel stats -p" results.

F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
Pkt has IP options 8 ICMP miss conn 1902388
TCP-SYN miss conn 22562058 TCP-other miss conn 37977470
UDP miss conn 15763555 Other miss conn 87704
VPN returned F2F 23595 Uni-directional viol 0
Possible spoof viol 677 TCP state viol 252
SCTP state affecting 0 Out if not def/accl 0
Bridge src=dst 0 Routing decision err 0
Sanity checks failed 40 Fwd to non-pivot 0
Broadcast/multicast 0 Cluster message 12379687
Cluster forward 22819 Chain forwarding 0
F2V conn match pkts 61128 General reason 0
Route changes 0 VPN multicast traffic 0
GTP non-accelerated 0 Unresolved nexthop 34

Enabled blades :
fw vpn cvpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation mon Scrub

[Timothy_Hall]

1) You have 96% accelerated packets (Accelerated pkts/Total pkts).  Accelerated conns/Total conns 3% is the accept template hit rate, which is fine.  Slowpath is 3%, which is also fine.

2) Hopefully, the components of bond0 are eth1-01 and eth1-03, and the components of bond1 are eth1-02 and eth1-04.  If that is not the case, you have a gigantic imbalance on both TX and RX sides, even though active-active is set, and you'll need to set L3+4 Transmit Hash Policy on both sides of every bond.  See my Be your own TAC Part Deux presentation.  RX-DRP rate is right on the borderline of 0.1%, and fixing your bonds should help.

3) Due to the high percentage of fastpath traffic, your two SND cores (0/1) can get overwhelmed if Dynamic Split is not enabled, is it?  show dynamic-balancing state from clish or dynamic_balancing -p from expert mode.

[George_Sas]

It's not me who installed this cluster ... was installed back in 2018 I believe and the main person responsible for it has just quit a few months back 🙂 So I got it pushed on my "desk" and was told to deal with it 🙂

We are running Active / Passive and Dynamic Balancing is OFF.

Dynamic Balancing is currently Off

Regarding network interfaces :
bond0 - Eth1-01 + eth1-03
bond1 - Eth1-02 + eth1-04

I guess my appliances arrived at EOL and need something more powerfull to handle all the new traffic ?
Still my CPU load increased drastically over night a in 16 January.

There where I have low CPU loads after this date , are due to switching the active member. Load moves with active member and no matter what I tried could not lower it as much as before 16 January.