Best CoreXL Firewall mode

[Moudar]

Hi,

I have a cluster of 6500 gateways and a VM management server, all running R81.20 with Take 84.

The gateways are currently operating in kernel mode. While I understand that user mode is the default for these gateways, I am unsure why they are configured to run in kernel mode.

fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name     |Status     |Interfaces               |Features                      |
+---------------------------------------------------------------------------------+
|0 |KPPAK    |enabled    |Sync,Mgmt,eth1-01,       |Acceleration,Cryptography     |
|  |         |           |eth1-03,eth1-04          |                              |
|  |         |           |                         |Crypto: Tunnel,UDPEncap,MD5,  |
|  |         |           |                         |SHA1,3DES,DES,AES-128,AES-256,|
|  |         |           |                         |ESP,LinkSelection,DynamicVPN, |
|  |         |           |                         |NatTraversal,AES-XCBC,SHA256, |
|  |         |           |                         |SHA384,SHA512                 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates   : enabled
NAT Templates    : enabled
LightSpeed Accel : disabled

fwaccel stats -s command shows:   

fwaccel stats -s
Accelerated conns/Total conns    : 228/71177 (0%)
LightSpeed conns/Total conns     : 0/71177 (0%)
Accelerated pkts/Total pkts      : 48951535737/54082458012 (90%)
LightSpeed pkts/Total pkts       : 0/54082458012 (0%)
F2Fed pkts/Total pkts            : 5130922275/54082458012 (9%)
F2V pkts/Total pkts              : 255589979/54082458012 (0%)
CPASXL pkts/Total pkts           : 1919756022/54082458012 (3%)
PSLXL pkts/Total pkts            : 46235738870/54082458012 (85%)
CPAS pipeline pkts/Total pkts    : 0/54082458012 (0%)
PSL pipeline pkts/Total pkts     : 0/54082458012 (0%)
QOS inbound pkts/Total pkts      : 0/54082458012 (0%)
QOS outbound pkts/Total pkts     : 0/54082458012 (0%)
Corrected pkts/Total pkts        : 0/54082458012 (0%)

from sk167052 i can see that if  30% or more of the traffic undergoes the PXL / Medium path, then Usermode is recommended!

As you can see 85% of traffic undergoes PXL.

The question:

Moving from Kernel mode to Usermode, do we need a service windows to do that?

What potential issues could arise during this transition?

[AkosBakos]

Hi @Moudar 

I suggest you that, always do changes in maintenance window, for safety's sake.

Ok, do you want to enable USFW mode. Except this statement, do you experience performance degradation? High load on CPU etc?

Does dynamic balancing enabled?

Because of this:

Procedure	Instructions
Recommended	Connect to the command line on the Security Gateway / each Cluster Member. Run: cpconfig Enter the number of the Check Point CoreXL option. Enter 3 to select Change firewall mode. Follow the instructions on the screen. Exit from the cpconfig menu. Reboot. In a cluster, this can cause a failover.

So always do it in maintanace window 🙂

Akos

[Moudar]

I don’t want to enable User mode just for the sake of it. My intention is to follow the recommendation, especially since the CPU occasionally reaches 100%.

Currently, the connection stats show: Accelerated conns/Total conns: 226/59,929 (0%). I’m not sure if enabling User mode would improve this!

[AkosBakos]

Hi @Moudar 

Yo are facing with performance issues. all CPUs reach the 100%? Or sometimes. What does the spike detector in cpview say?

You need to call the “Super Seven” Commands for help.

#fwaccel stat
#fwaccel stats -s
#grep -c ^processor /proc/cpuinfo
#fw ctl affinity -l -r
#netstat -ni
#fw ctl multik stat
#cpstat os -f multi_cpu -o 1

Esepecially the first one. My idea is that the acceleration is not working properly.

What do you see under Accept Templates - > Security disables template offloads from rule #XX

Akos

[_Val_]

In my personal opinion, with 8 cores on the appliance, moving to USFW mode will not give you any advantages. The only reason to switch would be about TLS Inspection. You cannot do TLS 1.3 and QUIC without USFW on

[Chris_Atkinson]

Note USFW and UPPAK are not the same things, the later is SecureXL terminology not CoreXL and applies to Quantum Force appliances.

[AkosBakos]

THX, I changed the UPPAK to USFW. 

[Timothy_Hall]

The other recommendation in the SK is that if fastpath traffic is in excess of 80% KMFW is preferred; your firewall is at 90% which is why it may have been changed.  The SK may also recommend that if "30% or more of the traffic undergoes the PXL / Medium path, then Usermode/USFW is recommended", but USFW is less efficient than KMFW for Medium Path and Slowpath due to having to cross the kernel/userspace boundary; the penalty is 20-30%. 

However the default for the 6000 series is USFW which you should probably use to get the latest features like TLS/QUIC/connview/Hyperflow, as these will not work with KMFW.  The extra 20-30% speed in KMFW is not worth the functionality tradeoff in my opinion.  Also the QA testing of the code for the 6000 boxes was conducted in USFW mode.

Changing from KMFW to USFW will not improve " Accelerated conns/Total conns: 226/59,929 (0%)" as that is a templating issue, run fwaccel templates -R to diagnose; you almost certainly have rulebase construction issues causing the 0%.

[Moudar]

When I run the command fwaccel templates -R i get this:

fwaccel templates -R

Matched connections not allowed to use templates:
% Prevention : 0.482%

Reason                                  Count      Reason Prevented From Matched %

Non-Syn/Empty First Packet              |478496    |0.374     %
Src/dst IP Blacklisted                  |137822    |0.108     %
--------------------

Connections failed to create templates:
% Fail to Create : 39.533%

Reason                                  Count      Reason Fail To Create %

Multicast Conn                          |558836    |0.246     %
NON TCP/UDP PROTO                       |3701462   |1.628     %
Conn Not Accelerated                    |7439049   |3.271     %
NAT Disallowed Conn                     |77340198  |34.010    %
DHCP Check Feature Isn't Supported Or Disabled|36        |0.000     %
General Error                           |545518    |0.240     %
Malicious Destination IP Detected       |66431     |0.029     %
Prevented By Policy Rules               |249288    |0.110     %

What could be causing NAT to block or disallow connections?

[AkosBakos]

What does #fwaccel stat say?

[Moudar]

that information is covered at the beginning of the post. Please take a look there!

[AkosBakos]

Indeed, sorry. 

[AkosBakos]

As I see you opened a thread with almost the same topic.

https://community.checkpoint.com/t5/Security-Gateways/nat-disallows/m-p/228235#M43999

Before you tried to understand it. Some housekeeping steps may could help 🙂

• if you do a failover the issue persists?
  • I know the policy is the same but have a try
• how much is the uptime? 
  
  • it should be 60-90 days
• this behaviour was earlier too? Before take 84?

However what about the TAC? 

akos

[Moudar]

I’m aware of my old posts😃, but the question here is specifically about User mode versus Kernel mode. I have a feeling that transitioning from Kernel mode to User mode might be a potential solution. but maybe not

[AkosBakos]

As _Val_ told that, this would not be the solution, but up to you. If the performance getting worse you could revert.

And how much is the trougput overall on the GW member?

Akos

[Timothy_Hall]

Did you run that command on the standby member of a cluster or the active?  That looks like the standby.  If not I would assume that means that NAT templates are not forming for some reason and forcing a full NAT rulebase lookup in the slowpath, even though it looks like NAT templates are fully enabled.  R&D will probably have to comment ( @PhoneBoy ), also see here:

(nat disallows)

[Moudar]

This is from the active gateway, which has been active for 16 days.

[PhoneBoy]

According to @CheckMatesAI, the reasons won't NAT Template include the following:

1. NAT Templates are Disabled: If NAT templates are not enabled, NATed traffic cannot be templated. Refer to sk71200 for more details.

2. VPN Traffic: VPN traffic cannot be templated.

3. Complex Connections: Connections involving complex protocols such as FTP, H323, SQL, etc., cannot be templated.

4. Non-TCP/Non-UDP/ICMP Traffic: Traffic that is not TCP, UDP, or ICMP cannot be templated.

5. Specific Rules in the Rule Base:

   • Rules with service 'Any' (resolved from R75.40).
   • Rules with a service that has a 'handler'. When the chosen service has a protocol type defined, instead of 'None', it might have a handler configured on it. This setting can be changed only in SmartDashboard R7X and lower. For R80.XX, changes can only be done by cloning the service.

6. Network Quota: When the SmartDefense/IPS protection "Network Quota" is enabled, SecureXL Accept Templates/NAT Templates/Drop Templates are automatically disabled. Refer to sk31630 for more details.

7. Overlapping NAT: Overlapping NAT does not support any form of acceleration on any platform, including SecureXL or IPSO Flows. Refer to sk44091 for more details.

8. Point-to-Point Interfaces: SecureXL does not support Point-to-Point interfaces (PPP, PPTP, PPPoE). If a PPP-interface is detected, SecureXL disables itself on that interface. Refer to sk79880 for more details.

9. Global DHCP Services: Using global DHCP services in the policy disables SecureXL Accept Templates. Use local 'dhcp' related services in the domain's rulebase to avoid this behavior. Refer to sk162544 for more details.

That said, we probably already covered all that, so I also asked @CheckMatesAI  how to debug NAT templates.
It suggested the following commands:

fwaccel dbg resetall
fwaccel dbg -m general + nat
fw ctl debug 0
fw ctl debug -buf 32000
fw ctl debug -m fw + conn packet nat xlate xltrc
fw ctl kdebug -T -f > /var/log/kernel_debug.txt

To turn off:

fw ctl debug 0
fwaccel dbg resetall

Hopefully that will help track it down.