topic Checkpoint policy drops traffic while installing the policy. in General Topics

Checkpoint policy drops traffic while installing the policy.

Venky_21 — Tue, 28 Apr 2020 12:43:08 GMT

Hi,

Please help me with this. While installing the checkpoint policy , i can see some traffic is dropped, due to high cpu utilization. Suggest me with all possible solution.

Logs for your reference:

[Expert@US063FW-17]# fw ver
This is Check Point VPN-1(TM) & FireWall-1(R) R75.47 - Build 171
[Expert@US063FW-17]# enabled_blades
fw
[Expert@US063FW-17]# cphaprob stat

Cluster Mode: Sync only (OPSEC) with IGMP Membership

Number Unique Address Firewall State (*)

1 (local) 192.168.2.1 Active
2 192.168.2.2 Active

(*) FW-1 monitors only the sync operation and the security policy
Use OPSEC's monitoring tool to get the cluster status

[Expert@US063FW-17]#
[Expert@US063FW-17]# fwaccel stat
Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #194
Drop Templates : disabled
NAT Templates : disabled by user
Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, WireMode,
DropTemplates, NatTemplates, Streaming,
MultiFW, AntiSpoofing, DoS Defender, ViolationStats,
Nac, AsychronicNotif, ERDOS
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

[Expert@US063FW-17]# fw ctl affinity -l -r
CPU 0: Mgmt eth3-02 eth3-03
CPU 1: Sync eth3-04
CPU 2: fw_9
CPU 3: fw_8
CPU 4: fw_7
CPU 5: fw_6
CPU 6: fw_5
CPU 7: fw_4
CPU 8: fw_3
CPU 9: fw_2
CPU 10: fw_1
CPU 11: fw_0
All: eth3-01
rtmd mpdaemon in.geod in.asessiond in.atelnetd fwd cpd cprid
[Expert@US063FW-17]# sim affinity -l
Sync : 1
eth3-04 : 1
eth3-01 : 1
eth3-02 : 0
eth3-03 : 0
Mgmt : 0
[Expert@US063FW-17]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 571201 0 0 0 4457209 0 0 0 BMRU
Mgmt:1 1500 0 - no statistics available - BMRU
Sync 1500 0 7135512270 0 635978368 0 36840380808 0 0 0 BMRU
eth3-01 1500 0 2741822495663 0 25756871 0 824242839551 0 0 0 BMRU
eth3-02 1500 0 892711572797 0 15075500 0 2825803749984 0 0 0 BMRU
eth3-03 1500 0 126627899400 0 2561531 0 99619763573 0 0 0 BMRU
eth3-04 1500 0 41407655257 0 2155 0 36830188659 0 0 0 BMRU
eth3-04.420 1500 0 19108740474 0 0 0 17185259684 0 0 0 BMRU
eth3-04.421 1500 0 127223682 0 0 0 154860694 0 0 0 BMRU
eth3-04.422 1500 0 22114010854 0 0 0 19425860838 0 0 0 BMRU
eth3-04.423 1500 0 40243731 0 0 0 64171757 0 0 0 BMRU
lo 16436 0 21963018 0 0 0 21963018 0 0 0 LRU
[Expert@US063FW-17]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
-------------------------------------------
0 | Yes | 11 | 3451 | 36359
1 | Yes | 10 | 8451 | 28885
2 | Yes | 9 | 2721 | 30418
3 | Yes | 8 | 2772 | 31418
4 | Yes | 7 | 7274 | 29085
5 | Yes | 6 | 2656 | 33469
6 | Yes | 5 | 2611 | 33813
7 | Yes | 4 | 6460 | 31557
8 | Yes | 3 | 3360 | 28837
9 | Yes | 2 | 2776 | 44874
[Expert@US063FW-17]# free -m
total used free shared buffers cached
Mem: 11998 11671 326 0 670 7544
-/+ buffers/cache: 3456 8541
Swap: 26654 0 26654

Re: Checkpoint policy drops traffic while installing the policy.

Timothy_Hall — Tue, 28 Apr 2020 13:52:50 GMT

A few notes:

1) Your code level R75.47 has not been supported for a very long time.

2) The Sync interface which is handling state table synchronization is thoroughly overloaded with an RX-DRP rate approaching 9%. If you run fw ctl pstat and look at the sync statistics at the end of the output you will see massive loss and retransmissions, which will cause persistently high CPU on all worker cores (fw_X). If your Sync interface is 100Mbps it needs to be upgraded to 1Gbit if possible, or if that is not possible you need to turn off sync for services DNS, HTTP, and HTTPS. You need to get the health of that Sync network fixed ASAP as that is going to directly impact CPU load during your policy installs.

3) How many rules are in your policy? Templating getting disabled by rule #194 is not a big deal if there are only 200 rules in your rulebase, but if you have many more rules than that the overhead of rule base lookups will cause high CPU on your workers.

4) Once the Sync network is fixed if policy installs still cause a traffic disruption, consider setting Connection Persistence from "rematch connections" to "keep all connections" on your cluster object. This will substantially reduce CPU load during policy installs, but existing connections that are not allowed by the newly-installed policy will still be allowed through until the connection ends.

Re: Checkpoint policy drops traffic while installing the policy.

HeikoAnkenbrand — Tue, 28 Apr 2020 14:33:27 GMT

R75.47 is a long time out of support.

Check CUL/Freeze state in /var/log/messages:

# cat /var/log/messages | grep freeze

# cat /var/log/messages | grep CUL

Re: Checkpoint policy drops traffic while installing the policy.

Lloyd_Crosby — Tue, 28 Apr 2020 15:16:27 GMT

The long and the short here... R75.47 was god awful. It's also 32bit, and CoreXL was very much in it's infancy.

Is this an open platform or appliance? If it's an appliance see if it can goto R80. Based off of the core and memory it should be ok... if the processor allows it. If it's an open platform... see if the CPU is supported in the HCL. It won't fix everything but 64bit processing and actually being able to use all of that RAM... and multithreaded processes does wonders.

With that rule set... I'm going to suspect that your rule base is massive. You are going to need to do some rule base clean up.

I also suspect that this isn't a "Hey there is only one thing causing issues here." It's going to be a lot of issues that need to be tackled in a logical fashion.

Re: Checkpoint policy drops traffic while installing the policy.

PhoneBoy — Tue, 28 Apr 2020 19:08:13 GMT

SecureXL is disabled temporarily during a policy install, which can cause a significant CPU increase if the gateway is loaded.
This is something we've minimized greatly in the most recent R80.x versions (R80.20+).
In any case, you're on a version of code that's been End of Support for 4 years now and it's best to upgrade to a supported release.

Re: Checkpoint policy drops traffic while installing the policy.

Venky_21 — Wed, 29 Apr 2020 00:37:37 GMT

Thanks for taking your time to reply.
I have 2659 rules in total.
Settings for Sync: Speed: 1000Mb/s Duplex: Full driver: e1000e
How to turn off sync for services DNS, HTTP, and HTTPS?

Re: Checkpoint policy drops traffic while installing the policy.

Timothy_Hall — Wed, 29 Apr 2020 11:39:44 GMT

It would be great to get that templating as shown by fwaccel stat much further down into in your big rulebase, but in that version many many things will stop templating such as DHCP/traceroute/domain/time/RPC/DCE objects among many others.

With a rulebase that size the rematch is going to slam the CPU during a policy load, would strongly recommend setting "keep all connections" as mentioned in an earlier post if you are OK with the security ramifications.

For those three services uncheck this box on the Advanced screen like this to disable sync for them: