topic Re: High CPU Load while packets processed through slow path in General Topics

High CPU Load while packets processed through slow path

Martin_Reppich — Tue, 07 Apr 2020 12:18:45 GMT

Hello community,

we moved to R80 in February and since then from time to time we receive alert Mail from our SMS (R80.30) that it lost connection to the active gateway of our Check Point Cluster (R80.10).
The result of my investigation was a high CPU Load (100%) on all cores due to high load on the fw_worker processes across this period.

This issue had impact to all parts of the network which are routed through the firewall. We had increased latency in the network and our SMS couldn't get data from the fw node. That is why we have a brake in the graph of SmartView Monitor and had to investigate on the affected node.

In the affected period I recognized an increased amount of inbound packets/sec on our external interface with CPVIEW history. Futhermore I saw also a rise of packets/sec handled by slow path (FW). The amount of inbound packets on the external interface and packets handled by slow path are quite close.

I created a CPInfo file with export of the CPVIEW History for visualization and compared the graph of fw_inbound packets and system_performance and they correlate.

That leads me to the conclusion that the packets has been inspected by default inspection but i'm not able to find information about inspected packets in the logging and although dynamic dispatcher is active, in CPVIEW no Top-Connections are listet under CPVIEW.CPU.Top-Connections.

I'm not sure how to find the connections that was responsible to that behaviour.

Here some values of CPVIEW of the second before and while high CPU load.

CPVIEW.Overview 25Mar2020 14:17:46
|---------------------------------------|
| Num of CPUs: 6 |
| CPU Used |
| 2 45% |
| 1 44% |
| 3 40% |
|---------------------------------------|
| CPU: |
| CPU User System Idle I/O wait Interrupts |
| 0 0% 25% 75% 0% 41,234 |
| 1 12% 32% 56% 0% 41,234 |
| 2 17% 28% 55% 0% 41,234 |
| 3 13% 27% 60% 0% 41,234 |
| 4 0% 1% 99% 0% 41,234 |
| 5 0% 0% 100% 0% 41,234
|---------------------------------------|
| Traffic Rate: |
| Total                                           FW       PXL          SecureXL |
| Inbound packets/sec 155K 9,255   1,432        145K |
| Outbound packets/sec 156K 9,805     1,432         145K |
| Inbound bits/sec 958M           6,380K   10,354K   941M |
| Outbound bits/sec 1,002M     33,917K 10,537K 958M |

CPVIEW.Overview 25Mar2020 14:17:47
|---------------------------------------|
| Num of CPUs: 6 |
| CPU Used |
| 1 89% |
| 2 89% |
| 3 89% |
|---------------------------------------|
| CPU: |
| CPU User System Idle I/O wait Interrupts |
| 0 0% 29% 71% 0% 39,075 |
| 1 22% 67% 10% 0% 39,075 |
| 2 4% 85% 11% 0% 39,075 |
| 3 3% 86% 11% 0% 39,075 |
| 4 0% 0% 100% 0% 39,075 |
| 5 0% 0% 100% 0% 39,075 |
|---------------------------------------|
| Traffic Rate: |
| Total                                            FW         PXL         SecureXL |
| Inbound packets/sec 182K      30,459     1,032     150K |
| Outbound packets/sec 157K   5,877    1,032     150K |
| Inbound bits/sec 1,010M         12,437K   7,394K 991M |
| Outbound bits/sec 1,040M       24,715K 7,526K 1,008M |

I also attached the graph of CPU-Load (one core), fw_inbound, RX on external interface.
I did a manual failover to see if the cpu load is just an issue of one node. You can see it in the graph.

The load suddenly went down at around 13:15 at the 26 of march.

I hope you have ideas for further investigation or preventing this. I thought about creating an own inspection profile and set the most actions to inactive.

Thanks in advance and best regards. Stay healthy

Martin Reppich
System Administrator
Helmholtz-Zentrum Potsdam
Deutsches GeoForschungsZentrum GFZ

Re: High CPU Load while packets processed through slow path

Wolfgang — Tue, 07 Apr 2020 14:59:10 GMT

@Martin_Reppich

which hardware are you running, is it an open server?

Wolfgang

Re: High CPU Load while packets processed through slow path

Benedikt_Weissl — Tue, 07 Apr 2020 16:14:35 GMT

I think CoreXL balances the per connection, so a high CPU usage on all cores would rule out one elephant connection causing the problem. Something like a parallel download of windows updates maybe, but i can't explain why this suddenly a problem with R80.10. I'd upgrade to R80.30 if possible and definitly fine-tune your thread prevention profile. An application control rule to limit non-essential apps like twitch, youtube and spotify to 50mbit also works wonders sometimes.
I assume you're using an open server with 6 cores, licensed for 4 cores, right? Can you post the output of "enabled_blades" and "fwaccel stats -s" please?

Re: High CPU Load while packets processed through slow path

HeikoAnkenbrand — Tue, 07 Apr 2020 17:19:40 GMT

Hi @Martin_Reppich,

You can see that the main part of the packets goes through the SecureXL past path red marked.

| Traffic Rate: |
| Total                                           FW       PXL          SecureXL |
| Inbound packets/sec 155K 9,255   1,432        145K |
| Outbound packets/sec 156K 9,805     1,432         145K |
| Inbound bits/sec 958M           6,380K   10,354K   941M |
| Outbound bits/sec 1,002M     33,917K 10,537K 958M |

More to firewall path F2F, PXL, Acceleration path read here in my article:
RIt could also be elephant flows!80.x - Security Gateway Architecture (Logical Packet Flow)

Please could you provide the following information. Then I can be more specific.

# fwaccel stats
# fw ctl affinity -l
# top | grep fw_worker

Do you use 10GBit/s or 1 GBit/s interfaces?

With 1 GBit/s interfaces, your interface can be busy (red).
Inbound bits/sec 958M 6,380K 10,354K 941M
| Outbound bits/sec 1,002M 33,917K 10,537K 958M

Provide infos to nic errors:
# netstat -in

With 10GBit/s interfaces enable multi queueing. More read here: R80.x - Performance Tuning Tip - Multi Queue

To reduce the load on the CoreXL instances, you can do the following:
- use IPS default profile (only events with no performnce impact)
- use IPS only to and from the internet
- optimize av, anti-bot, https interception,...

It could also be elephant flows!

The big question is, how do you found elephat flows on an R80 gateway?
Evaluation of heavy connections (epehant flows)

A first indication is a high CPU load on a core if all other cores have a normal CPU load. This can be displayed very nicely with "top". Ok, now a core has 100% CPU usage. What can we do now? For this there is a SK105762 to activate "Firewall Priority Queues". This feature allows the administrator to monitor the heavy connections that consume the most CPU resources without interrupting the normal operation of the Firewall. After enabling this feature, the relevant information is available in CPView Utility. The system saves heavy connection data for the last 24 hours and CPDiag has a matching collector which uploads this data for diagnosis purposes.

Heavy connection flow system definition on Check Point gateways:

Specific instance CPU is over 60%
Suspected connection lasts more than 10s
Suspected connection utilizes more than 50% of the total work the instance does. In other words, connection CPU utilization must be > 30%

Enable the monitoring of heavy connections.

To enable the monitoring of heavy connections that consume high CPU resources:

# fw ctl multik prioq 1

# reboot

Found heavy connection on the gateway with print_heavy connections“

On the system itself, heavy connection data is accessible using the command:

# fw ctl multik print_heavy_conn

Found heavy connection on the gateway with cpview

# cpview CPU > Top-Connection > InstancesX

More read here:
R80.x - Performance Tuning Tip - Elephant Flows (Heavy Connections)

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Tue, 07 Apr 2020 18:11:53 GMT

Hello Wolfgang,

it's open server hardware.

We use two IBM x3550 M4.

Best regards
Martin

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Tue, 07 Apr 2020 18:30:47 GMT

Hi Benedikt,

yes we use open server hardware (IBM x3550 M4) and have licensed 4 cores.
Currently we don't use any threat prevention. Here are the outputs you asked for.

[Expert@gw2:0]# enabled_blades
fw vpn mon vpn

[Expert@gw2:0]# fwaccel stats -s
Accelerated conns/Total conns : 24367/32954 (73%)
Accelerated pkts/Total pkts : 51251182933/57415297311 (89%)
F2Fed pkts/Total pkts : 6074120093/57415297311 (10%)
PXL pkts/Total pkts : 89994285/57415297311 (0%)
QXL pkts/Total pkts : 0/57415297311 (0%)

Kind regards
Martin

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Tue, 07 Apr 2020 19:06:43 GMT

Hi @HeikoAnkenbrand

at first thanks for detailed reply.

I agree with you that most of the packets goes through SecureXL fast path.
You refered to the output of CPVIEW Traffic Rate when the cpu utilization was fine.
In my second output with traffic rate also most of the packets go through SecureXL fast path but additionaly the packet rate for slow path is increased and after reading serveral posts regarding high cpu load i thought this probably is the reason in a kind of way.

We use two 10Bit/s interfaces with VLANs where the most traffic went through and four dedicated 1GBit/s interfaces without VLANs.
We don't use any threat prevention only the default inspection which is installed with the access policy is active.

Here are the outputs you asked for.

[Expert@gw2:0]# enabled_blades
fw vpn mon vpn

[Expert@gw2:0]# fwaccel stats
Name Value Name Value
-------------------- --------------- -------------------- ---------------

Accelerated Path
------------------------------------------------------------------------------
accel packets 51409121939 accel bytes 35156919263700
conns created 301133996 conns deleted 180397325
C total conns 32193 C templates 0
C TCP conns 23063 C delayed TCP conns 0
C non TCP conns 9130 C delayed nonTCP con 0
conns from templates 3272 temporary conns 15524747
nat conns 5879735 dropped packets 3603731
dropped bytes 302928490 nat templates 0
port alloc templates 0 conns from nat tmpl 0
port alloc conns 0 conns auto expired 105182996

Accelerated VPN Path
------------------------------------------------------------------------------
C crypt conns 32 enc bytes 1143833360
dec bytes 27895854656 ESP enc pkts 13239884
ESP enc err 42 ESP dec pkts 22706430
ESP dec err 0 ESP other err 40
AH enc pkts 0 AH enc err 0
AH dec pkts 0 AH dec err 0
AH other err 0 espudp enc pkts 0
espudp enc err 0 espudp dec pkts 0
espudp dec err 0 espudp other err 0

Medium Path
------------------------------------------------------------------------------
PXL packets 90048662 PXL async packets 90073798
PXL bytes 67374416312 C PXL conns 40
C PXL templates 0 PXL FF conns 0
PXL FF packets 0 PXL FF bytes 0
PXL FF acks 0

Accelerated QoS Path
------------------------------------------------------------------------------
QXL packets 0 QXL async packets 0
QXL bytes 0 C QXL conns 0
C QXL templates 0

Firewall Path
------------------------------------------------------------------------------
F2F packets 6090346522 F2F bytes 458028184070
C F2F conns 8671 TCP violations 22964531
C partial conns 0 C anticipated conns 0
port alloc f2f 0 C no-match ranges 0

GTP
------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0

General
------------------------------------------------------------------------------
memory used 0 free memory 0
C used templates 0 pxl tmpl conns 0
C conns from tmpl 0 C non TCP F2F conns 239
C tcp handshake conn 3271 C tcp established co 15167
C tcp closed conns 4625 C tcp f2f handshake 297
C tcp f2f establishe 7689 C tcp f2f closed con 446
C tcp pxl handshake 0 C tcp pxl establishe 39
C tcp pxl closed con 1 outbound packets 51409121237
outbound pxl packets 90048662 outbound f2f packets 6107881879
outbound bytes 35899348079681 outbound pxl bytes 68762023621
outbound f2f bytes 3349002292752

(*) Statistics marked with C refer to current value, others refer to total value

[Expert@gw2:0]# fw ctl affinity -l
eth0: CPU 0
eth1: CPU 0
eth2: CPU 0
eth3: CPU 0
eth8: CPU 0
eth9: CPU 0
Kernel fw_0: CPU 3
Kernel fw_1: CPU 2
Kernel fw_2: CPU 1
Daemon mpdaemon: CPU 1 2 3
Daemon vpnd: CPU 1 2 3
Daemon fwd: CPU 1 2 3
Daemon in.asessiond: CPU 1 2 3
Daemon lpd: CPU 1 2 3
Daemon cpd: CPU 1 2 3
Daemon cprid: CPU 1 2 3
The current license permits the use of CPUs 0, 1, 2, 3 only.

[Expert@gw2:0]# top | grep fw_worker
5712 admin 15 0 0 0 0 R 32 0.0 7431:42 fw_worker_0
5713 admin 15 0 0 0 0 S 32 0.0 6815:17 fw_worker_1
5714 admin 15 0 0 0 0 R 24 0.0 5219:04 fw_worker_2

[Expert@gw2:0]# netstat -in
No RX-ERR and TX-ERR on any Interface.

I read your article to firewall path and activate the monitoring of heavy connections.

Thank you and best regards.
Martin

Re: High CPU Load while packets processed through slow path

Timothy_Hall — Tue, 07 Apr 2020 22:06:10 GMT

Item 1:

we moved to R80 in February and since then from time to time we receive alert Mail from our SMS (R80.30) that it lost connection to the active gateway of our Check Point Cluster (R80.10). The result of my investigation was a high CPU Load (100%) on all cores due to high load on the fw_worker processes across this period.

F2F/slowpath is 6.3M inbound and 33.9M outbound by one measurement and 12.4M inbound and 24.7 M outbound in another measurement. Traffic directly to and from the gateway (including logging) always goes F2F, and these imbalances and the behavior described above screams excessive logging rate, and fwd is starving out when the 3 worker cores get busy, not sure if that is just a symptom or cause. Would suggest trying to reduce logging rate, focus on disabling logging for DNS, NetBIOS and other UDP-based protocols, perhaps HTTP connections as well.

Item 2: There is only one SND core so Multi-Queue will not help.

Item 3: Please provide output of fwaccel stat, packet and connections acceleration looks good but if NAT templates are disabled it will cause a lot of F2F and worker overhead if you have a high new connection rate, see next item. Would caution against enabling NAT templates unless latest Jumbo HFA has been installed on your R80.10 gateway, early implementations of NAT Templates caused SecureXL problems when enabled.

Item 4: During a high CPU period please provide new connection rate, on cpview screen this can be seen on the Overview page, scroll down to the Network section make sure to show the whole Network section.

Item 5: Your single SND core seems to be doing OK so there is probably not much RX-DRP, but your bandwidth numbers under load are suspiciously close to exactly 1Gbit as Heiko noticed, if you are pushing a 1Gbps interface that close to its theoretical limit you are probably racking up overruns (RX-OVR) like crazy. Please provide full output of netstat -ni for analysis.

Item 6: You're not using SHA-384 for any of your VPN's are you? Looks like most of your VPN traffic is fully accelerated but using SHA-384 will cause VPN traffic to go F2F.

Item 7: Even under load it looks like your firewall workers are pretty evenly balanced by the Dynamic Dispatcher, so I doubt it is an elephant flow issue but still worth investigating as Heiko mentioned.

Item 8: High load can be caused by an overloaded sync network in a cluster, please provide output of fw ctl pstat. Might need to do selective synchronization for protocols mentioned in Item 1, as combo of suddenly high connection rates and having to state sync them too is a nasty double whammy on the workers.

Item 9: Doubtful you are having memory issues, but please provide output of free -m anyway.

Item 10: You could be getting flooded with fragmented packets during a high CPU event (frags always go F2F in your version), please provide output of fwaccel stats -p.

Re: High CPU Load while packets processed through slow path

PhoneBoy — Wed, 08 Apr 2020 00:25:33 GMT

Any particular reason you're still on R80.10 for your gateway?
Lot of improvements in R80.20+ performance wise.
Not saying it will solve this particular issue but worth considering.

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Wed, 08 Apr 2020 05:15:12 GMT

We did the upgrade with a service provider and they recommend to go to R80.10 and wait until R80.30 with kernel 3.10 is available for our open server IBM x3550 M4.

If such an release isn't planed we probably can talk about an upgrade too.

Re: High CPU Load while packets processed through slow path

PhoneBoy — Wed, 08 Apr 2020 05:17:24 GMT

We're not planning support for additional Open Servers on R80.30-3.10.
R80.40 was released for all supported appliances with 3.10 kernel (no more 2.6 kernel).

Re: High CPU Load while packets processed through slow path

Martin_Raska — Wed, 08 Apr 2020 06:28:05 GMT

you can print violations for SXL

SecureXL violations (F2F packets) - fwaccel stats -p

Did you manage to find what kind of traffic is that? Maybe some DoS or DDoS from Internet, fragmented traffic, IP options, some other protocol that is not TCP/UDP,

what kind of encryption algorithms you are using for VPN?

Maybe activate drop templates?

CPVIEW - top connections in the time of the problem.

Or analyze traffic by Traffic analysis using the 'CPMonitor' tool - sk103212

Re: High CPU Load while packets processed through slow path

HeikoAnkenbrand — Wed, 08 Apr 2020 08:18:32 GMT

Hi @Martin_Reppich,

@Timothy_Hall has already given a couple of good tips here.

Here is good to see that the SecureXL must optimize.

Path	Packets
Accelerated Path	51409121939
F2F Path	6090346522
Accelerated VPN Path ESP enc pkts	13239884
Medium Path PXL packets	90048662

If you use 10 Gbps, you should definitely use multi queueing. This means you need more SND's to use MQ. I would definitely try that. More read here R80.x - Performance Tuning Tip - Multi Queue and here R80.x - Performance Tuning Tip - Intel Hardware.

With the "Accelerated VPN Path" you can see that only low processing is done here and many packets probably run into the F2F path. I would adjust the VPN encryption algorithms here so that more are processed in the "Accelerated VPN Path". With "fwaccel stat" you can see which algorithms are supported in SecureXL.

Regards
Heiko

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Thu, 09 Apr 2020 05:30:55 GMT

@Timothy_Hall

Thank you for the detailed information.

I answered the Itmes and attached it as txt file for the formatting.

Best regards
Martin

Re: High CPU Load while packets processed through slow path

Timothy_Hall — Fri, 10 Apr 2020 11:40:27 GMT

Item 1: OK great so it looks like you did have excessive logging due to duplicating your logs to another log server, good to see a 10% drop in CPU.

Item 3: While your accept templating rate is pretty good, due to output of fwaccel stat take a look at rule #11. You probably have some kind of DCE/RPC service in use there. Try to move that rule or service as far down in your rulebase as you can, right before your cleanup rule if possible. Install policy and check fwaccel stat again, rinse and repeat until disablement of template offloads doesn't happen until one of the very last rules in your rulebase, this will help reduce F2F by increasing connection templating rate even further.

NAT Templates are disabled as expected, it might help to enable them but we won't know until I see new connection rate during high CPU period as requested in Item 4.

Item 5: Your interfaces are running clean and your single SND is doing fine with the load; you don't need to adjust your CoreXL split to add another SND or enable Multi-Queue. Some negligible RX-DRP on eth9 but not nearly enough to worry about.

Item 6: Confirmed no SHA-384 in use for VPNs.

Item 7: Definitely do NOT want to adjust CoreXL split and reduce number of firewall workers.

Item 8: Hmm your sync network does appear to be struggling a bit based on fw ctl pstat (but underlying network sync interface is running clean) which can cause high CPU on firewall workers, would suggest disabling state synchronization for DNS and HTTP for sure and perhaps HTTPS service as well like this:

Also look at Top-Protocols in cpview and consider disabling synchronization for those services as well to help reduce sync traffic.

Item 9: Memory looks good, no allocation failures.

Item 10: Given overall number of frames, frag numbers do not look excessive.

Summary: Major action items are #3 and #8 which should help quite a bit. Beyond that enabling NAT Templates (with latest Jumbo HFA) may help, but we need to see Network screen of cpview and new connection rate during a high CPU period first.

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Tue, 14 Apr 2020 11:12:39 GMT

Item 3: Rule #11 had zero hits since the upgrade and I checked it. This rule is not necessary anymore so I disabled it for now. Afterwards the output of fwaccel stat shows the following.

Accept Templates : disabled by Firewall
Layer Network disables template offloads from rule #69
Throughput acceleration still enabled.

Rule #69 is a rule for nfs to a ftp-filer and has the service group "NFS" included which has RPC-Services inside.

I analyzed the rule and only five services are matching. Thats why I changed the current services from NFS, udp/111, tcp/111 and tcp-high-ports to nfsd-tcp (2049), nfsd (2049/udp), tcp/111, udp/111 and tcp/4046.

Of course after performing fwaccel stat now again i've got a new rule which i have to analyze. I probably I will go your way and put the upcoming rules from fwaccel stat to the bottom of the rule base. It needs a little time because I have to check if it is easyly possible to move those rules down in context to the rulebase.

Item4: Last weekend the problem occured again. Here are the first outputs of the cpview overview after it startet at 11Apr2020 03:13:27 AM. The period ended at 12Apr2020 22:35.

1Apr2020 03:13:27 AM
CPU Used
1 25%
2 25%
3 24%
----------------------------------------
Network:
Bits/sec 184M
Packets/sec 43,240
Connections/sec 332
Concurrent connections 24,367

11Apr2020 3:13:28
CPU Used
1 81%
2 81%
3 81%
----------------------------------------
Network:
Bits/sec 261M
Packets/sec 49,371
Connections/sec 362
Concurrent connections 25,677

11Apr2020 3:15:27
CPU Used
1 99%
2 99%
3 98%
----------------------------------------
Network:
Bits/sec 270M
Packets/sec 51,216
Connections/sec 370
Concurrent connections 25,425

We had a failover at 11Apr2020 10:38:56 and I attached two screenshots because I mentioned that the rulebase drop rate for this period rised strongly.

Failover from GW2 --> GW1

Item 8: I disabled the synchronization for domain-udp, http and https. I will have a look on the average load.

Thanks and best regards

Martin

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Tue, 14 Apr 2020 12:38:57 GMT

Item 3: Rule #11 had zero hits since the upgrade and I checked it. This rule is not necessary anymore so I disabled it for now. Afterwards the output of fwaccel stat shows the following.

Accept Templates : disabled by Firewall
Layer Network disables template offloads from rule #69
Throughput acceleration still enabled.

Rule #69 is a rule for nfs to a ftp-filer and has the service group "NFS" included which has RPC-Services inside.

I analyzed the rule and saw that only five services are matching. These are nfsd-tcp (2049), nfsd (udp/2049), tcp/111, udp/111 and tcp/4046.
I reduced the services for the rule to the necessary ones from formerly NFS (Group), udp-111, tcp-111 and tcp-high-ports.

After installation of the policy a new rule is given when executing fwaccel stat. I try to follow your advice and move those rules down to the rulebase. It will need a little time because I have to check that does not have negative effect in context to the rule base.

Item 4: The problem occured again last weekend for the period 11Apr 03:13:27 AM until 12Apr 10:35:00 PM. Here are the requested outputs of CPVIEW Overview when the cpu load starts to increase.

11Apr2020 3:13:27
| CPU Used
| 1 25%
| 2 25%
| 3 24%
| ----------------------------------------
| Network:
| Bits/sec 184M
| Packets/sec 43,240
| Connections/sec 332
| Concurrent connections 24,367
| ----------------------------------------

11Apr2020 3:13:28
| CPU Used
| 1 81%
| 2 81%
| 3 81%
| ----------------------------------------
| Network:
| Bits/sec 261M
| Packets/sec 49,371
| Connections/sec 362
| Concurrent connections 25,677
| ----------------------------------------

11Apr2020 3:15:27
CPU Used
| 1 99%
| 2 99%
| 3 98%
| ----------------------------------------
| Network:
| Bits/sec 270M
| Packets/sec 51,216
| Connections/sec 370
| Concurrent connections 25,425

In this time a failover happend because one interface was down.
Failover GW2 --> GW1
I mentioned a huge increase of firewall rulebase drop in this time. You can see this in the screenshots.

Item 8: As suggested I deactivated the synchronization for domain-udp, http and https.

Thank you, best regards
Martin

Re: High CPU Load while packets processed through slow path

Timothy_Hall — Tue, 14 Apr 2020 13:15:52 GMT

Your concurrent connections numbers are consistently and suspiciously close to 25,000 which is the old default hard limit for number of connections. On the firewall/cluster object under Optimizations, do you have "Capacity Optimization" set to Automatically?

Assuming you are set for Automatically, it looks like your firewall is getting blasted with drops (not legit accepted connections) during the high CPU period. You will need to look at the firewall logs during the slow periods and try to figure out if these blasts are coming from the inside/DMZ or the outside, I don't think you can view policy drops per interface from something like cpview but perhaps your graphing application that provided those screenshots can. The policy optimizations you are doing via fwaccel stat won't help for dropped traffic which still has to go through the whole rulebase. It is also possible these drops are a red herring caused by the failover since you enabled selective sync for various services.

If the blasts are from the outside, I would suggest enabling the SecureXL penalty box feature which will blacklist the offending IP address(es) and efficiently drop all their traffic for awhile in SecureXL. If the blasts are coming from the inside you'll have some more latitude to hopefully deal with whatever is causing them, be on the lookout for NMS systems sending lots of SNMP/ICMP probes, and auditors running some kind of automated port-scanning tools. sk74520: What is the SecureXL penalty box mechanism for offending IP addresses?

The SecureXL penalty box will only drop traffic coming in from outside interfaces by default, but it can be set to penalty-box traffic coming from the inside as well. However I wouldn't recommend doing that until you understand what is going on, as the penalty box will drop ALL traffic from an offending IP address on the inside, so that could adversely impact production.

Re: High CPU Load while packets processed through slow path

Martin_Reppich — Wed, 15 Apr 2020 11:49:03 GMT

"Capacity Optimization" is set to Automatically. And here is the calculated value of the active node.

Active Node
[Expert@gw1:0]# fw tab -t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 30566 65171 91753

I used Check Point diagnostics view for the visualization of the cpview history database. In my opinion there is a correlation. The packet rate on our outside interface increases and at the same time the system performance and rulebase drops go up. See the attached Screenshot.

Every Sunday I generate a Rule Usage Report with Tufin Secure Track. The result from last sunday, our most used rule is the CleanUp Rule with 91% (20,694,963,845 Hits) followed by another CleanUp rule with 2% hits (536,587,767).

We have activated Logging for this rules. Maybe not the drop is the issue but the risen logging rate?

And maybe it's useful to say that we have a public class B network.

I filtert with SmartConsole for the last period with high cpu load last weekend (4/11/2020 - 4/13/2020). I have a result for top sources and services. It's suspicious that there are several IP addresses from the same network. I also attached a Screenshot.

SecureXL penalty box sounds interesting I never heard of it before read the SK at first.

Best regards

Re: High CPU Load while packets processed through slow path

Timothy_Hall — Wed, 15 Apr 2020 14:51:21 GMT

> And maybe it's useful to say that we have a public class B network.

Oooof, yep you are going to have a lot of drops from all the background noise crap of the Internet with a potential attack surface of that size. Looking at your screenshots confirms that you have lots of drops coming in from the Internet. This is a common issue, here is what you need to do:

1) If possible, configure Geo Policy in a blacklist configuration to drop traffic to/from countries that your organization has no business talking to.
Geo Policy drops happen very early in INSPECT processing (right after antispoofing), and can substantially lower CPU usage (I've seen CPU load dive 30% after configuring it with clients like yourself who have a large block of Internet-routable addresses). Normally I'd suggest using Geo Updatable Objects here instead, but support for those was added in R80.20 and you're still on R80.10 gateway. Depending on the scope and geographic reach of your organization using Geo Policy may not be feasible though. Edit: Be sure to define a Geo Policy exception for DNS traffic (both TCP and UDP port 53) to avoid some rather random-looking DNS problems that can result, this nasty situation was covered on pages 280-281 of my latest book)

2) Enable the SecureXL penalty box. The default is to penalty-box an IP address with 500+ drops/sec for a period of 3 minutes, and only for traffic on the external interface. I'd suggest enabling it with the default settings and see how it behaves for a week or so; assuming it doesn't break anything in production with the defaults, in the real world I usually tweak down the threshold from 500 drops down to the 200 range or so with no adverse impacts. Do not attempt to use the optimized drops feature as it is much older.

3) Over time if you have repeat offenders that keep getting tossed in the penalty box, create an explicit rule at the top of your rulebase that matches these bad actors and silently drops them with no logging.

Let us know how it goes...