topic Re: Performance issues : Loss of packets in Firewall and Security Management

Performance issues : Loss of packets

Zia — Mon, 02 Sep 2019 17:23:58 GMT

Hi everyone,

I would like the help of the experts here.

We have 2 firewall (5400 model) HA configured and a HP server that acts as the SMS. All of them run under Gaia R80.10.

Here are my main issues:

-We have severe case of packet loss in all of the interfaces of the active firewall and as a result the network is very slow.

Thank you in advance for all of your suggestions and helpful tips.

*********************************************************************

[Expert@Firewall-1:0]# enabled_blades
fw vpn cvpn urlf av appi ips identityServer anti_bot vpn

*********************************************************************

[Expert@Firewall-1:0]# fwaccel stats -s
Accelerated conns/Total conns : 2/1574 (0%)
Accelerated pkts/Total pkts : 118218/167295927 (0%)
F2Fed pkts/Total pkts : 6917099/167295927 (4%)
PXL pkts/Total pkts : 160260610/167295927 (95%)
QXL pkts/Total pkts : 0/167295927 (0%)
*********************************************************************

[Expert@Firewall-1:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 1 | 823 | 11716
1 | Yes | 0 | 766 | 11359

*********************************************************************

[Expert@Firewall-1:0]# free -m
total used free shared buffers cached
Mem: 7744 7160 584 0 449 3521
-/+ buffers/cache: 3188 4555
Swap: 18394 19 18375

*********************************************************************

[Expert@Firewall-2]# fwaccel stats -s
Accelerated conns/Total conns : 0/32 (0%)
Accelerated pkts/Total pkts : 0/2924244 (0%)
F2Fed pkts/Total pkts : 2924244/2924244 (100%)
PXL pkts/Total pkts : 0/2924244 (0%)
QXL pkts/Total pkts : 0/2924244 (0%)

Re: Performance issues : Loss of packets

PhoneBoy — Mon, 02 Sep 2019 17:46:01 GMT

I think the output of the "Super Seven" commands would be helpful.
See: https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac/m-p/40528

With most of your traffic hitting PXL (expected because of App Control, IPS, and Anti-Bot being active), some policy optimization may be required.

Re: Performance issues : Loss of packets

Ilya_Yusupov — Mon, 02 Sep 2019 18:43:07 GMT

according to your fw ctl multik stat you have only 2 FW instances, can you increase it?

Re: Performance issues : Loss of packets

Timothy_Hall — Mon, 02 Sep 2019 19:35:39 GMT

Need to see the "Super Seven" outputs as Dameon suggested, especially netstat -ni; my guess is your packet loss can be attributed to RX-DRPs. Also please identify which interface name is used for cluster sync.

The 5400 is a 2-core system which puts it between a rock and a hard place to some degree, the only possible CoreXL adjustment is to disable it thus producing a 1/1 split of SND/IRQ cores vs. Firewall Worker cores as opposed to your current default 2/2 split which causes cache thrashing on the cores under load due to overlapping functions.

Re: Performance issues : Loss of packets

HeikoAnkenbrand — Mon, 02 Sep 2019 20:35:42 GMT

Hi @Zia,

Could you see RX errors?

# netstat -in

Could you see CPU performance issues (software interruts or hw interrupts)?

# top + key 1

Which network card drivers are you use?

# ethtool -i ethX

On firewall 1 I can see 95% PXL traffic on firewall 2 I can see only 0% and heavy F2F traffic (100%). I think SecureXL is disabled on firewall 2. Check SecureXL on FW 2.

# fwaccel stat

Are deamons to be visible they generating high load?

# top

(More see here: Check Point Processes and Daemons)

Regards
Heiko

Re: Performance issues : Loss of packets

HeikoAnkenbrand — Mon, 02 Sep 2019 20:41:01 GMT

If you use R80.20+ check this:

# fw ctl multik utilize > shows the CoreXL queue utilization for each CoreXL FW instance

# fw ctl multik print_heavy_conn > shows the table with heavy connections

Re: Performance issues : Loss of packets

Timothy_Hall — Mon, 02 Sep 2019 22:43:13 GMT

> On firewall 1 I can see 95% PXL traffic on firewall 2 I can see only 0% and heavy F2F traffic (100%). I think SecureXL is disabled on firewall 2. Check SecureXL on FW 2.

Actually Heiko if Firewall-2 is the standby member in a ClusterXL HA cluster it is normal to see 100% F2F, as all traffic on that system is to and from the standby firewall itself which always goes F2F. So SecureXL is probably enabled on Firewall-2.

Re: Performance issues : Loss of packets

HeikoAnkenbrand — Tue, 03 Sep 2019 10:19:57 GMT

👍