topic Re: High latency after Checkpoint firewall from R77.30 to R80.10 in Firewall and Security Management

High latency after Check Point firewall from R77.30 to R80.10

Ben_Fung — Wed, 21 Mar 2018 14:53:40 GMT

As before we are running on CP R77.30 hardware model is 13500 with cluster appliance with smooth and normal performance. But after upgrade to R80.10 all network performance to slow down, for example, we have PRTG monitor (network via checkpoint) have monitor our website performance, on R77.30 the loading time around 5x-1xx ms, after R80.10 drop to 5xx - 4xxxx ms. The second case is we have ping test under our core switch, the interface plug into Checkpoint firewall directly. The R77.30 ping time is below 5ms, after R80.10 over 1x - 5x ms.

I know that's very simple evidence and test result. But how to be next step? Hope I can get more advice, Tks guys

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Vladimir — Wed, 21 Mar 2018 15:13:24 GMT

Can you verify that the cluster and the network interfaces are in a good state using cli?

the ICMP echo from adjacent switch should not be taking that long.

Are the cluster members connected to the HSRP pair of switches?

Do you have a vMAC enabled on the cluster?

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Ben_Fung — Wed, 21 Mar 2018 15:16:55 GMT

Seems no packet lost when I check in cli.

Our core switch is H3C, i remember it's using VRRP.

for vMAC, are you said the Checkpoint cluster enable vMAC?

Tks Vladimir

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Kaspars_Zibarts — Wed, 21 Mar 2018 16:00:49 GMT

Almost sounds like firewall under load. Have you checked CPU/ram/throughput? How do you run your ping from FW to switch or vice versa?

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Vladimir — Wed, 21 Mar 2018 16:22:02 GMT

Does the speed and duplex look good on all interfaces?

As to vMAC, you have option of enabling it:

Depending on topology of your network, use of STP, etc., it may make sense to enable it.

If you can, do the "tcpdump -tttt -ne host <IP of the cluster>" on your switch and check if the replies are coming from the expected interface of the active cluster member or vMAC.

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Timothy_Hall — Wed, 21 Mar 2018 16:46:50 GMT

Tough to say but it sounds like you may have cluster-related issues if latency has spiked that much after the upgrade. Easy way to determine that is:

1) Power off the standby cluster member. Does the latency situation improve? If it does, your problem is cluster-related and you need to have a look at logs with the "type:Control" log filter to see what is going on. If the cluster is not stable and constantly failing over you may need to set broadcast mode for CCP.

2) Next step is to ensure the network is running cleanly. Are all firewall interfaces running at 1Gbps or higher speed? Any Fast Ethernet (100Mbps) interfaces need to be checked for a duplex mismatch.

3) Was the firewall upgrade in-place (i.e. upgraded the same hardware directly from R77.30 to R80.10 with CPUSE) or forklift (new firewall hardware scratch-loaded and dropped into place). If it is the latter you may have lost some local adjustments in fwkern.conf or /etc/rc.local, or may be running with the default CoreXL split of 2/14 which almost always needs to be adjusted on a 13500 in the real world. You can check the CoreXL split with fw ctl affinity -l -r.

Failing all of those, please post the output from the following "Super Seven" commands run in expert mode on the primary/active cluster member during an observed period of high latency and I can advise further:

fwaccel stat

fwaccel stats -s

grep -c ^processor /proc/cpuinfo

fw ctl affinity -l -r

netstat -ni

fw ctl multik stat

cpstat os -f multi_cpu -o 1

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Ben_Fung — Thu, 22 Mar 2018 01:29:27 GMT

As checked the speed and duplex look good.

for vMAC, we haven't enable on ClusterXL. But it will make network down when enable this function? Thanks

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Ben_Fung — Thu, 22 Mar 2018 04:02:13 GMT

I can give some ping test result as below

CPU loading

Ping Result

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Kaspars_Zibarts — Thu, 22 Mar 2018 05:02:39 GMT

What about other commands that Tim asked for? Acceleration, interface stats, CoreXL?

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Kaspars_Zibarts — Thu, 22 Mar 2018 05:04:10 GMT

Also what interface speed is set on the interface you run ping on and what is the throughput there?

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Eyal_Rashelbach — Thu, 22 Mar 2018 08:09:30 GMT

Hi Ben,

Have you contact our support ?
if so, Please share SR#....

Best regards

Eyal Rashelbach

R80 Desk Manager | Solution Center | Check Point Software Technologies

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Ben_Fung — Fri, 23 Mar 2018 01:21:14 GMT

Hi Tim

Here is infromation follow your "Super Serven" command, Tks for your help, and know new terms of checkpoint "Super Serven".

[Expert@LMPRCPFW01:0]# fwaccel stat
Accelerator Status : on
Accept Templates : disabled by Firewall
Layer PDC_CP_Firewall Security disables template offloads from rule #116
Throughput acceleration still enabled.
Drop Templates : disabled
NAT Templates : disabled by user
NMR Templates : enabled
NMT Templates : enabled

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
McastRoutingV2, NMR, NMT, NAT64, GTPAcceleration,
SCTPAcceleration
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
[Expert@LMPRCPFW01:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/94 (0%)
Accelerated pkts/Total pkts : 0/6854226 (0%)
F2Fed pkts/Total pkts : 6854226/6854226 (100%)
PXL pkts/Total pkts : 0/6854226 (0%)
QXL pkts/Total pkts : 0/6854226 (0%)
[Expert@LMPRCPFW01:0]# grep -c ^processor /proc/cpuinfo
16
[Expert@LMPRCPFW01:0]# fw ctl affinity -l -r
CPU 0: eth2-01 eth2-02 eth1-07 eth1-08 eth2-04 Mgmt Sync
CPU 1: eth1-01 eth1-02 eth1-03 eth1-04 eth1-05 eth1-06 eth2-03
CPU 2: fw_13
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 3: fw_12
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 4: fw_11
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 5: fw_10
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 6: fw_9
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 7: fw_8
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 8: fw_7
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 9: fw_6
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 10: fw_5
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 11: fw_4
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 12: fw_3
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 13: fw_2
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 14: fw_1
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
CPU 15: fw_0
fwd pepd lpd usrchkd in.msd mpdaemon pdpd vpnd fwucd rad in.emaild.mta in.acapd cpd cprid
All:
[Expert@LMPRCPFW01:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 13702966 0 0 0 15962552 0 0 0 BMRU
Sync 1500 0 333650207 0 0 0 121769745 0 0 0 BMRU
bond2 1500 0 1441565252 1 0 0 1627548069 0 0 0 BMmRU
bond2.10 1500 0 238388484 0 0 0 147686142 0 0 0 BMmRU
bond2.163 1500 0 818903388 0 0 0 341775356 0 0 0 BMmRU
bond2.700 1500 0 384199869 0 0 0 214359826 0 0 0 BMmRU
bond3 1500 0 12815465 0 0 0 23478665 0 0 0 BMmRU
bond3.801 1500 0 4692593 0 0 0 4421277 0 0 0 BMmRU
bond3.1014 1500 0 1162293 0 0 0 595678 0 0 0 BMmRU
bond3.1209 1500 0 2280454 0 0 0 668717 0 0 0 BMmRU
bond3.1688 1500 0 4643707 0 0 0 4425602 0 0 0 BMmRU
eth1-01 1500 0 81281889 0 472 472 83202416 0 0 0 BMRU
eth1-02 1500 0 199227419 0 111 111 146528821 0 0 0 BMRU
eth1-03 1500 0 390331603 0 0 0 882904179 0 0 0 BMRU
eth1-04 1500 0 66707804 0 0 0 19987560 0 0 0 BMRU
eth1-04.210 1500 0 66707804 0 0 0 8818908 0 0 0 BMRU
eth1-05 1500 0 1012244443 0 0 0 443288453 0 0 0 BMRU
eth1-06 1500 0 4077432 0 0 0 4185548 0 0 0 BMRU
eth1-07 1500 0 8568255 0 0 0 11757975 0 0 0 BMRU
eth1-08 1500 0 333934975 0 0 0 122084202 0 0 0 BMRU
eth2-01 1500 0 724739031 1 0 0 508740663 0 0 0 BMsRU
eth2-02 1500 0 716826232 0 0 0 1118807427 0 0 0 BMsRU
eth2-03 1500 0 11301424 0 0 0 11329734 0 0 0 BMsRU
eth2-04 1500 0 1514041 0 0 0 12148931 0 0 0 BMsRU
lo 16436 0 2623169 0 0 0 2623169 0 0 0 LRU
[Expert@LMPRCPFW01:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 15 | 6753 | 8506
1 | Yes | 14 | 3750 | 5886
2 | Yes | 13 | 1529 | 6240
3 | Yes | 12 | 3078 | 6010
4 | Yes | 11 | 5029 | 6191
5 | Yes | 10 | 4485 | 6400
6 | Yes | 9 | 4878 | 6110
7 | Yes | 8 | 4577 | 6070
8 | Yes | 7 | 4711 | 5961
9 | Yes | 6 | 3953 | 6203
10 | Yes | 5 | 4908 | 6233
11 | Yes | 4 | 3546 | 6288
12 | Yes | 3 | 4134 | 5883
13 | Yes | 2 | 3878 | 6141

[Expert@LMPRCPFW01:0]#cpstat os -f multi_cpu -o 1
Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 2| 99| 1| ?| 3313|
| 2| 0| 0| 100| 0| ?| 3313|
| 3| 1| 1| 98| 2| ?| 3313|
| 4| 2| 3| 96| 4| ?| 3313|
| 5| 1| 1| 98| 2| ?| 3313|
| 6| 0| 1| 98| 2| ?| 3313|
| 7| 0| 0| 100| 0| ?| 3314|
| 8| 0| 1| 100| 0| ?| 3315|
| 9| 0| 1| 99| 1| ?| 3315|
| 10| 2| 2| 96| 4| ?| 3315|
| 11| 0| 1| 98| 2| ?| 3315|
| 12| 0| 1| 98| 2| ?| 3315|
| 13| 0| 1| 100| 0| ?| 3315|
| 14| 0| 1| 99| 1| ?| 3315|
| 15| 0| 1| 99| 1| ?| 3315|
| 16| 0| 1| 99| 1| ?| 3315|
---------------------------------------------------------------------------------

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Ben_Fung — Fri, 23 Mar 2018 01:25:31 GMT

The case now handle by the local distributor, I think they haven't created SR at this moment. the local distributor has arrange the local / remote session but no finding. So I want to find another channel to help this case.

Anyway, on remote session yesterday, I very strange that why the local distributor TAC said ping high latency is normal at CheckPoint firewall.

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Ben_Fung — Fri, 23 Mar 2018 01:27:51 GMT

All interface speed is 1G

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Ben_Fung — Fri, 23 Mar 2018 01:32:30 GMT

one more test. on Checkpoint firewall active node ping the local interface, seems lager ping time suddenly.

Re: High latency after Checkpoint firewall from R77.30 to R80.10

RickLin — Fri, 23 Mar 2018 02:53:56 GMT

Hi Ben

Do you ever try to check ring buffer size issue?

sk42181

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Kaspars_Zibarts — Fri, 23 Mar 2018 05:35:18 GMT

Acceleration is not working at all (neither fast path nor medium) so you need to dig into that

F2Fed pkts/Total pkts : 6854226/6854226 (100%)

Pls add full stats

faccel stats

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Kaspars_Zibarts — Fri, 23 Mar 2018 06:11:43 GMT

You may read and try yourself advance guide for SecureXL ATRG: SecureXL

But good start would be cpview tool and

On the 'SysInfo' tab, refer to 'Configuration Information:' section - look at 'PPack Status'
On the 'Traffic' tab, go to 'Overview' menu - refer to section 'Templates:'
On the 'I/S' tab, go to 'SXL' menu - go to 'Overview' menu

Re: High latency after Checkpoint firewall from R77.30 to R80.10

Kaspars_Zibarts — Fri, 23 Mar 2018 06:14:52 GMT

Send a screenshot of rule 116 btw

Re: High latency after Checkpoint firewall from R77.30 to R80.10

G_W_Albrecht — Fri, 23 Mar 2018 10:50:43 GMT

I would suggest to consult sk98348: Best Practices - Security Gateway Performance - Including some useful tips for SecureXL, CoreXL, HyperThreading, Multi-Queue, and some references to other very good SKs about ClusterXL or VPN.