At times get "the site can't be reached"...

Deepraj_Patil · ‎2024-09-26

Over the past few weeks, we've noticed intermittent 'the site can't be reached' errors for some users. These errors often resolve themselves with a refresh or appear spontaneously. Despite successful ping and nslookup tests, the issue persists. There are no packet drops or internet connectivity problems. Our network environment includes authenticated domain users with internet access, an identity collector on the server, and all domain controllers added to the collector engine.

So can someone guide me or help me in identifying the issue and to fix this issue.

Thank you in advance.

PhoneBoy · ‎2024-09-26

Version/JHF?
Output of enabled_blades command?
Also super seven output would be helpful: https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/m-p/40...

Deepraj_Patil · ‎2024-09-26

Version 81.20 Jumbo Hot Fix Take 76.

Outout of enabled_blades:

[Expert@XXXXXXXXXXX]# enabled_blades
fw vpn cvpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation content_awareness mon Scrub

Accept Templates : enabled
Drop Templates : disabled
NAT Templates : enabled
LightSpeed Accel : disabled

[Expert@XXXXXXXXXXX]# fwaccel stats -s
Accelerated conns/Total conns : 1/13775 (0%)
LightSpeed conns/Total conns : 0/13775 (0%)
Accelerated pkts/Total pkts : 654639227/2274138001 (28%)
LightSpeed pkts/Total pkts : 0/2274138001 (0%)
F2Fed pkts/Total pkts : 1619498774/2274138001 (71%)
F2V pkts/Total pkts : 6625658/2274138001 (0%)
CPASXL pkts/Total pkts : 369901856/2274138001 (16%)
PSLXL pkts/Total pkts : 284639260/2274138001 (12%)
CPAS pipeline pkts/Total pkts : 0/2274138001 (0%)
PSL pipeline pkts/Total pkts : 0/2274138001 (0%)
QOS inbound pkts/Total pkts : 0/2274138001 (0%)
QOS outbound pkts/Total pkts : 0/2274138001 (0%)
Corrected pkts/Total pkts : 0/2274138001 (0%)

[Expert@XXXXXXXXXXX]# grep -c ^processor /proc/cpuinfo
4

[Expert@XXXXXXXXXXX]# fw ctl affinity -l -r
CPU 0: eth1 eth2 eth6 eth3 eth7 eth8 Sync Mgmt
CPU 1: fw_2 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 2: fw_1 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 3: fw_0 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
All:

[Expert@XXXXXXXXXXX]# fw ctl affinity -l -r
CPU 0: eth1 eth2 eth6 eth3 eth7 eth8 Sync Mgmt
CPU 1: fw_2 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 2: fw_1 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 3: fw_0 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
All:
[Expert@SGAPUNE-CPFW01:0]# fw ctl affinity -l -r
CPU 0: eth1 eth2 eth6 eth3 eth7 eth8 Sync Mgmt
CPU 1: fw_2 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 2: fw_1 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 3: fw_0 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
All:

[Expert@XXXXXXXXXXX]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 6769108 0 0 0 9772102 0 0 0 BMRU
Sync 1500 0 247734611 0 0 0 116223983 0 0 0 BMRU
eth1 1500 0 87270644 0 0 0 90832101 0 0 0 BMRU
eth2 1500 0 587442539 0 111 111 858732289 0 0 0 BMRU
eth2.200 1500 0 587431184 0 0 0 858732416 0 88 0 BMRU
eth3 1500 0 6379837 0 0 0 10186615 0 0 0 BMRU
eth6 1500 0 11679951 0 0 0 2378101 0 0 0 BMRU
eth7 1500 0 324835379 0 0 0 161929835 0 0 0 BMRU
eth8 1500 0 491893470 0 0 0 440956673 0 0 0 BMRU
lo 65536 0 10206078 0 0 0 10206078 0 0 0 ALMdNRU

Version 81.20 Jumbo Hot Fix Take 76.

Outout of enabled_blades:

[Expert@XXXXXXXXXXX]# enabled_blades
fw vpn cvpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation content_awareness mon Scrub

Accept Templates : enabled
Drop Templates : disabled
NAT Templates : enabled
LightSpeed Accel : disabled

[Expert@XXXXXXXXXXX]# fwaccel stats -s
Accelerated conns/Total conns : 1/13775 (0%)
LightSpeed conns/Total conns : 0/13775 (0%)
Accelerated pkts/Total pkts : 654639227/2274138001 (28%)
LightSpeed pkts/Total pkts : 0/2274138001 (0%)
F2Fed pkts/Total pkts : 1619498774/2274138001 (71%)
F2V pkts/Total pkts : 6625658/2274138001 (0%)
CPASXL pkts/Total pkts : 369901856/2274138001 (16%)
PSLXL pkts/Total pkts : 284639260/2274138001 (12%)
CPAS pipeline pkts/Total pkts : 0/2274138001 (0%)
PSL pipeline pkts/Total pkts : 0/2274138001 (0%)
QOS inbound pkts/Total pkts : 0/2274138001 (0%)
QOS outbound pkts/Total pkts : 0/2274138001 (0%)
Corrected pkts/Total pkts : 0/2274138001 (0%)

[Expert@XXXXXXXXXXX]# grep -c ^processor /proc/cpuinfo
4

[Expert@XXXXXXXXXXX]# fw ctl affinity -l -r
CPU 0: eth1 eth2 eth6 eth3 eth7 eth8 Sync Mgmt
CPU 1: fw_2 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 2: fw_1 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 3: fw_0 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
All:

[Expert@XXXXXXXXXXX]# fw ctl affinity -l -r
CPU 0: eth1 eth2 eth6 eth3 eth7 eth8 Sync Mgmt
CPU 1: fw_2 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 2: fw_1 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 3: fw_0 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
All:
[Expert@SGAPUNE-CPFW01:0]# fw ctl affinity -l -r
CPU 0: eth1 eth2 eth6 eth3 eth7 eth8 Sync Mgmt
CPU 1: fw_2 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 2: fw_1 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
CPU 3: fw_0 (active)
watermark_cp_file_convertd cprid mpdaemon fwd rad cp_file_convertd vpnd wsdnsd rtmd lpd core_uploader usrchkd in.asessiond scrubd pdpd in.acapd scrub_cp_file_convertd scanengine_b pepd cprid cpd msgd
All:

[Expert@XXXXXXXXXXX]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
-----------------------------------------------
0 | Yes | 3 | 6798 | 9257
1 | Yes | 2 | 6756 | 11156
2 | Yes | 1 | 6973 | 11116

[Expert@XXXXXXXXXXX]#

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 7| 12| 82| 18| ?| 59727|
| 2| 41| 5| 54| 46| ?| 59728|
| 3| 44| 5| 51| 49| ?| 59729|
| 4| 46| 7| 48| 52| ?| 59726|

PhoneBoy · ‎2024-09-26

A majority of the packets are going F2F (slowpath), which would likely account for this.
Is ISP Redundancy used?
What does fwaccel templates -R say?

Deepraj_Patil · ‎2024-09-26

We are using both the internet connections in load balance mode.

Matched connections not allowed to use templates:
% Prevention : 2.591%

Reason Count Reason Prevented From Matched %

Non-Syn/Empty First Packet |119 |0.006 %
Src/dst IP Blacklisted |48793 |2.585 %
--------------------

Connections failed to create templates:
% Fail to Create : 98.875%

Reason Count Reason Fail To Create %

MISP Is Active |3726114 |59.806 %
NON TCP/UDP PROTO |15622 |0.251 %
Conn Not Accelerated |339041 |5.442 %
NAT Disallowed Conn |5632 |0.090 %
General Error |1951596 |31.324 %
Connection Policy Fetch Retry |6 |0.000 %
Malicious Destination IP Detected |830 |0.013 %
Prevented By Policy Rules |121362 |1.948 %

PhoneBoy · ‎2024-09-26

ISP Redundancy prevents traffic to/from the relevant interfaces from being accelerated by SecureXL.
This would explain the large F2F percentage.

Possible changing the CoreXL split might provide a little more headroom (maybe @Timothy_Hall has experience with this).
Otherwise, not sure what else can be done here short of disabling ISP Redundancy.

Timothy_Hall · ‎2024-09-26

Indeed ISP Redundancy in Load Sharing mode will cause a high amount of F2F/slowpath traffic and interfere with Accept Templates too. If the bandwidth is asymmetric on your ISP links (cable modem/DSL especially) and/or the bandwidth weights are not set appropriately these can wreak havoc on performance with Load Sharing mode. You have quite a few blades enabled too with only 4 cores which is not helping. I don't think a CoreXL static split adjustment will help as the single SND seems to be doing OK handling NIC traffic.

What I would suggest is try setting ISP Redundancy to Primary/Backup mode (ISP with highest bandwidth as the primary) and see if the situation improves. This mode will not interfere with SecureXL.

Attend my online "Be your Own TAC: Part Deux" CheckMates event
March 27th with sessions for both the EMEA and Americas time zones

Are you a member of CheckMates?

At times get &quot;the site can't be reached&quot; when accessing the internet.

At times get "the site can't be reached" when accessing the internet.