VPN disturbances

[Moudar]

Hi,

I’ve received reports from clients experiencing unstable VPN connections (remote clients).

Upon reviewing the VPND logs, I found the following:

 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory
 SvcSk_close: refraining from closing socket -1
 httpMime_destroy_db: Destroying the MIME database.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: A handler [0x80e18e0] blocked for 6 seconds.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: Handler info: Library [vpnd], Function offset [0x1e8e0].
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: A handler [0xf6019d40] blocked for 6 seconds.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: Handler info: Library [/opt/CPshrd-R81.20/lib/libmessaging.so], Function offset [0x1d40].
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: A handler [0xf5a82d50] blocked for 6 seconds.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: Handler info: Library [/opt/CPshrd-R81.20/lib/libComUtils.so], Function offset [0x1f350].
 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory

What factors can cause a handler to become blocked?

What is the /dev/fw6v0 device, and why might it be missing?

Could a blocked handler impact the VPN client experience, potentially causing connection issues?

[G_W_Albrecht]

What is the issue in detail, unstable RA VPN usually means disconnect and reconnect ? I found sk181526: Intermittent authentication failures to AzureAD on an Identity Awareness Seurity Gateway with this error message.

[Moudar]

The link is about Identity Awareness (pdpd.elg)! Even though you have right it is the same log!

Yes, disconnect and reconnect

[PhoneBoy]

What version/JHF level?
Perhaps a Super Seven output would be useful here, as it could be performance related: https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/m-p/40...

[Moudar]

81.20 take 76

[Moudar]

I've now run the 7-command script. What stands out to me as particularly interesting is:

Accelerated conns/Total conns    : 105/52071 (0%)

here is fawaccel stat:

 fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name     |Status     |Interfaces               |Features                      |
+---------------------------------------------------------------------------------+
|0 |KPPAK    |enabled    |Sync,Mgmt,eth1-01,       |Acceleration,Cryptography     |
|  |         |           |eth1-03,eth1-04          |                              |
|  |         |           |                         |Crypto: Tunnel,UDPEncap,MD5,  |
|  |         |           |                         |SHA1,3DES,DES,AES-128,AES-256,|
|  |         |           |                         |ESP,LinkSelection,DynamicVPN, |
|  |         |           |                         |NatTraversal,AES-XCBC,SHA256, |
|  |         |           |                         |SHA384,SHA512                 |
+---------------------------------------------------------------------------------+

Accept Templates : disabled by Firewall
                   Layer Network disables template offloads from rule #62
                   Throughput acceleration still enabled.
Drop Templates   : enabled
NAT Templates    : disabled by Firewall

Rule 62 is for SMTP, and it's challenging to move it down!

here is fwaccel stats -s:

fwaccel stat -s

fwaccel: illegal option -- s
Invalid option '?'
[Expert@fw01:0]# fwaccel stats -s
Accelerated conns/Total conns    : 105/49378 (0%)
LightSpeed conns/Total conns     : 0/49378 (0%)
Accelerated pkts/Total pkts      : 46429315584/53547009472 (86%)
LightSpeed pkts/Total pkts       : 0/53547009472 (0%)
F2Fed pkts/Total pkts            : 7117693888/53547009472 (13%)
F2V pkts/Total pkts              : 255709233/53547009472 (0%)
CPASXL pkts/Total pkts           : 2084716164/53547009472 (3%)
PSLXL pkts/Total pkts            : 42403141530/53547009472 (79%)
CPAS pipeline pkts/Total pkts    : 0/53547009472 (0%)
PSL pipeline pkts/Total pkts     : 0/53547009472 (0%)
QOS inbound pkts/Total pkts      : 0/53547009472 (0%)
QOS outbound pkts/Total pkts     : 0/53547009472 (0%)
Corrected pkts/Total pkts        : 0/53547009472 (0%)

so, any suggestions?

[PhoneBoy]

What exactly is the rule that is disabling templates?
A screenshot (with sensitive details blurred) will help.

The fact that (almost) no connections are accelerated is at least contributing to the issue, if not the actual cause.

[Moudar]

It is a rule that allows connection (source) from Microsoft Outlook worldwide among other services to our Mail servers (destination) with services and application = smtp 

[the_rock]

You can try disable securexl and test. If that works, then I would open TAC case and investigate further.

Andy

[PhoneBoy]

Doubt it will help in this case since almost none of his connections are accelerated as it is and disabling SecureXL only disables the templating.

[the_rock]

Makes sense.

[PhoneBoy]

Even though it's not documented anywhere, I suspect it's the object in the Destination field (Logical Server) that is causing traffic to not be templated.

I have an idea of how to work around this, but it involves creating an inline layer (change action from Accept to an Inline Layer, create a new one with just Firewall blade active). 
The top level rule involves a regular host object (ls_smtp_ip) that is the same IP as ls_smtp (the Logical Server object).
The UI will give you a warning when you create an object with the same IP (which is expected).
The inline layer will only be evaluated if the top level rule matches, which is why the rules in the inline layer aren't specific.
When it's all said and done, it should look something like this:

What should happen (assuming installing policy isn't blocked with this configuration) is that the traffic matched by rules below this should now template.
That would mean the percentage of Accelerated Connections should dramatically increase.

[Moudar]

Only Firewall is chosen in the inline layer settings!

Attempting to move the rule down fails due to the presence of multiple inline layers:

So, what other options do we have?

[Tal_Paz-Fridman]

Are you able to install policy with this Validation Error? It should not allow to Publish or Install Policy

[Moudar]

No, the policy failed to install with the above error message

[the_rock]

I dont think it has anything to do the with the blade selected, its telling you that object can NOT be used in that inline layer, for whatever reason. Though, if you check solution from the sk indicated, its pretty clear why.

Andy

[PhoneBoy]

I remembered that SK existed, but forgot to check it for suggesting this workaround.

Sadly, the only other suggestion I have is to not use a Logical Server object.
With almost all of your connections not templating, you'll probably have other performance-related issues down the road.

[Moudar]

That's unfortunate. How do other companies address this issue?

[PhoneBoy]

Logical Server objects are rarely used, therefore the limitations they impose are rarely encountered.
Load balancing access to SMTP servers is usually done via DNS.

[Moudar]

We have now removed the logical server object, and the SecureXL status currently appears as follows:

fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name     |Status     |Interfaces               |Features                      |
+---------------------------------------------------------------------------------+
|0 |KPPAK    |enabled    |Sync,Mgmt,eth1-01,       |Acceleration,Cryptography     |
|  |         |           |eth1-03,eth1-04          |                              |
|  |         |           |                         |Crypto: Tunnel,UDPEncap,MD5,  |
|  |         |           |                         |SHA1,3DES,DES,AES-128,AES-256,|
|  |         |           |                         |ESP,LinkSelection,DynamicVPN, |
|  |         |           |                         |NatTraversal,AES-XCBC,SHA256, |
|  |         |           |                         |SHA384,SHA512                 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates   : enabled
NAT Templates    : enabled
LightSpeed Accel : disabled

It has been running for 3 days, but the "Accelerated conns/Total conns" ratio remains very low:

fwaccel stats -s
Accelerated conns/Total conns    : 1013/58378 (1%)
LightSpeed conns/Total conns     : 0/58378 (0%)
Accelerated pkts/Total pkts      : 66915308684/75883929440 (88%)
LightSpeed pkts/Total pkts       : 0/75883929440 (0%)
F2Fed pkts/Total pkts            : 8968620756/75883929440 (11%)
F2V pkts/Total pkts              : 357709365/75883929440 (0%)
CPASXL pkts/Total pkts           : 2824937581/75883929440 (3%)
PSLXL pkts/Total pkts            : 61384916652/75883929440 (80%)
CPAS pipeline pkts/Total pkts    : 0/75883929440 (0%)
PSL pipeline pkts/Total pkts     : 0/75883929440 (0%)
QOS inbound pkts/Total pkts      : 0/75883929440 (0%)
QOS outbound pkts/Total pkts     : 0/75883929440 (0%)
Corrected pkts/Total pkts        : 0/75883929440 (0%)

The gateway is 6500:

This is Check Point CPinfo Build 914000248 for GAIA
[MGMT]
        HOTFIX_R81_20_JUMBO_HF_MAIN     Take:  65
[IDA]
        No hotfixes..
[CPFC]
        No hotfixes..
[FW1]
        HOTFIX_R81_20_JHF_T53_BLOCK_PORTAL_MAIN Take:  2
        HOTFIX_GOT_TPCONF_AUTOUPDATE
        HOTFIX_R80_40_MAAS_TUNNEL_AUTOUPDATE
        HOTFIX_R81_20_JUMBO_HF_MAIN     Take:  65
        HOTFIX_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE

any ideas?!

[Timothy_Hall]

This topic is fully covered in my Gateway Performance Optimization Course.  fwaccel stat showing Accept Templates are "enabled" yet the templating rate (Accelerated conns) is zero or very close to zero can be investigated with a new command line switch introduced in R81.20 GA: fwaccel templates -R

Generally "Prevented by Policy Rules" means one or both of the following is occurring:

1) In Access Control you have a blade other than "Firewall" enabled in your first/top policy layer in the case of Ordered Layers, or you have a blade other than "Firewall" enabled in the top/parent layer of a unified/inline policy implementation.  In either case the firewall policy has been made capable of matching connections using applications/categories/data types instead of just straight port numbers, which SecureXL templating cannot handle and the Accelerated conns rate goes to zero.

2) You have enabled the "Protocol Signature" option on a service object that is used in your policy.  This option is never set by default.  Doing this requires streaming inspection to be invoked for a full rulebase match for protocol verification, which once again is incompatible with SecureXL templating.

Please note that the situations above DO NOT impact SecureXL Throughput Acceleration at all, in other words these conditions do not force traffic into a less efficient path such as F2F/slowpath, they only affect templating.

[Moudar]

Here is how it look like with: fwaccel templates -R

fwaccel templates -R

Matched connections not allowed to use templates:
% Prevention : 0.764%

Reason                                  Count      Reason Prevented From Matched %

Non-Syn/Empty First Packet              |128015    |0.635     %
Src/dst IP Blacklisted                  |25854     |0.128     %
Dynamic VPN Connection                  |2         |0.000     %
--------------------

Connections failed to create templates:
% Fail to Create : 84.743%

Reason                                  Count      Reason Fail To Create %

NON TCP/UDP PROTO                       |4377809   |1.202     %
Conn Not Accelerated                    |7985235   |2.193     %
NAT Disallowed Conn                     |23091058  |6.341     %
General Error                           |870977    |0.239     %
Malicious Destination IP Detected       |249686    |0.069     %
Prevented By Policy Rules               |272030517 |74.699    %
-------------------

I've located my inline layers in Access Control that have the Applications & URL Filtering blade enabled. I've now unchecked the URL Filtering option, and I will update you with the results shortly

[Moudar]

Prevented by policy rules is going down very slowly, so 5% in 3 days! after removing URL blade from some inline rules we have.

fwaccel templates -R

Matched connections not allowed to use templates:
% Prevention : 1.239%

Reason                                  Count      Reason Prevented From Matched %

Non-Syn/Empty First Packet              |181618    |0.712     %
Src/dst IP Blacklisted                  |134396    |0.527     %
Dynamic VPN Connection                  |2         |0.000     %
--------------------

Connections failed to create templates:
% Fail to Create : 82.093%

Reason                                  Count      Reason Fail To Create %

NON TCP/UDP PROTO                       |4524945   |1.159     %
Conn Not Accelerated                    |8558463   |2.193     %
NAT Disallowed Conn                     |34132098  |8.744     %
DHCP Check Feature Isn't Supported Or Disabled|10        |0.000     %
General Error                           |921701    |0.236     %
Malicious Destination IP Detected       |259938    |0.067     %
Prevented By Policy Rules               |272047377 |69.694    %

But Accelerated conns/Total conns : 556/61263 (0%) still shows 0%!

Maybe I need to wait a week or two to see some result?

fwaccel stats -s
Accelerated conns/Total conns    : 556/61263 (0%)
LightSpeed conns/Total conns     : 0/61263 (0%)
Accelerated pkts/Total pkts      : 73234561042/82749335447 (88%)
LightSpeed pkts/Total pkts       : 0/82749335447 (0%)
F2Fed pkts/Total pkts            : 9514774405/82749335447 (11%)
F2V pkts/Total pkts              : 395727119/82749335447 (0%)
CPASXL pkts/Total pkts           : 3041455470/82749335447 (3%)
PSLXL pkts/Total pkts            : 67212887275/82749335447 (81%)
CPAS pipeline pkts/Total pkts    : 0/82749335447 (0%)
PSL pipeline pkts/Total pkts     : 0/82749335447 (0%)
QOS inbound pkts/Total pkts      : 0/82749335447 (0%)
QOS outbound pkts/Total pkts     : 0/82749335447 (0%)
Corrected pkts/Total pkts        : 0/82749335447 (0%)