Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Moudar
Advisor

VPN disturbances

Hi,

I’ve received reports from clients experiencing unstable VPN connections (remote clients).

Upon reviewing the VPND logs, I found the following:

 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory
 SvcSk_close: refraining from closing socket -1
 httpMime_destroy_db: Destroying the MIME database.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: A handler [0x80e18e0] blocked for 6 seconds.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: Handler info: Library [vpnd], Function offset [0x1e8e0].
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: A handler [0xf6019d40] blocked for 6 seconds.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: Handler info: Library [/opt/CPshrd-R81.20/lib/libmessaging.so], Function offset [0x1d40].
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: A handler [0xf5a82d50] blocked for 6 seconds.
[vpnd 2393]@fw01[10 Sep 11:05:18] Warning:cp_timed_blocker_handler: Handler info: Library [/opt/CPshrd-R81.20/lib/libComUtils.so], Function offset [0x1f350].
 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory
 Unable to open '/dev/fw6v0': No such file or directory

What factors can cause a handler to become blocked?

What is the /dev/fw6v0 device, and why might it be missing?

Could a blocked handler impact the VPN client experience, potentially causing connection issues?

0 Kudos
22 Replies
G_W_Albrecht
Legend Legend
Legend

What is the issue in detail, unstable RA VPN usually means disconnect and reconnect ? I found sk181526: Intermittent authentication failures to AzureAD on an Identity Awareness Seurity Gateway with this error message.

CCSE / CCTE / CTPS / CCME / CCSM Elite / SMB Specialist
0 Kudos
Moudar
Advisor

The link is about Identity Awareness (pdpd.elg)! Even though you have right it is the same log!

Yes, disconnect and reconnect

0 Kudos
PhoneBoy
Admin
Admin

What version/JHF level?
Perhaps a Super Seven output would be useful here, as it could be performance related: https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/m-p/40...

0 Kudos
Moudar
Advisor

81.20 take 76

0 Kudos
Moudar
Advisor

I've now run the 7-command script. What stands out to me as particularly interesting is:

Accelerated conns/Total conns    : 105/52071 (0%)

here is fawaccel stat:

 fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name     |Status     |Interfaces               |Features                      |
+---------------------------------------------------------------------------------+
|0 |KPPAK    |enabled    |Sync,Mgmt,eth1-01,       |Acceleration,Cryptography     |
|  |         |           |eth1-03,eth1-04          |                              |
|  |         |           |                         |Crypto: Tunnel,UDPEncap,MD5,  |
|  |         |           |                         |SHA1,3DES,DES,AES-128,AES-256,|
|  |         |           |                         |ESP,LinkSelection,DynamicVPN, |
|  |         |           |                         |NatTraversal,AES-XCBC,SHA256, |
|  |         |           |                         |SHA384,SHA512                 |
+---------------------------------------------------------------------------------+

Accept Templates : disabled by Firewall
                   Layer Network disables template offloads from rule #62
                   Throughput acceleration still enabled.
Drop Templates   : enabled
NAT Templates    : disabled by Firewall

Rule 62 is for SMTP, and it's challenging to move it down!

here is fwaccel stats -s:

fwaccel stat -s

fwaccel: illegal option -- s
Invalid option '?'
[Expert@fw01:0]# fwaccel stats -s
Accelerated conns/Total conns    : 105/49378 (0%)
LightSpeed conns/Total conns     : 0/49378 (0%)
Accelerated pkts/Total pkts      : 46429315584/53547009472 (86%)
LightSpeed pkts/Total pkts       : 0/53547009472 (0%)
F2Fed pkts/Total pkts            : 7117693888/53547009472 (13%)
F2V pkts/Total pkts              : 255709233/53547009472 (0%)
CPASXL pkts/Total pkts           : 2084716164/53547009472 (3%)
PSLXL pkts/Total pkts            : 42403141530/53547009472 (79%)
CPAS pipeline pkts/Total pkts    : 0/53547009472 (0%)
PSL pipeline pkts/Total pkts     : 0/53547009472 (0%)
QOS inbound pkts/Total pkts      : 0/53547009472 (0%)
QOS outbound pkts/Total pkts     : 0/53547009472 (0%)
Corrected pkts/Total pkts        : 0/53547009472 (0%)

 

so, any suggestions?

0 Kudos
PhoneBoy
Admin
Admin

What exactly is the rule that is disabling templates?
A screenshot (with sensitive details blurred) will help.

The fact that (almost) no connections are accelerated is at least contributing to the issue, if not the actual cause.

0 Kudos
Moudar
Advisor

It is a rule that allows connection (source) from Microsoft Outlook worldwide among other services to our Mail servers (destination) with services and application = smtp 

smtp.JPG

0 Kudos
the_rock
Legend
Legend

You can try disable securexl and test. If that works, then I would open TAC case and investigate further.

Andy

0 Kudos
PhoneBoy
Admin
Admin

Doubt it will help in this case since almost none of his connections are accelerated as it is and disabling SecureXL only disables the templating.

the_rock
Legend
Legend

Makes sense.

0 Kudos
PhoneBoy
Admin
Admin

Even though it's not documented anywhere, I suspect it's the object in the Destination field (Logical Server) that is causing traffic to not be templated.

I have an idea of how to work around this, but it involves creating an inline layer (change action from Accept to an Inline Layer, create a new one with just Firewall blade active). 
The top level rule involves a regular host object (ls_smtp_ip) that is the same IP as ls_smtp (the Logical Server object).
The UI will give you a warning when you create an object with the same IP (which is expected).
The inline layer will only be evaluated if the top level rule matches, which is why the rules in the inline layer aren't specific.
When it's all said and done, it should look something like this:

image.png

What should happen (assuming installing policy isn't blocked with this configuration) is that the traffic matched by rules below this should now template.
That would mean the percentage of Accelerated Connections should dramatically increase.

0 Kudos
Moudar
Advisor

policy-install-failed.JPG

 

Only Firewall is chosen in the inline layer settings!

 

Attempting to move the rule down fails due to the presence of multiple inline layers:

inline.JPG

So, what other options do we have?

0 Kudos
Tal_Paz-Fridman
Employee
Employee

Are you able to install policy with this Validation Error? It should not allow to Publish or Install Policy

0 Kudos
Moudar
Advisor

No, the policy failed to install with the above error message

0 Kudos
the_rock
Legend
Legend

I dont think it has anything to do the with the blade selected, its telling you that object can NOT be used in that inline layer, for whatever reason. Though, if you check solution from the sk indicated, its pretty clear why.

Andy

0 Kudos
PhoneBoy
Admin
Admin

I remembered that SK existed, but forgot to check it for suggesting this workaround.

Sadly, the only other suggestion I have is to not use a Logical Server object.
With almost all of your connections not templating, you'll probably have other performance-related issues down the road.

0 Kudos
Moudar
Advisor

That's unfortunate. How do other companies address this issue?

0 Kudos
PhoneBoy
Admin
Admin

Logical Server objects are rarely used, therefore the limitations they impose are rarely encountered.
Load balancing access to SMTP servers is usually done via DNS.

0 Kudos
Moudar
Advisor

We have now removed the logical server object, and the SecureXL status currently appears as follows:

 

fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name     |Status     |Interfaces               |Features                      |
+---------------------------------------------------------------------------------+
|0 |KPPAK    |enabled    |Sync,Mgmt,eth1-01,       |Acceleration,Cryptography     |
|  |         |           |eth1-03,eth1-04          |                              |
|  |         |           |                         |Crypto: Tunnel,UDPEncap,MD5,  |
|  |         |           |                         |SHA1,3DES,DES,AES-128,AES-256,|
|  |         |           |                         |ESP,LinkSelection,DynamicVPN, |
|  |         |           |                         |NatTraversal,AES-XCBC,SHA256, |
|  |         |           |                         |SHA384,SHA512                 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates   : enabled
NAT Templates    : enabled
LightSpeed Accel : disabled

 

 

It has been running for 3 days, but the "Accelerated conns/Total conns" ratio remains very low:

fwaccel stats -s
Accelerated conns/Total conns    : 1013/58378 (1%)
LightSpeed conns/Total conns     : 0/58378 (0%)
Accelerated pkts/Total pkts      : 66915308684/75883929440 (88%)
LightSpeed pkts/Total pkts       : 0/75883929440 (0%)
F2Fed pkts/Total pkts            : 8968620756/75883929440 (11%)
F2V pkts/Total pkts              : 357709365/75883929440 (0%)
CPASXL pkts/Total pkts           : 2824937581/75883929440 (3%)
PSLXL pkts/Total pkts            : 61384916652/75883929440 (80%)
CPAS pipeline pkts/Total pkts    : 0/75883929440 (0%)
PSL pipeline pkts/Total pkts     : 0/75883929440 (0%)
QOS inbound pkts/Total pkts      : 0/75883929440 (0%)
QOS outbound pkts/Total pkts     : 0/75883929440 (0%)
Corrected pkts/Total pkts        : 0/75883929440 (0%)

 

 

 

The gateway is 6500:

 

This is Check Point CPinfo Build 914000248 for GAIA
[MGMT]
        HOTFIX_R81_20_JUMBO_HF_MAIN     Take:  65
[IDA]
        No hotfixes..
[CPFC]
        No hotfixes..
[FW1]
        HOTFIX_R81_20_JHF_T53_BLOCK_PORTAL_MAIN Take:  2
        HOTFIX_GOT_TPCONF_AUTOUPDATE
        HOTFIX_R80_40_MAAS_TUNNEL_AUTOUPDATE
        HOTFIX_R81_20_JUMBO_HF_MAIN     Take:  65
        HOTFIX_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE

 

any ideas?!

 

0 Kudos
Timothy_Hall
Legend Legend
Legend

This topic is fully covered in my Gateway Performance Optimization Coursefwaccel stat showing Accept Templates are "enabled" yet the templating rate (Accelerated conns) is zero or very close to zero can be investigated with a new command line switch introduced in R81.20 GA: fwaccel templates -R

templatesR.jpg

Generally "Prevented by Policy Rules" means one or both of the following is occurring:

1) In Access Control you have a blade other than "Firewall" enabled in your first/top policy layer in the case of Ordered Layers, or you have a blade other than "Firewall" enabled in the top/parent layer of a unified/inline policy implementation.  In either case the firewall policy has been made capable of matching connections using applications/categories/data types instead of just straight port numbers, which SecureXL templating cannot handle and the Accelerated conns rate goes to zero.

2) You have enabled the "Protocol Signature" option on a service object that is used in your policy.  This option is never set by default.  Doing this requires streaming inspection to be invoked for a full rulebase match for protocol verification, which once again is incompatible with SecureXL templating.

Please note that the situations above DO NOT impact SecureXL Throughput Acceleration at all, in other words these conditions do not force traffic into a less efficient path such as F2F/slowpath, they only affect templating.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
Moudar
Advisor

Here is how it look like with: fwaccel templates -R

fwaccel templates -R

Matched connections not allowed to use templates:
% Prevention : 0.764%

Reason                                  Count      Reason Prevented From Matched %

Non-Syn/Empty First Packet              |128015    |0.635     %
Src/dst IP Blacklisted                  |25854     |0.128     %
Dynamic VPN Connection                  |2         |0.000     %
--------------------

Connections failed to create templates:
% Fail to Create : 84.743%

Reason                                  Count      Reason Fail To Create %

NON TCP/UDP PROTO                       |4377809   |1.202     %
Conn Not Accelerated                    |7985235   |2.193     %
NAT Disallowed Conn                     |23091058  |6.341     %
General Error                           |870977    |0.239     %
Malicious Destination IP Detected       |249686    |0.069     %
Prevented By Policy Rules               |272030517 |74.699    %
-------------------

I've located my inline layers in Access Control that have the Applications & URL Filtering blade enabled. I've now unchecked the URL Filtering option, and I will update you with the results shortly

 

0 Kudos
Moudar
Advisor

Prevented by policy rules is going down very slowly, so 5% in 3 days! after removing URL blade from some inline rules we have.

fwaccel templates -R

Matched connections not allowed to use templates:
% Prevention : 1.239%

Reason                                  Count      Reason Prevented From Matched %

Non-Syn/Empty First Packet              |181618    |0.712     %
Src/dst IP Blacklisted                  |134396    |0.527     %
Dynamic VPN Connection                  |2         |0.000     %
--------------------

Connections failed to create templates:
% Fail to Create : 82.093%

Reason                                  Count      Reason Fail To Create %

NON TCP/UDP PROTO                       |4524945   |1.159     %
Conn Not Accelerated                    |8558463   |2.193     %
NAT Disallowed Conn                     |34132098  |8.744     %
DHCP Check Feature Isn't Supported Or Disabled|10        |0.000     %
General Error                           |921701    |0.236     %
Malicious Destination IP Detected       |259938    |0.067     %
Prevented By Policy Rules               |272047377 |69.694    %

But Accelerated conns/Total conns : 556/61263 (0%) still shows 0%!

Maybe I need to wait a week or two to see some result?

fwaccel stats -s
Accelerated conns/Total conns    : 556/61263 (0%)
LightSpeed conns/Total conns     : 0/61263 (0%)
Accelerated pkts/Total pkts      : 73234561042/82749335447 (88%)
LightSpeed pkts/Total pkts       : 0/82749335447 (0%)
F2Fed pkts/Total pkts            : 9514774405/82749335447 (11%)
F2V pkts/Total pkts              : 395727119/82749335447 (0%)
CPASXL pkts/Total pkts           : 3041455470/82749335447 (3%)
PSLXL pkts/Total pkts            : 67212887275/82749335447 (81%)
CPAS pipeline pkts/Total pkts    : 0/82749335447 (0%)
PSL pipeline pkts/Total pkts     : 0/82749335447 (0%)
QOS inbound pkts/Total pkts      : 0/82749335447 (0%)
QOS outbound pkts/Total pkts     : 0/82749335447 (0%)
Corrected pkts/Total pkts        : 0/82749335447 (0%)

 

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events