cancel
Showing results for 
Search instead for 
Did you mean: 
Post a Question
Laxi_D
Nickel

R80.10 - VPND process utilizes a large amount of CPU

Recently, Checkpoint 4800 R77.30 cluster (4GB RAM - OS is 32bit)  was  upgraded  to R80.10 and after few days we observes VPND process consume large amount of CPU. Top commands shows VPND consume more than 100% CPU.

The cluster has only one site - site VPN and 2-3 concurrent remote access vpn tunnels

Already open TAC case but still no solution found.

10 Replies

Re: R80.10 - VPND process utilizes a large amount of CPU

We are experiencing the same problem on our 4600 cluster. Since the switchover to 64 bit, the VPND proces has gone wild...

0 Kudos

Re: R80.10 - VPND process utilizes a large amount of CPU

Any interesting messages getting dumped in $FWDIR/log/vpnd.elg?

For site to site VPNs VPND handles IKE negotiations outside the kernel, are there constantly failing IKE negotiations happening over and over?

The rest of VPND's functions involve handling various special Remote Access VPN situations, such as:

  • NAT-T
  • Tunnel Test
  • Reliable Datagram Protocol (RDP)
  • Topology Update for SecureClient
  • SSL Network Extender (SNX)
  • SSL Network Extender (SNX) Portal
  • Remote Access Client configuration
  • Visitor Mode
  • L2TP

Are any of these extensively used by your clients?  If you disable all remote access temporarily, does the high CPU usage by VPND stop?

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
Laxi_D
Nickel

Re: R80.10 - VPND process utilizes a large amount of CPU

Hi Tim,

Thanks for reply

last 20 lines of $FWDIR/log/vpnd.elg file

4416]@CPFW02[12 Sep 21:04:42] ------------ VPND Starting: Wed Sep 12 21:04:42 2018

[vpnd 5481 2012694416]@CPFW02[12 Sep 21:04:42] VPN-1 daemon: stopping debug - Wed Sep 12 21:04:42 2018

vpn_update_isakmp_user: trap was already called for this ip: 2.50.12.10

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] vpnd: Wed Sep 12 22:28:16 2018

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] ------------ VPND Starting: Wed Sep 12 22:28:16 2018

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] VPN-1 daemon: stopping debug - Wed Sep 12 22:28:16 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] vpnd: Wed Sep 12 22:30:04 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] ------------ VPND Starting: Wed Sep 12 22:30:04 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] VPN-1 daemon: stopping debug - Wed Sep 12 22:30:04 2018

[vpnd 31168]@CPFW02[12 Sep 22:30:11] Warning:cp_timed_blocker_handler: A handler [0xbebbe0] blocked for 5 seconds.
[vpnd 31168]@CPFW02[12 Sep 22:30:11] Warning:cp_timed_blocker_handler: Handler info: Library [/opt/CPshrd-R80/lib/libfwdclient.so], Function offset [0x1be0].

Found two core dump files for vpnd process

-rw-r--r-- 1 admin root 35062942 Sep 12 22:30 vpnd.25525.core.gz
-rw-r--r-- 1 admin root 40017513 Sep 12 22:28 vpnd.5481.core.gz

entries related to VPND process in /var/log/messages file

Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: corename = |/etc/coredump/compress.sh /var/log/dump/usermode/vpnd.5481.core
Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: argv_arr[0] = /etc/coredump/compress.sh
Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: argv_arr[1] = /var/log/dump/usermode/vpnd.5481.core
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_0];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_0];VPN-1: AES-NI is not supported on this hardware
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_1];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_1];VPN-1: AES-NI is not supported on this hardware
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_2];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_2];VPN-1: AES-NI is not supported on this hardware

are there constantly failing IKE negotiations happening over and over? no, tunnel is stable and its ikev2 tunnel with Azure cloud gateway 

0 Kudos

Re: R80.10 - VPND process utilizes a large amount of CPU

Given the timestamps on the vpnd core dumps, and the fact that a maximum of 2 core dumps per process are allowed to be retained by default, it looks like vpnd is constantly crashing.  TAC should be able to analyze those core dumps and figure out what is going on. 

Also I assume you have the latest R80.10 GA Jumbo HFA loaded?   Always a good idea particularly when stability issues are encountered.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
0 Kudos
Laxi_D
Nickel

Re: R80.10 - VPND process utilizes a large amount of CPU

 I disabled IKV2 tunnel then CPU consumption of VPND process is reduced( Now only 3%). Remote access VPN is still enabled.

Tunnel parameters

Phase 1

AES-256, SHA-1 and Group 2 

Life time - 480 Minutes

Phase 2

AES-256, SHA-1

Life time - 3600 seconds 

No PFS

NAT-T is enabled

How to debug issues like these?

Thanks

0 Kudos

Re: R80.10 - VPND process utilizes a large amount of CPU

Interesting find, sounds like these issues:

sk116776: Instability issues in VPN Tunnel with Cisco using IKEv2

sk106222: VPN tunnels S2S and C2S are unstable due to VPND crashes every 20-30 seconds

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
0 Kudos
Laxi_D
Nickel

Re: R80.10 - VPND process utilizes a large amount of CPU

yes its latest GA 121 installed

0 Kudos
Danny
Pearl

Re: R80.10 - VPND process utilizes a large amount of CPU

Could you please show me a screenshot of the main menu of our ccc script running? This would provide a good overview to your system to me (To see if RAM is already swapping, if SexureXL is enabled etc.).

0 Kudos
Laxi_D
Nickel

Re: R80.10 - VPND process utilizes a large amount of CPU

This was taken at off-peak time

0 Kudos
Danny
Pearl

Re: R80.10 - VPND process utilizes a large amount of CPU

With all these blades enabled on R80.10 and only 4GB RAM available, and with such a high load during off-peak times I strongly suggest planning an appliance upgrade (trade-in) to a 5900 appliance with activated SMT and 64-bit.

Alternatively you could try if disabling IPS is a game changer here.

0 Kudos