R80.10 - VPND process utilizes a large amount of CPU

[Laxi_D]

Recently, Checkpoint 4800 R77.30 cluster (4GB RAM - OS is 32bit)  was  upgraded  to R80.10 and after few days we observes VPND process consume large amount of CPU. Top commands shows VPND consume more than 100% CPU.

The cluster has only one site - site VPN and 2-3 concurrent remote access vpn tunnels

Already open TAC case but still no solution found.

[Jeroen_Roegiers]

We are experiencing the same problem on our 4600 cluster. Since the switchover to 64 bit, the VPND proces has gone wild...

[Timothy_Hall]

Any interesting messages getting dumped in $FWDIR/log/vpnd.elg?

For site to site VPNs VPND handles IKE negotiations outside the kernel, are there constantly failing IKE negotiations happening over and over?

The rest of VPND's functions involve handling various special Remote Access VPN situations, such as:

• NAT-T
• Tunnel Test
• Reliable Datagram Protocol (RDP)
• Topology Update for SecureClient
• SSL Network Extender (SNX)
• SSL Network Extender (SNX) Portal
• Remote Access Client configuration
• Visitor Mode
• L2TP

Are any of these extensively used by your clients?  If you disable all remote access temporarily, does the high CPU usage by VPND stop?

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Laxi_D]

Hi Tim,

Thanks for reply

last 20 lines of $FWDIR/log/vpnd.elg file

4416]@CPFW02[12 Sep 21:04:42] ------------ VPND Starting: Wed Sep 12 21:04:42 2018

[vpnd 5481 2012694416]@CPFW02[12 Sep 21:04:42] VPN-1 daemon: stopping debug - Wed Sep 12 21:04:42 2018

vpn_update_isakmp_user: trap was already called for this ip: 2.50.12.10

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] vpnd: Wed Sep 12 22:28:16 2018

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] ------------ VPND Starting: Wed Sep 12 22:28:16 2018

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] VPN-1 daemon: stopping debug - Wed Sep 12 22:28:16 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] vpnd: Wed Sep 12 22:30:04 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] ------------ VPND Starting: Wed Sep 12 22:30:04 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] VPN-1 daemon: stopping debug - Wed Sep 12 22:30:04 2018

[vpnd 31168]@CPFW02[12 Sep 22:30:11] Warning:cp_timed_blocker_handler: A handler [0xbebbe0] blocked for 5 seconds.
[vpnd 31168]@CPFW02[12 Sep 22:30:11] Warning:cp_timed_blocker_handler: Handler info: Library [/opt/CPshrd-R80/lib/libfwdclient.so], Function offset [0x1be0].

Found two core dump files for vpnd process

-rw-r--r-- 1 admin root 35062942 Sep 12 22:30 vpnd.25525.core.gz
-rw-r--r-- 1 admin root 40017513 Sep 12 22:28 vpnd.5481.core.gz

entries related to VPND process in /var/log/messages file

Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: corename = |/etc/coredump/compress.sh /var/log/dump/usermode/vpnd.5481.core
Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: argv_arr[0] = /etc/coredump/compress.sh
Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: argv_arr[1] = /var/log/dump/usermode/vpnd.5481.core
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_0];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_0];VPN-1: AES-NI is not supported on this hardware
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_1];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_1];VPN-1: AES-NI is not supported on this hardware
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_2];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_2];VPN-1: AES-NI is not supported on this hardware

are there constantly failing IKE negotiations happening over and over? no, tunnel is stable and its ikev2 tunnel with Azure cloud gateway 

[Timothy_Hall]

Given the timestamps on the vpnd core dumps, and the fact that a maximum of 2 core dumps per process are allowed to be retained by default, it looks like vpnd is constantly crashing.  TAC should be able to analyze those core dumps and figure out what is going on. 

Also I assume you have the latest R80.10 GA Jumbo HFA loaded?   Always a good idea particularly when stability issues are encountered.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Laxi_D]

 I disabled IKV2 tunnel then CPU consumption of VPND process is reduced( Now only 3%). Remote access VPN is still enabled.

Tunnel parameters

Phase 1

AES-256, SHA-1 and Group 2 

Life time - 480 Minutes

Phase 2

AES-256, SHA-1

Life time - 3600 seconds 

No PFS

NAT-T is enabled

How to debug issues like these?

Thanks

[Timothy_Hall]

Interesting find, sounds like these issues:

sk116776: Instability issues in VPN Tunnel with Cisco using IKEv2

sk106222: VPN tunnels S2S and C2S are unstable due to VPND crashes every 20-30 seconds

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Laxi_D]

yes its latest GA 121 installed

[Danny]

Could you please show me a screenshot of the main menu of our ccc script running? This would provide a good overview to your system to me (To see if RAM is already swapping, if SexureXL is enabled etc.).

[Laxi_D]

This was taken at off-peak time

[Danny]

With all these blades enabled on R80.10 and only 4GB RAM available, and with such a high load during off-peak times I strongly suggest planning an appliance upgrade (trade-in) to a 5900 appliance with activated SMT and 64-bit.

Alternatively you could try if disabling IPS is a game changer here.