topic Re: R80.10 - VPND process utilizes a large amount of CPU in General Topics

R80.10 - VPND process utilizes a large amount of CPU

Laxi_D — Tue, 11 Sep 2018 05:33:55 GMT

Recently, Checkpoint 4800 R77.30 cluster (4GB RAM - OS is 32bit) was upgraded to R80.10 and after few days we observes VPND process consume large amount of CPU. Top commands shows VPND consume more than 100% CPU.

The cluster has only one site - site VPN and 2-3 concurrent remote access vpn tunnels

Already open TAC case but still no solution found.

Re: R80.10 - VPND process utilizes a large amount of CPU

Jeroen_Roegiers — Tue, 11 Sep 2018 07:55:52 GMT

We are experiencing the same problem on our 4600 cluster. Since the switchover to 64 bit, the VPND proces has gone wild...

Re: R80.10 - VPND process utilizes a large amount of CPU

Timothy_Hall — Tue, 11 Sep 2018 23:00:34 GMT

Any interesting messages getting dumped in $FWDIR/log/vpnd.elg?

For site to site VPNs VPND handles IKE negotiations outside the kernel, are there constantly failing IKE negotiations happening over and over?

The rest of VPND's functions involve handling various special Remote Access VPN situations, such as:

NAT-T
Tunnel Test
Reliable Datagram Protocol (RDP)
Topology Update for SecureClient
SSL Network Extender (SNX)
SSL Network Extender (SNX) Portal
Remote Access Client configuration
Visitor Mode
L2TP

Are any of these extensively used by your clients? If you disable all remote access temporarily, does the high CPU usage by VPND stop?

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: R80.10 - VPND process utilizes a large amount of CPU

Laxi_D — Wed, 12 Sep 2018 19:03:08 GMT

Hi Tim,

Thanks for reply

last 20 lines of $FWDIR/log/vpnd.elg file

4416]@CPFW02[12 Sep 21:04:42] ------------ VPND Starting: Wed Sep 12 21:04:42 2018

[vpnd 5481 2012694416]@CPFW02[12 Sep 21:04:42] VPN-1 daemon: stopping debug - Wed Sep 12 21:04:42 2018

vpn_update_isakmp_user: trap was already called for this ip: 2.50.12.10

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] vpnd: Wed Sep 12 22:28:16 2018

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] ------------ VPND Starting: Wed Sep 12 22:28:16 2018

[vpnd 25525 2012862352]@CPFW02[12 Sep 22:28:16] VPN-1 daemon: stopping debug - Wed Sep 12 22:28:16 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] vpnd: Wed Sep 12 22:30:04 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] ------------ VPND Starting: Wed Sep 12 22:30:04 2018

[vpnd 31168 2012084112]@CPFW02[12 Sep 22:30:04] VPN-1 daemon: stopping debug - Wed Sep 12 22:30:04 2018

[vpnd 31168]@CPFW02[12 Sep 22:30:11] Warning:cp_timed_blocker_handler: A handler [0xbebbe0] blocked for 5 seconds.
[vpnd 31168]@CPFW02[12 Sep 22:30:11] Warning:cp_timed_blocker_handler: Handler info: Library [/opt/CPshrd-R80/lib/libfwdclient.so], Function offset [0x1be0].

Found two core dump files for vpnd process

-rw-r--r-- 1 admin root 35062942 Sep 12 22:30 vpnd.25525.core.gz
-rw-r--r-- 1 admin root 40017513 Sep 12 22:28 vpnd.5481.core.gz

entries related to VPND process in /var/log/messages file

Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: corename = |/etc/coredump/compress.sh /var/log/dump/usermode/vpnd.5481.core
Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: argv_arr[0] = /etc/coredump/compress.sh
Sep 12 22:28:11 2018 CPFW02 kernel: do_coredump: argv_arr[1] = /var/log/dump/usermode/vpnd.5481.core
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_0];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_0];VPN-1: AES-NI is not supported on this hardware
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_1];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_1];VPN-1: AES-NI is not supported on this hardware
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_2];VPN-1: AES-NI is allowed on this machine. Testing hardware support
Sep 12 22:28:16 2018 CPFW02 kernel: [fw4_2];VPN-1: AES-NI is not supported on this hardware

are there constantly failing IKE negotiations happening over and over? no, tunnel is stable and its ikev2 tunnel with Azure cloud gateway

Re: R80.10 - VPND process utilizes a large amount of CPU

Timothy_Hall — Wed, 12 Sep 2018 20:08:22 GMT

Given the timestamps on the vpnd core dumps, and the fact that a maximum of 2 core dumps per process are allowed to be retained by default, it looks like vpnd is constantly crashing. TAC should be able to analyze those core dumps and figure out what is going on.

Also I assume you have the latest R80.10 GA Jumbo HFA loaded? Always a good idea particularly when stability issues are encountered.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: R80.10 - VPND process utilizes a large amount of CPU

Laxi_D — Thu, 13 Sep 2018 12:10:54 GMT

yes its latest GA 121 installed

Re: R80.10 - VPND process utilizes a large amount of CPU

Danny — Thu, 13 Sep 2018 13:47:09 GMT

Could you please show me a screenshot of the main menu of our ccc script running? This would provide a good overview to your system to me (To see if RAM is already swapping, if SexureXL is enabled etc.).

Re: R80.10 - VPND process utilizes a large amount of CPU

Laxi_D — Fri, 14 Sep 2018 00:51:17 GMT

This was taken at off-peak time

Re: R80.10 - VPND process utilizes a large amount of CPU

Danny — Fri, 14 Sep 2018 23:57:31 GMT

With all these blades enabled on R80.10 and only 4GB RAM available, and with such a high load during off-peak times I strongly suggest planning an appliance upgrade (trade-in) to a 5900 appliance with activated SMT and 64-bit.

Alternatively you could try if disabling IPS is a game changer here.

Re: R80.10 - VPND process utilizes a large amount of CPU

Laxi_D — Thu, 20 Sep 2018 15:17:55 GMT

I disabled IKV2 tunnel then CPU consumption of VPND process is reduced( Now only 3%). Remote access VPN is still enabled.

Tunnel parameters

Phase 1

AES-256, SHA-1 and Group 2

Life time - 480 Minutes

Phase 2

AES-256, SHA-1

Life time - 3600 seconds

No PFS

NAT-T is enabled

How to debug issues like these?

Thanks

Re: R80.10 - VPND process utilizes a large amount of CPU

Timothy_Hall — Thu, 20 Sep 2018 16:02:23 GMT

Interesting find, sounds like these issues:

sk116776: Instability issues in VPN Tunnel with Cisco using IKEv2

sk106222: VPN tunnels S2S and C2S are unstable due to VPND crashes every 20-30 seconds

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com