Hi all,
I have a strange issue with VPN clients disconnecting around 11:00am and 3pm each day that has only started happening since the start of January. We have over 3000 users connect via VPN .
We have a pair of 16200's running in Active-Standby. They were rebuilt back in November by professional services but immediately had issues with the SND's not balancing (a single SND would max out) this was resolved with sk165853 and through December we had no issues with performance.
From the 3rd Jan (1st day back after Christmas) at 11am, we had loads of reports that users were disconnected from VPN, at this point we failed over the firewall to the standby and it seemed to resolve the issue. The same thing happened the next day and has been happing each day since. Sometimes at 3pm too.
Looking through the logs, I have noticed a few things. At the time of disconnections we get CPU spikes, the spikes seem to happen every 3 hours. Some of the spikes are a couple of minutes but there are a couple that have been running for weeks. Below are a couple from today, both starting before Christmas. The week or so before Christmas a lot of staff break up, therefore this issue could have started before Jan. Both firewalls have been rebooted in the last 10 days.
Jan 25 09:28:02 2022 fwxxxxx spike_detective: spike info: type: thread, thread id: 81691, thread name: fwk0_32, start time: 22/01/22 04:18:12, spike duration (sec): 277789, initial cpu usage: 100, average cpu usage: 97, perf taken: 0
Jan 25 09:32:25 2022 fwxxxxx spike_detective: spike info: type: thread, thread id: 81672, thread name: fwk0_13, start time: 21/01/22 17:00:53, spike duration (sec): 318691, initial cpu usage: 100, average cpu usage: 97, perf taken: 0
Another anomaly we notice around the time of the disconnections is a massive increase of 'ESP enc Errors' in CPView under VPN > SecureXL. Normally around the 10 - 100 mark, at the time of the CPU spike and disconnections they jump up to 5000+ for a couple of minutes.
Any thoughts or help?
Many thanks
Rich