Pushing policy destroys Skype calls

[Tommy_Forrest]

Does anyone else have issues where when they push policy to their internet edge gateway Skype calls are utterly destroyed for a solid 30-90 seconds?

We have a 3 node cluster in HA mode running on 15600 gateways with 80.10 (our 80.30 migration starts in December).  CPUs average around 30% at peak during the day.

Connection Persistence is configured for "Keep all connections".

It does not matter the time of day (or load) when policy is pushed.  We can push it at 4am and it will disrupt Skype calls.

What is the solution for this?  Aside from only pushing policy after hours (which will be an enormous burden to my team).

[G_W_Albrecht]

I would involve TAC here - they at least can explain why policy install will disturb VoIP calls even if they can not resolve it...

[Danny]

What are the firewall and system logs showing regarding this issue? Is QoS and LLQ configured for this traffic? What is policy installation debugging showing? What is fw ctl zdebug drop showing? How is the specific service configured? What blades are inspecting this traffic? Does failover occur during policy installation? Keep in mind R80.10 is pushing the entire policy, not just the delta as R80.20 or .30

[Tommy_Forrest]

...firewall and system logs showing regarding this issue?  What logs would I be looking for specifically?  Not sure I'd know what to be looking for in the logs.

QOS is not enabled on this gateway.

How is the specific service configured?   Which specific service and configured how?

What blades are inspecting this traffic? IPS/AV/Anti-Bot are enabled.  But don't cause issues outside of policy pushing.  No other traffic is impacted during policy push.

Does failover occur during policy installation?  No, the cluster is stable.

Keep in mind R80.10 is pushing the entire policy, not just the delta as R80.20 or .30 - I am aware of this.  And my understanding is that full delta pushing isn't due until 80.40.  Not sure if that has changed or not.  Our MDSen go 80.30 on 12/7.  Gateway upgrades will start in January.

[Timothy_Hall]

As Dameon said, on R80.10 and earlier SecureXL is completely restarted every time policy is pushed to it can re-sync it own connections table with the rematched one in the Firewall Workers.  This can most definitely cause latency, and in some cases packet loss if the firewall is overloaded.

In R80.20 and later SecureXL continues running during a policy push and rematches its own connections table with the Firewall Workers via the F2V path.

This may sound a bit counter-intuitive, but pick some IP addresses doing Skype calls and force them F2F permanently as detailed here: sk104468: How to disable SecureXL for specific IP addresses.  See if doing so helps for those particular IPs when policy pushes are performed.

[Nick_Doropoulos]

Hi Tommy,

Considering that the policy installation is known to consume CPU resources, I would start by checking that the firewall's resources are at normal levels to rule out the likelihood of the firewall finding itself under a high strain during policy push. Some of the typical commands would be the following:

- top
- free -m
- vmstat

Then, a few questions for you if you don't mind:

- How is the policy installed? Is it just access control or threat prevention as well? Also, is it pushed to all selected gateways or to each member individually?

- What about installing different policy packages? Does the problem appear again?

- Which ports have been configured on the firewall to allow skype?

- Have you tried to install policy from the firewall with fw fetch? If so, what was the result?

- Does the issue in question occur independently of which cluster member is in active state? In other words, if you carry out a fail over and push policy would the result be the same?

Thanks.

[PhoneBoy]

One thing that happens during policy installation is everything goes F2F while SecureXL reloads after a policy installation.
This process is much more efficient in R80.20+.
It may be what's responsible for the issue you're seeing.

[Jerry]

as far as I'm aware Skype uses UDP ports so I guess policy push temporarily destroy those sessions therefore it's "reconnect" seem required and normal, not sure I understand correctly but have you tried fw monitor that traffic noticing any drops when "fetch" over the sic happens? I guess it isn't related to 80.10 or 80.30 but Skype especially skype-for-business or nowadays Teams uses udp/5060 (sip) or soft-sip for the calls (voice calls) so I presume this interruption is either caused by wrong secureXL (as Dameon mentioned F2F) associations or simply not tuned well by the fw accell . wonder what really is happening on the logs as well when the policy is pushed (what is happening with the calls and its packages)/.