Hiccups and general trouble with Videoconferences (Teams,Google Meet, etc.)

[fps]

I've started working in a productive environment and the users have started complaining of severe hiccups in video in audio in their videoconferences since August. I've tried diagnosing the problem using different tests, different networks.. and almost every evidence is pointing to the firewall. I've tried watching any dropped packages with fw ctl debug and watching the traffic with fw monitor but to no avail. I've looked extensively at the forums but found no suitable solution, i know this is a vague problem , but i post in case someone has been through or knows a plan of action i could be using to try to clear the problem

Thanks in advance.

SPECS:

I have 2 checkpoint firewalls in cluster active passive with the following specs:

fw01-cbra> cpinfo -y all

This is Check Point CPinfo Build 914000176 for GAIA
[IDA]
HOTFIX_R80_10

[KAV]
HOTFIX_R80_10

[CPFC]
HOTFIX_R80_10

[FW1]
HOTFIX_R80_10

FW1 build number:
This is Check Point's software version R80.10 - Build 423
kernel: R80.10 - Build 431

[SecurePlatform]
No hotfixes..

[PPACK]
HOTFIX_R80_10

[CPinfo]
No hotfixes..

[CVPN]
HOTFIX_R80_10

[DIAG]
No hotfixes..

[rtm]
No hotfixes..

[_Val_]

To start with the basics, this environment has been out of support for a while now. R80.10 is an old version.

Did you look into any performance issues with the FW when conference apps are affected?

[fps]

I did not see any performance issues, in fact the firewall is underused even with what we consider ''heavy load'' (Everybody in the environment working at the office)

[Timothy_Hall]

For a version that old, you almost certainly are having performance problems and need to make some manual tuning adjustments.  Please post the results of command enabled_blades and the Super Seven outputs:

S7PAC - Super Seven Performance Assessment Commands

[fps]

Sorry for the wait, ill add the info. At the moment users are not experiencing any issues, ill try to run the tests again and add them when they are having trouble

[Expert@fw01-cbra:0]# enabled_blades
fw av ips anti_bot

[Timothy_Hall]

Agree with Val.  The firewall looks fine from a load perspective, at least when you ran s7pac.  I doubt it is a memory issue but please post the output of free -m.

The firewall does have only 2 cores however, and therefore both cores are pulling double duty as both a SND and Worker core in a 2/2 split.  It wouldn't take much traffic to overload the CPUs even though almost all of the traffic is fully accelerated by SecureXL.  If you can identify a particular day within the last 30 days when you had the most connection issues (let's say for example it was the 14th of the month which is used in the following command), please post the output of sar  -f  /var/log/sa/sa14  -P  ALL

But I doubt these two commands will reveal any firewall resource issues, and you'll have to look elsewhere.  Could be some kind of Check Point code issue in such an old release I suppose, and you need to upgrade.

[fps]

Thank you for your time, The users are having issues right now, but i dont detect any overloads with the free or sar commands. Ill add the results of the s7pac now that were having trouble just in case you are able to get something i dont.

The only issue ive found is in the:

+-----------------------------------------------------------------------------+
| Command #2: fwaccel stats -s |
| |
| Check for : Accelerated conns/Totals conns: >25% good, >50% great |
| Accelerated pkts/Total pkts : >50% great |
| PXL pkts/Total pkts : >50% OK |
| F2Fed pkts/Total pkts : <30% good, <10% great |
| |
| Chapter 9: SecureXL throughput acceleration |
| Page 287, Packet/Throughput Acceleration: The Three Kernel Paths |
+-----------------------------------------------------------------------------+
| Output: |
Accelerated conns/Total conns : 689/1083 (63%)
Accelerated pkts/Total pkts : 8209410/8989786 (91%)
F2Fed pkts/Total pkts : 762744/8989786 (8%)
PXL pkts/Total pkts : 17632/8989786 (0%)
QXL pkts/Total pkts : 0/8989786 (0%)

It should be more than 50% but its almost 0, could this be the issue?

[Timothy_Hall · Accepted solution]

No that low PXL value indicating partial acceleration is just fine, since the majority of traffic is in the much more efficient fully-accelerated path (Accelerated pkts).  Based on the limited blades you have enabled a low PXL is expected.  You aren't having any resource issues at all in the latest s7pac but it may yet be some kind of code issue.

I did think of one other thing to try: SecureXL was completely overhauled in R80.20 yet you are still running R80.10; it is possible that acceleration is interfering with your connections.  Try disabling SecureXL for awhile and see if things improve by running the fwaccel off command.  This command will not survive a reboot but you can turn SecureXL permanently off in that release via  cpconfig.  Note that doing so will cause much higher CPU load (although you appear to have plenty of headroom) and for all traffic to end up in the F2F path, so you may want to closely monitor CPU load after disabling SecureXL to ensure the firewall does not get overloaded.  If this fixes the problem the best solution will be a code upgrade to at least R81.10.

[Timothy_Hall]

So did disabling SecureXL under R80.10 solve the problem?  I see you accepted my post as a solution.