Poor MS Teams quality, what can we do?

[Thomas_Eichelbu]

Hello Check Mates, 

a weird issueiam encountering more and more often these days, costumers are complaining about poor MS Teams quality or other poor VoIP applications performance.

for example MS Teams.

with this tool:
https://docs.microsoft.com/de-de/microsoftteams/dimensions-and-measures-available-in-call-quality-da...

and a powerful Excel PowerBI it is possible to do extensive research on your local MS Teams calls. i dont want to go into the details of this tool but it creates a visual output like this. Here are all MS Teams for one tenant within a specific subnet.

at the beginning we focused on the Jitter / Latency part.
Of course the firewall which has to do packetinspection / AV, ABot all that stuff, this will most propably cause delay.
so i created an empty IPS Profile to match the MS Teams connection and additionally some FAST ACCEL rules for the same IP connections. this happend at 22-05-03, after that u can see a decline in the Jitter/Latency! A good start.
Also people said it had helped so far.

but still we have to deal with packetloss. As you can see in the grafic packetoss is way to high!

Question:
any ideas to improve it any further?
sometimes the workaround with empty IPS and FAST ACCEL dosnt work for some customers, still the user experience is bad.

We see:
+ no high cpu
+ no outtages during policy installs
+ no overutilization of the internet line
+ no drops on the FW regarding MS Teams or Cisco Jabber

for Cisco Jabber the quality over client VPN is especially bad, people moved back to use mobile phones in the homeoffice instead of the softphone applications.

perhaps you have also encountering the same ...
best regards
Thomas

[Kaspars_Zibarts]

Do you do HTTPS interception on O365 traffic? We bypass O365 using updatable object and all seems working fine

[Thomas_Eichelbu]

Hello, 
well yes of course, we followed all hints Microsoft gave us to exclude O365 from everything.

• Bypass Optimize endpoints on network devices and services that perform traffic interception, SSL decryption, deep packet inspection, and content filtering.
• Bypass on-premises proxy devices and cloud-based proxy services commonly used for generic Internet browsing.
• Prioritize the evaluation of these endpoints as fully trusted by your network infrastructure and perimeter systems.
• Prioritize reduction or elimination of WAN backhauling, and facilitate direct distributed Internet-based egress for these endpoints as close to users/branch locations as possible.
• Facilitate direct connectivity to these cloud endpoints for VPN users by implementing split tunneling.
• Ensure that IP addresses returned by DNS name resolution match the routing egress path for these endpoints.
• Prioritize these endpoints for SD-WAN integration for direct, minimal latency routing into the nearest Internet peering point of the Microsoft global network.

Some sites do run HTTPS inspection, others don´t. The overall user experience is the same, not a good one.
Its the same in the HQ with many many network devices and also on smaller remote sites of only a handfull of network infrstructure.

when users are sitting at homeoffice, MS Teams is "always good" but it the office its never as good as at home!
also we asked with users have lame or slow clients or if they share some high resolutions applications etc ... 

the firewalls are mostly relaxed ... 
fwaccel stats are all good 
no severe NIC errors

so still a weird thing.

[Sorin_Gogean]

@Kaspars_Zibarts , does Teams Voice/Sharing uses HTTPS ? As I've seen it does go on some higher ports....

"For Teams to function correctly, you must open TCP ports 80 and 443 from the clients to the internet, and UDP ports 3478 through 3481 from the clients to the internet. The TCP ports are used to connect to web-based content such as SharePoint Online, Exchange Online, and the Teams Chat services."

[Kaspars_Zibarts]

Correct, I just wanted to make sure  that general MS guidlines are followed when it comes to O365. And they are 🙂

[Sorin_Gogean]

Just adding here, 

Our users complain also for Teams issues, but what we noticed is that it happens in meetings with over 50 - 70 users, where a big part are sitting in the same office. (and this happens in several countries around the globe)

Voice was never the problem, screen/file sharing was pointed not being OK.

Still our traffic doesn't pass over CheckPoint GW, so I can't pinpoint that CKP GW are the problem.

[Thomas_Eichelbu]

Hello, 

Yes thats also a big point!
some internals: at some degree the customer makes calls with severall hundreds participants (yes hundreds, no joke!)
its often done by a Cisco room solution via a Webex/Teams integration, yes it has some bottlenecks we are aware off.
but thats the harcore example!
other customers, much smaller customers have small meetings with a couple of persons, even they are not good in quality, delayed audio/video.

of course smaller costumers have small firewalls. but still.
at home its super cool.
in the office it is contantly getting worse.

[Sorin_Gogean]

I see @Thomas_Eichelbu , so you have the GW as the default route for your traffic. 

Do you use QOS? could it be that from the office, traffic is not prioritized in any way up to the CKP GW exiting Internet ?

(as I don't know your design, I can't say much more)

In our case, the majority of complains were related to big meetings (like you said) and all others were OK.
Still what I can say, is that in our case, where we have over 20 (or more) users from same place, they complain. We have a meeting happening every Friday with over 150 users from around Europe, not many sharing same office/place, and all is good there. But others where we have multiple sharing the office, come back with Complains . 

And like I said, in our design, Teams traffic doesn't pass through CheckPoint GW, so we're suspecting MS doing some stuff....

Have you tried also to look over the codecs used for Voice/Sharing ?

Thank you,

[Thomas_Eichelbu]

@Sorin_Gogean 
well the discussion i have started is more in generell and affects more then one costumer, and each costumer is completly different in size and hardware models/software.

but QoS is never turned on.
the only thing we did at the larger customer was to double the size of the NIC ringbuffers of the LAN uplink because of NIC drivers crashes. perhaps larger buffers cause more delay on MS Teams traffic ...

So when you say your MS Teams traffic doesnt passthrough a Check Point and is also not perfect ... iam also happy with this answer.
Besides MS Teams we still fight with Cisco Jabber softphones which has also bad quality over Client VPN. 
Its the question if exluding the Cisco Jabber communiction from VPN might help there.

The Codec in MS Teams is always the new "SATIN" codec ...

[Wolfgang]

@Thomas_Eichelbu we had a similar issue with Teams-conferences and a R80.40 gateway. No IPS, no HTTPS-Inspection , no proxy... Teams calls broken, no initiation possible, sometimes video broken, screen-sharing broken .....

After some debugs with TAC we changed the following kernel parameter "psl_max_future_segments"  from 4096 to 16384.

"fw ctl set int psl_max_future_segments 16384" Everything looks fine since the change.

Maybee Check Point can share a best practice document for O365/Teams traffic .

[Kaspars_Zibarts]

I can only say that it works for us without any special tweaks. We are talking 1000+ users behind this particular cluster. 🙂 Running R80.40 T139

Large meetings must be organised in special way, else you will end up connecting to the nearest Teams front door of the first person joining the meeting. For example - if you have a 100 user meeting and 99 are in Europe and one in Brazil. If it happens so that Brazilian person joins the meeting first, it will start the meeting in SaoPaulo MS datacentre and remaining 99 people from Europe will be forced to connect to that one too causing extra latency.

Your recursive DNS is extremely important as it will dictate which front door will be chosen for your users. Make sure that users use correct recursive DNS (i.e. centralised DNS solution located geographically far away from users may cause issues)

All depends on the geography and your DNS, WAN / local breakout setup. We had a lot of issues approx 2 years ago, especially in South Latin America as MS had too few datacentres and our DNS was not positioned in the best possible way.

Surely you know to use https://connectivity.office.com/ for quick checks

[Sorin_Gogean]

@Kaspars_Zibarts , can you explain a bit more "Large meetings must be organised in special way," ?

I'll also go to our Teams Responsible Team and see what they say about it .

Ty,

PS: in our case, they are all on same continent.

[Kaspars_Zibarts]

I think for us going to teams live event made massive difference. You may start reading up here and see the best option for you 

https://support.microsoft.com/en-us/office/best-practices-for-a-large-teams-meeting-ce2cdb9a-0546-43... 

But I concur - Teams can be challenging to troubleshoot. You really need to dig into it. We did a lot of packet capture actually to start addressing and discovering or problems. And i doubt that there's a universal solution for all as setups differ a lot.