Saving excel files slow on file shares last 6 months impacting multiple users

[hirogen10]

we have users who access network file shares the usual way over DFS and have tried directly to rule out the issue being DFS.

see attached they sometimes see "connecting to Ad.mydomain.net and the file server path

All users are on windows 10 22H2 - Office semi annual O365 (deployed through intune azureAD joined but no longer on the domain). So checkpoint endpoint vpn  version 86.80 now is always running for every user otherwise can't access anything obviously.

 We've expanded the timeout  over 12 hours for the checkpoint this has also possibly caused a side effect where some users have DNS problems and have to do clear browser cache and flushDNS cmdline in order to access citrix workspace apps but that's a separate issue (stale vpn session as they don't reboot their desktops until the weekend for patching), 

And I don't know if it's linked to that as we moved to intune built devices the last 6 months or so from domain joined to reduce teh attack vectors primary the reason for the desktops and laptops being on a workgroup  and when users try to save files and its intermittent, they get slowness and some errors sometimes if its a complex linked macros The issue is now escalated and I am not sure if it's a netapp issue or because we're reliant on checkpoint running all the time to access our domain based services, we're currently migrating file shares to sharepoint online so you would think the load on the netapp servers has reduced substantially and the migration continues still continuing. 

users workaround has been to get someone else to save the file after running the macro potentially in office or outside working from home or in another region.

 Now I was wondering on what to do to trouleshoot we've tried disabling the webclient.exe locally didn't help. any ideas as the problem is potentially impacting other sites abroad too not just the UK, - some actions I was going to suggest to a manager who's looking into this now after it's been escalated  - I am not in the infrastructure team just local support but would like to know if anyone has seen this

I believe they get a mydomain.ad.net loading bar too where it hangs there for a while too, which is an indication of slowness?

Would it be worth trying some of these Actions?

Change the office updates method for a test device

Rejoin a device to the domain

Run Network monitor and Process monitor

Consider disabling Antivirus/remove as a test

Use FW monitor

create a network TCP dump

any other actions you guys can recommend

[G_W_Albrecht]

I would suggest to contact CP TAC to get this resolved ! Or did you do that already ?

[hirogen10]

I will find out 

[the_rock]

I know customer had similar issue couple of years ago and TAC had them disable a specific blade to get this solved, but I cant recall which one though. They ended up doing bunch of debugs to tweak a setting in the portal at the end. Apologies, I really cant remember now exactly what was done, so as @G_W_Albrecht said, maybe best to get TAC help.

[hirogen10]

I've just asked the infra guys to log a ticket they have 1 with Microsoft too, should be interesting, I found some users who have the issue so will do some testing later today with them, maybe run a network and procmon capture

[hirogen10]

Apologies for the delay the issue has finally been escalated into a global issue and it’s now a p3 within the company, apparently the 2 sites which just happen to be nearest to the Netapp share’s don’t experience the slowness however they’re both on Cisco VPN so I believe there’s still a chance it could be our vpn, especially considering we moved away from non-domain based computers over the last 6 months. So the vpn has to be active. Hopefully they will log a TAC case once we have tested the cisco vpn on a UK machine

[Chris_Atkinson]

Are you able to test Endpoint version E87.30 and which blades are enabled other than VPN?

Resolved in E87.30:

[AHTP-26963] Threat Emulation may cause slowness to applications handling files in the network folder.

[hirogen10]

Hello sire -Wow that was a quick response - we're currently on E86.20 Build 986103717- that's quite interesting having run a procmon capture I did notice some a lot of DLP entries. I will raise this with them!

[hirogen10 · Accepted solution]

During the months of investigating, we did together with Microsoft we have checked every single component we could (DFS, NetApp, network, client settings, Office and OS version), but at the end came to the conclusion that this is caused by latency in combination with the way how Office apps work (and always has worked). After talking to multiple users, we understood that this issue originated already on the old SWING platform but has gotten worse with every step we have made to the modern world. There are still components that are not migrated to the modern world like NetApp. When we migrated a dataset to SharePoint Online the issue did not appear anymore.

In combination with the statement of Microsoft that there is nothing they can do from their side (see below), we can only advice to speed up the migration to SPO. As a result, we will close the ticket.

Statement Microsoft Engineer:

I wanted to be certain of the Office versions, OS version and various network constraints before I can officially confirm our understanding about the time taken. I have been working on my lab which took longer then expected.

After various tests, I can confirm that with latency, office applications are bound to take time save files on the network where there is a latency. This is due to the design of the office application. The transactions that the office application does is to ensure that the user data is safe and thus there are multiple copies on both local and network device which provides ability of office applications to do auto recovery, on demand recovery etc.

These features are present by default and are useful in case of accidental application close/crash or loss of the open file due to any other event. The downside to it is that this task requires a lot of network transactions for office, locally as well as remotely. If there is a latency in the network this will certainly add to the delay as we are observing. That said, all we can do in this issue is to try tweak a few things to optimize but that won’t be a drastic change thus there is no point in taking any further data/logs. This is a by-design behavior of office application due to latency of the network.

[the_rock]

Thanks for sharing. I also always find that MS support is super diligent in giving the answer at the end of the issue, they sure do their investigation.

[Timothy_Hall]

Thanks for the followup, sounds like those older Microsoft protocols are extremely serial with no ability to parallelize or pipeline these requests.  It is probably Request/Response, Request/Response, etc...  with no overlap allowed.  Needless to say high latency would really mess up the overall performance of these protocols, which were almost certainly written in another era and under the assumption that they would always be used on high-speed LANs with consistently low latency.  Oops...