Randomly losing VPN connection in Harmony Endpoint/Sandblast

[RickyDan]

Some users are randomly losing VPN connection while working from home and the only fix is to restart the computer.

Harmony Endpoint version is 84.50.7526. Windows versions are from 1909 to 20H2 with different hardware (Dell/Acer).

The standalone VPN clients continue to work perfectly with no dropouts or issues so I do not suspect the gateways are at fault in any way. Packet captures (tcpdump/fw monitor) showed that absolutely no traffic was hitting the gateways. This issue only happens with the Harmony Endpoint suite.

Anyone else experiencing this and has a resolution?

[the_rock]

I recall one customer having this problem, but maybe not exactly the same as what you described. So if you run fw monitor on the firewall and filter on port 18234 (tunnel test), you dont see anything at all? I know what we asked them to do after extensive TAC troubleshooting was to change sleep timer settings on their laptops to "never" and that did actually help largely. Is this something it had been happening since the beginning with sandblast?

[RickyDan]

Yea one user reported it the following day after installing Sandblast. 

I will repeat fw monitor with that port and update with the results. I only did it with src/dst IP addresses.

I'll also try the sleep setting and see if that has a noticeable impact.

[the_rock]

Just run below on firewall from expert mode when issue is happening:

fw monitor -e "accept port (18234);"

[RickyDan]

What would I be looking for exactly? If nothing shows up what is the next step? TAC?

[the_rock]

Pretty much, yes. You would look to see if office mode IP addresses are communicating on port 18234.

Andy

[Chris_Atkinson]

Have you tried any newer client versions other than E84.50 ?

[RickyDan]

Hey, I did upgrade everyone to E86.00 which was the latest at the time I pushed the upgrade but it did not help. However, I will give E86.20 a go for a few affected users and see how that helps. Thanks for the suggestion.

[the_rock]

@Chris_Atkinson brought up a good point...E86.20 is the newest version, but very stable, so I would certainly give that a go.

Andy

[RickyDan]

Thanks, will certainly give it a shot.

[the_rock]

Keep us posted on the results please. 

Cheers,

Andy

[RickyDan]

Hi guys, pushed E86.20 in the evening to a user who was having the issue constantly and this morning the VPN dropped again. I asked them to let me know the instant it happens so that I can do the fw monitor as well as look for anything related in Windows logs.

[the_rock]

I would definitely recommend TAC case, if you dont have one already. I know issues like this are not easy to troubleshoot, specially given the fact they would also have to do captures when issue is occurring, otherwise, it would not show the actual problem.

[RickyDan]

I got a trace going and the only thing hitting the gateway was tunnel test (UDP\18234).

May or may not be related but the user also has a ton of windows system event logs 7034 saying the Threat Emulation service terminated unexpectedly. To compare, I checked another user who does not experience the issue and they have no such logs. I removed Threat Emulation from the user but they still experienced the issue. 

I'll open another TAC case with this info. I had one previously but the issue was so seldom that no meaningful troubleshooting could be done. Thanks everyone so far for the suggestions.

[YuryRus]

Hi,

We are experiencing a similar issue. Endpoint Client displays as connected, but all network resources are unreachable until machine is restarted. It looks almost as "Host isolation" feature turned on for the host (During isolation all traffic is dropped except the connection to the management server).

Was your issue resolved and what did it take? and what was causing it?

[Chris_Atkinson]

E86.25 was also made available recently FYI.

[the_rock]

Right, but only managed, NOT standalone vpn client.