VPN client disappearing while loading virtual adapter

[cdooer]

Hey folks. Anyone ever see a Checkpoint VPN client go through the login process normally, but then right when it gets to the point of Loading Virtual Adapter, the app simply disappears (no error message). It passes authentication, and even gets an Office Mode IP, but just crashes. Latest gateway version, and very new client version. Only affecting 2 out of 3 VPN clusters, and seems to have started out of the blue. I do see a drop from the client using fw ctl zdebug + drop, but there is no reason given;

@;3284747.10304;[vs_0];[tid_1];[fw4_1];fw_log_drop_ex: Packet proto=17 10.1.1.1:18001 -> 60.50.40.30:18234 dropped by vpn_drop_and_log Reason: ;

The issue started about 2 weeks after upgrading to R81.20, take 98 on the first set of gateways. About a week later, a second VPN cluster started exhibiting the same issue (no changes), and we're now down to one working cluster supporting about 1000 users, and given we don't know what's going on (nor does TAC), we're expecting to be out of business shortly. 

One very strange fact is that the MAC clients we have, which are far and few between, seem to be able to connect fine, it's just the Win10 clients who can't. Also, we rolled one of the gateways back to a R81.10 snapshot we had done previously, and it didn't fix the issue. Very confusing issue we're having here.

[PhoneBoy]

"Very new client version" please defined the exact versions used.
For Windows, E88.70 is the most recent.

[cdooer]

Ya, using 88.70, have also tried a couple of older versions. Haven't had any issues for several years, and this just sort of started out of the blue. No changes on gateways, which is what makes this extra strange. Clients got dropped mid day, and have been unable to connect ever since. 

And MAC's still work, for some reason. 

[PhoneBoy]

Curious, have you rebooted the affected clusters yet?

[cdooer]

Yep, problem persists. This is from the helpdesk.log file. The bolded line doesn't show up when successfully connecting to our working VPN cluster, not sure if it's a red herring though;

[7 Apr 13:49:52] Client state is idle
[7 Apr 13:49:52] User pressed connect
[7 Apr 13:49:52] Creating primary conn flow to VPNNAME
[7 Apr 13:49:53] Client state is connecting
[7 Apr 13:49:53] Interface change - location is OUT, restarting connection
[7 Apr 13:50:13] Transport is auto detect
[7 Apr 13:50:14] Sent ClientHello
[7 Apr 13:50:14] No need to upgrade client, client version is 986105823
[7 Apr 13:50:15] Starting new connection (3)
[7 Apr 13:50:27] No need to download topology
[7 Apr 13:50:27] Client state is connecting
[7 Apr 13:50:27] Policy changed, restarting connection (3)
[7 Apr 13:50:50] Sent ClientHello
[7 Apr 13:50:50] No need to upgrade client, client version is 986105823
[7 Apr 13:50:51] Starting new connection (3)
[7 Apr 13:51:06] Topology download in progress
[7 Apr 13:51:07] No need to upgrade client, client version is 986105823
[7 Apr 13:51:07] no need executing firewall step
[7 Apr 13:51:07] no need executing scv step
[7 Apr 13:51:08] No recorded Hot Spot connections
[7 Apr 13:51:11] Service was started
[7 Apr 13:51:11] chose single user mode
[7 Apr 13:51:13] Client initialized successfully
[7 Apr 13:51:13] Client state is idle
[7 Apr 13:51:13] received network OUT event while state is idle. no action

[the_rock]

Sounds like red herring to me personally...did you open TAC case for this?

Andy

[cdooer]

Ya, we've got a TAC case open. It's with R&D, not sure we have that much time though, it could happen to the last cluster at any time, then we'd be in trouble. 

[PhoneBoy]

The question is where the bug is (Endpoint, Gateway, both).
Please send me the SR in a PM.

[cdooer]

Ok, so here's where we're at. Had a call with TAC, the the engineer suggested changing this on the client;

client_policies VEC_STR    trac_client_1&#range&#desktop&#mep&#user_groups&#scv&#dns&#extended_ranges&#  GLOBAL  0

to 

"client_policies                    VEC_STR    trac_client_1&#range&#desktop&#user_groups&#scv&#dns&#extended_ranges&#  GLOBAL  0"

So basically removing mep&#. After doing this, rebooting, then recreating the site, we can connect. They can't tell us why this is all of a sudden causing an issue with 3 out of 4 gateways, but it certainly seemed to have solved the issue. Wish there was a way to set that on the gateway, rather not push the change out to all of our clients. 

[PhoneBoy]

I suspect there is an issue related to MEP that may be triggering the issue in the client.
This is hopefully a workaround while R&D tries to find the root cause.

[Sony_James]

Were you able to resolve this issue? How?

[cdooer]

Not yet, still with R&D. 

[Machine_Head]

Hi there, did you get a fix?

Whats the file TAC told you to modify earlier? About removing the MEP part?

Facing a similar issue on a setup with 2 clusters and automatic first to respond MEP.

The issue seems to happen when you connect directly to the "secondary" one...

[the_rock]

Does it happen or just one machine or multiple of same OS version? 

Andy

[cdooer]

Happens with every machine connecting, with the exception of a MAC. Wish I could put my finger on why the MAC works, and Win10 doesn't. They both use LDAP to auth, not sure what other differences there could be. 

[the_rock]

Someone from R&D explained the behavior difference to me while back between MACs and Windows when it comes to VPN clients, but I totally forgot. I worked a lot at one point with customer who was using 90% of MACs in their environment and we never had any issues, only with 10% of windows machines.

Personally, I can only logically assume it may have something to do with certain affected files being modified. What did TAC say?

Andy

[cdooer]

We've got a call with TAC shortly, I'll let you know. On a side note, trying to gather as many logs as possible. Do you know if this entry in the client side trac.defaults would possibly be suppressing anything valuable?

excluded_log_topics VEC_STR SALSOCKET&#tunnel>=3&#transport>=4&#TrComInf&#messaging&#IKE>=5&#DIAGNOSTIC&#negs&#MessageLoop>=5&#tcpserver>=5&#TR_SCVPOLICY>=5&# GLOBAL 0

[the_rock]

Never seen that value before, so no clue, sorry.

Andy