Disconnected Network with CP Endpoint Random/Intermittent

[Steve_Lander]

Is anyone else running into a problem where network connectivity is disabled (LAN/Wireless says connected, has correct IP address, but cannot connect to the internet) and the only way to get it back is by rebooting or ipconfig release/flush/renew, and even then sometimes that doesn't work.  I have the following blades below (We do not use Capsule Docs or Endpoint VPN), and its happening on E80.70 and also the HFA1/2 versions of that client running on Windows 7 Enterprise 64bit.  We also use a Firewall policy that switches to an offline policy which there might be an issue with switching between policies?

I cannot seem to replicate this issue, but seems to be happening more with remote users.

[Kim_Moberg]

Hello Steve,

Lately we have seen some wierd issues loosing connection to our DFS fileshare. Internet access works. 

We havent been able to localize if it is related to E80.70.0209 or E80.70 HF1 or if it is related a Windows update being deployed on server or client pc.

We are using Windows 10 64bit ver 1703 and lately testing on ver 1709.

Are you totally being disconnected from the internal Network?

Best regards

Kim

[Steve_Lander]

Kim,

We are working with support to try to find a fix for this issue but it seems the issue is getting worse as now many of our remote staff cannot get on the internet to VPN into our network.  

We are also testing with Windows 10 64bit Enterprise 1703 and 1709 and it seems that the problem is also happening on Windows 10.   

In regards to your question Kim, when the machine encounters this issue on our internal network, it seems that port 80 and 443 get blocked.  They lose all Internet connectivity, cannot connect to the Checkpoint Endpoint servers or the Checkpoint Identity agent, and lose connection to our Cisco Jabber app.  We can however, still remote into the machine using Landesk.  Nothing shows up being blocked in the logs, so this must be either a bug in the product or they need improve on what is being logged.

Has your problem been resolved for you?

Steve

[Larry_Porschen]

If you check the task manager do you see anything as being suspended?

I have a similar issue with a web proxy program called websense/ForcePoint.

The websense - "wepsvc.exe" gets suspended and restarts over and over. I end up with 100 suspended wepsvc.exe in the taskmgr.

I have tried to exclude it from the anti-bot or threat emulation blade ? can't remember exactly- but it has not helped.

It seems to be a very intermittent problem. Just when I think it is fixed it comes back?

Basically the result is not being able to hit the internet until the machine is restarted. It has on occasion remained suspended through one reboot and a second reboot was necessary to clear it.

Good luck I will update as I learn more as well.

[Steve_Lander]

Larry,

I'm seeing a bunch of svchost.exe (around 10) but no processes that have duplicates over 10 on the machine having my issue.  Also, my problem persists through a reboot/shutdown.  I am seeing a bunch of chrome processes (around 4) everytime I boot the machine, I am thinking that is preloaded for the Chrome SBA extension?

Have you added that process into both Threat Emulation policies?  There is the Forensics Policy and the Threat Emulation Policy that you need to add it to. 

[Kim_Moberg]

Steve,

I have also seen a lot of svchost.exe running on my windows 10 ent 1709, but no of them have been put in suspended mode.

While using sysinternal ProcessExplorer I was not able to find any abnormal about it. 

Have CheckPoint given any feedback to what you are seing?

Best regards

Kim

[Kim_Moberg]

Hi Steve,

It is interesseting to see what are actually happing. Well for our user it is not that funny though.

I have seen a total lock down of the machine with almost no internet access. It is not specific related to 80/tcp or 443/tcp. 

What I have seen have been Threat Emulation blade were failing or not starting correct when machines boots into windows.

When this happens, I cannot ping websites or internal servers with hostname. reply times is also 4-5 minutes. But if I try to reach them by ip directly, I get a response right away.

So for example c:\> ping google.com 

takes 4-5 minues to resolve.

if I try c:\> ping 8.8.8.8

it takes less than a couple of seconds to reply.

I am in dialog with CheckPoint about this problem. Taking windows kernel dumps, and extracting different kernel dumps from running windows program. This can be done for example by running task manager and right clicking on a server or program, and click on create dump file.

Because we use sharepoint via webservice or web as part of office addin, they frezzes because of the 4-5 minutes timeout because we connect to hostname. So when it happens, the user cannot work or do anything.

When windows have been in locked mode, the only option right now is to reboot machine and hope that threat emulation starts. We have also seen if one blade is not running, the system are starting to generate different kind of problems.

Of course I hope CheckPoint developers are going to solve this. At the very moment it is only 1 machine affected of this bug.

So we spend a lot of time and effort to explain and finding the root course of the problem.

Hope my feedback help with to narrow the problem. 

Is this the same problem you see?

Best regards

Kim

[Steve_Lander]

Kim,

After many days and nights of testing, I believe it might be a combination of the Sandblast Blades and Anti-Malware Blade thats causing out issue.  I have seen your exact issue only on one of our machines, but our major recurring issue is different. We just rebuilt the machine that locks up and can only be hard shut down so after I figure out a solution for our ongoing recurring problem, I'll take a look at that one.

-Steve

[Kim_Moberg]

Steve,

I’ve Got feedback from developers today. Root cause to my problem were related to firewall and threat emulation blade. Which were blocking the machine.

I have today installed a fix  and need to monitor any problems the next couple of days.

Best Regards

Kim

[Daniel_Taney]

Kim,

Was this a software fix supplied by Check Point? or Registry / Windows changes recommended by Check Point?

[Kim_Moberg]

Danny,

Yes EA developers made a fix to our e80.71 client where we had to exchange three files.

right now it seems to have solved our issues.

I dont know if the developers have a fix for e80.70. I Think they will need to analyze the root course to your problems. The fix is already included to the future GA of e80.71.

[Daniel_Taney]

Would you be able to tell me which files they replaced? Is there any chance you'd be willing to message me your SR #, as well? I'd like to at least bring this to the attention of the team looking at our problem in case it is related and they just aren't making the association. 

Thank you in advance!

[Kim_Moberg]

Daniel,

Please take a look at this thead from our local CheckPoint SE published yesterday.

E80.71 is now in GA, and therefore you would be able to get the version which have the fix included.

Check Point Endpoint Security E80.71 is now GA! 

Please take a look at sk119676 to download E80.71

Thanks

[Daniel_Taney]

Great, thanks! We will pull down the GA release and see if it resolves anything for us. Fingers crossed!

[Steve_Lander]

Not sure if you use Full Disk Encryption but do you know if you had to upgrade your SmartConsole to be able to use FDE Recovery?  I am getting this message below that is saying the version of SmartEndpoint is not compatible with the uploaded client.  We are on R77.30.03 which as far as I know the latest version.  

[Kim_Moberg]

We are currently not using FDE but because EA engineer recommended to E80.71 GA release because it should be more smooth.

For running E80.71 we needed to upgrade our R77.30.03 to a newer version. We run ver. R77.30.03 (990002040) as endpoint mgmt.

does that help?

[Steve_Lander]

Yes thank you!  We will download the newer version of the smartconsole.

[Kim_Moberg]

Steve,

I noticed something interesting today.

As explain above we are having problems with DNS lookup but accessing webpages or service via ip addresses seems to work.

Meanwhile if you set up and filter in Windows Event Viewer log from the affected machine.

What exactly:
1. Open Event Viewer -> Windows Logs -> Application
2. In Right Actions pane choose Filter Current Log
3. Choose 7 days, Event level: critical, error -> OK
4. In the same actions pane: Save Filtered Log File as…

I have seen "Faulting application name: svchost.exe_Dnscache, version: 10.0.16299.15"

I am wondering if this faulty DNScache is related to this specific problem.

I found this explanation on Microsoft Forum, but I haven't had a possiblity to test it out.

this issue can occur if the address for the configured preferred DNS server on the client is invalid or unreachable.
You can check the issue by manually assign the DNS address in the Internet Protocol (IP) properties:
Right-click My Network Places, and then click Properties.
Right-click Local Area Connection, and then click Properties.
Click Internet Protocol (TCP/IP), and then click Properties.
Type the correct DNS address in the Preferred DNS server box.
Also, Update device driver for the NIC.

[Michael_Hecht]

Hi, I have the same Problem. I also lose the Connection to Internet on Win 10 with Fall Creators Update 1709. Before this, all was ok. Now, after disconnection from VPN I lose connection, even if Wifi seems to be ok. I'm on a Win 10 Acer Notebook. The only fix is a Network reset: "netsh winsock reset" with reboot.

[Kim_Moberg]

Michael, 

What version do you run? Have you tried to upgrade to e80.71.023x? Or one of the newest version available for example e80.81?

Regards

Kim

[Daniel_Taney]

For what its worth, it seems like we've seen massive improvements across the board with the e80.8x releases. I'd also recommend making sure your Win10 machines are booting using UEFI instead of legacy BIOS boot. We began switching our machines to UEFI and the performance differences across the board have been staggering. 

[Michael_Hecht]

Hello together,

I'm not sure whether or not you received my direct answer. So I'm posting it here. 

I'm using the latest client that I downloaded a week ago. The reason was that I hat to reset my Computer for win 10 fall creators update (1709-16299). I'm using UEFI - this was the reason my wi 10 update didn't work originally (too small EFI Partition).

So

UEFI check

win 10 1709 check

CP Client e80.81 check

Are there any logfiles I can send to help?

[Michael_Hecht]

Hmmm ... to be totally sure I searched for the e80.81 version and reinstalled it (prior to this I was sent to the download page by the old client - if I remember right). After reinstalling the software currently all seems to work. Possibly it wasn't the newest Version of the client?! Windows also sent another minor update meanwhile.

It might be, that it is necessary to work a while on the remote System. This I didn't check after reinstalling, i.e. I just established VPN and connected and disconnected to the remote system and disconnected cp client.

I will keep an eye on this and maybe come back. 

Until my return I will nevertheless thank you for your assistance.

[Kim_Moberg]

Michael,

Good to hear that right now it is working for you.

Did you install/upgrade other software prior to your deployment of e80.71?

I cannot remember if I gave the root cause to our problems with the version we used. 

We had been testing this version on machines over time, and we didn't detect any critical incidents. Suddenly some our our controllers from the finance department complained their machines block the use of local and external resources.

Form there we started our bigger investigation together with Check Point developers. 

It was memory dumps from different programs and extraction of cpinfo from machines and so.. we had no 80/tcp or 443/tcp access. Check Point developers was not able to get remote access to do their work.

After a week or so, we started to see a pattern of who was affected.

It showed while upgrading to e80.71 our team had installed a new Microsoft finance application. So while we started to ask questions if this software could be the reason why the endpoint blocked and locked down the computers. Check Point developers did a amazing work, and pointed to a file named navsip.dll in a certain Microsoft build. 

Together with Check Point, Microsoft had as similar problem in their endpoint client. They did also confirm we could exchange this version of navsip.dll with a newer version, and then everything worked.

Check Point made a fix to e80.72 which was released in Q1-Q2 of 2018 that have a fix to the above issue.

I know our problems was solved in e80.71.0232.

We do deal with high CPU usage while once in a while, but as I remember the newer version of the Check Point endpoint have solved this.. I recall it might have something to do with threat emulation of scanned files while or after a scheduled malware scan.

Looking forward to hear, what the developers tells after analyzing your endpoint incidents.