FED UP WITH ENDPOINT - Last Straw - Support for Server 2019 but not the Hyper-V part

[Chris_Butler]

Hi All,

<edit> It took nearly two hours to write this in a way that makes sense, despite how furious I am with what I heard from Support and our Reps today, and as soon as I did, the system marked it as SPAM.. I thankfully was able to hit the back key and get back to the edit window and retrieve this text. I saved it to a local document and will Re-Post if it gets taken down again<edit>

I have been a proponent of Check Point since starting my tenure at my company in 2015, and as the ecosystem has evolved, I have come to CPX NY and watched the presentations and webinars and can see a truly integrated single pane of glass that is almost a full reality coming, What Check Point does is literally amazing and on an incredible scale. However....

I have been struggling with Check Point Endpoint Security since drinking the cool aid a couple of years back and switching from Symantec.

The promise of the product was fantastic, Reversal of Ransomware Attacks? Cloud based integration protecting our endpoints in the same ecosystem that protects our perimeter? One pane of glass eventually for both? Zero Phishing? Threat Extraction at the desktop? Awesome!

With little exaggeration, I have spent about 20 percent of my man hours prior to the release of endpoint client 80.96 kludging workarounds for the way that the EP Client broke some of even our simplest workflows and vertical app access on bone stock freshly reinstalled workstations. The tickets are all in TAC to prove it out. First it broke our ActiveX based portfolio management system access at the browser level, then broke our access to our investment banking partner multiple times, and even breaking things as simple as the ESPN site with the TE and TX blades enabled, then later without them enabled. And the problems were inconsistent, involving GUIDbedit hacks with TAC and countless separate blade policies based on who was using which affected workflows, leaving many of the blades disabled for them.

For a single IT Pro (and as of this year, Director of IT) at a small but high net worth wealth management firm with FINRA and SEC expecting us to be secure, endpoint security has to be enabled and has to be a top priority. And since our CP EP experience affected many workflows, the 20 percent number is not surprising

We bought the product because it worked on our virtualized and bare metal servers protecting them from ransomware attacks, botnets, suspicious exe actviity, and the like and they have been TROUBLE here and there but they have run, and with the help of an excellent SE (who left the company in January) and a great sales rep (who left the company in February) and a replacement SE (who left the company in march)...  well, ok I HAD a great team with a rapport that was top notch who got me in touch with the right people and even did some above and beyond hand-holding considering my workload along the way, and it kept me drinking the Kool Aid, even at room temperature..

That mass exodus bode badly, but thankfully, once I had done all the work to get all the hardware and software purchased to begin our mass migration of our entire windows infrastructure due to Microsoft's End of Extended Support that affected literally everything in our environment (all workstations were still Windows 7 Professional, All server OSes were Windows Server 2008R2, Exchange server is 2010)  I was able to turn my attention back toward Endpoint and see if there were any improvements the latest clients might offer.

I was overjoyed to find that I was able to deploy 80.96 without a single issue on all the servers and workstations, and one by one I was able to enable blades that we had paid for for two years but could not use because they broke things. 

So, when I began the first deployment stage of our new Dell R740xd server with Server 2019 Std Hyper-V on the bare metal, I was expecting smooth sailing, It was Microsoft Best Practices to the best of my **bleep** retentive ability and I was methodical. 

So I deployed one 2016 VM for our Portfolio management system application server and another for its database server, turned that over to the consulting group handling the migration / upgrade from the existing servers to those. 

In parallel, I created another VM with the identical VM and guest OS configuration and brought it up for testing the Endpoint Client. I joined it to the domain, found it in the Endpoint Management console, and assigned deployment policy to it.

After a first stumble due to a Server 2016 and Compliance hotfix that might have caused the issue, I blew away that VM and created a new one, identical, with a different hostname and joined It to  the domain to try again.

Starting small, I enabled only the AntiMalware blade in the deployment policy.

Installed the Initial Client. All was well...

The client picked up the deployment policy and the upgrade began. As soon as the client instantiated after that.... WTF...  the Hyper-V guest restarts as if you pulled the "virtual power cord" out of it. It comes back up, you log in quickly get to the desktop and look around there was no BSOD, no way to get into safe mode and stay there, No memory dump file to go on. And errors in the Hyper-V logs on the Hypervisor Host...  Before you know it the cord is pulled again and it starts over..  If you left it to it's own devices, it would boot loop like that endlessly once for each of the 90 seconds it took for the server to come up, and the Endpoint Client to get to some particular state in its startup, whether you logged into windows or not. 

I had opened a ticket with TAC before the first VM attempt that I mentioned with the 2016 and compliance blade as part of the deployment, explained the environment to him, he did research and suggested that perhaps the 2016 hotfix would be needed. Nothing came to mind for him about Hyper-V not being supported at that point, so he continued his research as I created the second fresh VM and did not deploy the compliance blade that time, Just the anti malware as mentioned. After two days of us working on it he indicates the Release notes, with the inferrence that Hyper-V is not supported, just VMWare ESXi  and apologizes for the inconvenience this causes.

He was a nice guy, I have no problem with him at all, I am not angry with him nor do I doubt his capability as an engineer... All the TAC engineers I have worked with have been great so far.

However, this answer is clearly not acceptable.

I am an RTFM kind of guy. I read the release notes, search the support portal, checkmates, I did my research, I googled (before AND after deciding on Hyper-V as our new environment's Hypervisor) for "Check Point Endpoint" and Hyper-V and 2019 and 2016 and every derivation thereof, but NOWHERE did I ever see anything like a support matrix that expressly indicated that they support Windows Server 2019 or 2016  but do not support the Hyper-V component in it. NOWHERE. Nobody stating in any blog, "Endpoint does not work with Hyper-V"

All our current workloads are running fine with Check Point Endpoint 80.96 clients under a much more edge case hypervisor, namely Proxmox VE which is Debian Linux-based KVM virtualization. No catastrophic problems whatsoever.  The issues we DID have were exactly the same on the Server VMs as they were on the bone stock Windows 7 Pro workstations. The hypervisor did not come into it at all.

Though Debian KVM is not expressly supported in the release notes, we were able to do a test deployment in the environment when we first bought the management and endpoint packages so we were ok, even though KVM is not explicitly supported in any of the EP release notes or product pages, and we were not discouraged by our sales and engineering team about deploying the clients in those VMs as I recall. 

I have now wasted two weeks of juggling the overall workload of the infrastructure deployment and testing Endpoint with Hyper-V and trying to figure out what is going wrong while trying to keep the Portfolio Managment deployment going, and doing all the other jobs that a single IT Pro at a company like this one must do day to day; not seeing my kids awake,  coming in at 7am and leaving at STUPID PM every night and this is the answer I got. 80.96 was the light at the end of the tunnel but that light it is indeed the proverbial freight train. Rep and TAC hands are tied, pretty much certain no one will work with me to get it off the ground until official support for Hyper-V is reached in the CP EPS Roadmap.

This means we have wasted the money we spent this year for our Endpoint Managment and Client support and licenses, and will not be able to use the core parts that drew us to the product in the first place, where they are most needed , ON THE SERVERS. Our new environment will have more than 70 percent of its workload virtualized. Where does this leave us?

I am beyond frustration at this point but what really gets me is that the documentation is vague, or it is misleading, depending on how you look at it

At the VERY LEAST, someone should tell the documentation group that if they put windows server 2019 or 2016 in as a supported platform for the client, they need to include an asterisk and caveat because HYPER-V is a ROLE on those platforms. if it is not supported IN ANY WAY whether on the bare metal or running as a VM in it, it needs to be EXPLICITLY stated that it is not supported. Hyper-V is not a separate product, it is part of Windows Server.

I always base my purchasing decisions on what I read, and I do read the release notes for the clients. Server 2019 is supported without any asterisks. HYPER-V is a standard Role and has been since Windows Server 2008R2. If Check Point says it supports Server 2019, it must support the whole of the OS unless they state otherwise.

 I hope someone in Check Point engineering or support can help me at least see if there is something simple we can try, because the Triple Fault error seen in the Hyper-V logs was mentioned only for one thread on the MS blogs and though there was no hotfix yet, the official workaround was something simple, namely changing the MAC address on the VM, rebooting it, and changing it back, and rebooting it. This did not help in our use case, but that's what I am saying there may be something simple that can get us by so we don't have to finally give up on Check Point Endpoint protection and change all our reviews on Gartner to reflect how we now really feel at the end of all that promise. 

Sorry all, Just needed to vent. Feel free to flame me now 🙂

Some related links

There appears to be some Hyper-V VDI support

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

The 2016 server compliance Hotfix which was a red herring in this case

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

The Hyper-V Triple Fault Bug workaround, not related to Check Point, but this is the error we see in the Hyper-V host logs when I install CP EPS in the VM

http://www.checkyourlogs.net/?p=59953

Build from May 2019 where MS ostensibly fixed the bug, Ms Blog replies refute this as being fixed however

https://support.microsoft.com/en-us/help/4497934/windows-10-update-kb4497934

[Eyal_Magidish]

Dear Chris,

Thank you for your feedback, We do appreciate the time you put in this feedback.

I can understand your frustration and feeling, and I took this with our product manager.

As you may know, we are on a monthly release cycle, meaning we have a new Endpoint client every month (give or take) that allow us to keep our product with the maximum security and fix issues reported by our customers.

We will debrief this scenario and we will try to reproduce it in our lab.

I have found your SR number, If that's ok with you I will contact you offline.

Sincerely,

Eyal Magidish

[Chris_Butler]

@Eyal_Magidish 

ThanksEyal!

Feel free to contact me via private message here or, if you have access to my email address via the SR system, directly there.

I was contacted personally by another employee of Check Point from Israel on Friday afternoon, I was supposed to have a phone meeting with he and presumably one or two product team members about the issue.. I am not sure if you are involved in that, but again feel free to contact me directly and we can move forward. 

[Chris_Butler]

I would like to let you all know, that I was contacted by Eyal and Neatsun Ziv within 24 hours of my initial post.

I will post again later with details of how fantastic they were and how they appreciated my comments, and put me in remote sessions with their R&D and QA teams from Israel and we went all the way down the rabbit hole in my own Hyper-V environment since  they do not have an equivalent bare-metal Hyper-V server lab.

Suffice it to say, I am very very pleased with the professionalism, skillset, intelligence, and effectiveness of that group of people. Neatsun put me in touch with the top of the development chain because he saw my initial post for what it was, a frustrated vent, sure, but not a mindless Check Point bash, rather something with enough background to make it worth spending the valuable time of his teams with me personally to work on getting to the bottom of the issue.

I am beyond thankful for the way that Check Point comported itself and how quickly they were able to help me. 

I currently have a workaround for the problem making use of a GA client with the E2 Antimalware engine, It is always an available option for other reasons where the E1 engine does not satisfy requirements, and has been for some time, However it is not available on the website, but must be requested from Check Point support. Any time there is a new GA client release, there will be an available client package with the E2 AntiMalware engine, by the way

In the 81.10.7220 client with the E2 AntiMalware engine's case, the client is stable and does not cause Hyper-V Guest VMs to crash.

I will keep you all posted as they are working with Microsoft directly now with the kernel debugging sessions we did together at the Hypervisor OS level as well as the Guest VM level to get to the bottom of why the E1 AM engine causes Hyper-V to triple fault and kill the guest OS.

Thank you to Neatsun, Eyal, Konstantin, and Maksym, and most recently, Kiril for all their help.

You have restored my faith 🙂

Chris.

[_Val_]

Hi @Chris_Butler, as you assumed, there are two identical posts now. Since we have @Eyal_Magidish answer here, I will keep this one and remove the second, where I have explained the situation with SPAM flags.

Once more, there is nothing against your post, on the opposite we appreciate you being straight forward and thorough when explaining your issue. Yet, we cannot do much about automatic spam flagging, as, you know, we do have spam that needs cleaning. I hope in the future, you and other community members will give us a moment to clear the flag when it happens.

To make sure we did not hide anything, I am posting below the thread from the _other_ post.

--------------------------

@Chris_Butler, I understand you are very emotional and frustrated, but let me address the part where you complain about your posts being taken down.

Unless you are porting inappropriate things, we do not take down posts. This is our transparent and continuous policy. Your post was automatically flagged as spam because of multiple links in it.

We do release spam queue on a regular basis, but our admin team does not have 24/7 availability. That mean, if you post on a weekend or after hours (and here we cover both European and US hours, usually) you need to be just a bit patient and give us a chance to release your post before you suppose there is a conspiracy against you here.

Reporting abuse on your own post, as you did, will also not help to speed up things. It is getting sent to the same admin team I have mentioned before.

Please, in the future, if you have any issue, send a personal message to me or @PhoneBoy. But event in this case, I cannot promise you a momentary action, so just give us a chance. 

Your post is unmarked as spam now and is available for the community to read and comment.

Let me know if anything else. 

Thanks,

Val

--------------------------------

@Valeri_Loukine 

Thank you for your response, Val,

One interesting point, it seems, on further reflection, that the post was only marked as spam when I tried to share it to email recipients, My two last re-posts were not automatically flagged, and they had the same multiple links in them.

Please understand, also in the case of the SPAM marking, I am not frustrated or angry with you or your team, but I was furious and backed into a corner and trying to explain my experience as clearly and urgently as possible, when my post was marked as spam.  I was further infuriated by the fact that there was no way for me to go back and revise and re-post it or recover the body text within the user interface. Luckily the back button worked in this case and I got the form back pre-submission.  I had been working on that body text for the better part of two hours while doing damage control on the deployment which was impacted by the Endpoint Client incompatibility, so as you may imagine, that is why I put the edit in the new body text so I could voice my frustration at that as well. I was (and still am) under the gun on a deadline and needed that post to be seen as soon as possible because I was getting nowhere through normal channels.

I understand the posting / editing mechanism now, and in future will of course follow your guidance.  I also will not rely on the CheckMates UI's editor to put together any post that is more than a few sentences long, I will use notepad first then cut and paste it into the form. (I have used knowledgebase systems in the past which expired your session in the background while you were editing and did not inform you, so this is not a new practice for me 🙂 

As for your implication that I think there is a conspiracy against me, lol, not at all 😁  I have been involved in IT since before the advent of the world wide web and have been around the forum block since they were known as BBSes and required direct dial-up, and I realize how these systems often work. That was not the point of my edit note; The SPAM mechanism just happened to be set up in such a way that when I posted and it triggered, it was like insult to injury to me at the point where I was trying my last options for being heard.  I just finished trying to clearly and fairly but firmly explain a road fraught with potholes dealing with the product, and it was just my insanely bad luck that the way the system works would automatically pour some vinegar onto a fresh wound, you know? I was venting my frustration at the confluence of influences.

And, as for reporting abuse, at the point where all my work was erased on the site, and I was at the peak of frustration, A LINK CAME UP SAYING CLICK THIS IF YOU BELIEVE THE SPAM FLAGGING WAS IN ERROR.   I thought for CERTAIN that link would take me back to my post to revise it, but instead, it was a standard report abuse form, and my body text was apparently gone, and nobody would see this until whenever the admins were able to review it. So, in that abuse report message, you saw the result of my final frustration. I was livid at that point.

I am bothering to respond to your reply with this much detail, because  my re-posting  until it stuck (after I learned not to use the email sharing feature) was out of a time-sensitive need, and it turned out to be appropriate, because I was contacted by someone who has the purview at CheckPoint to possibly help me, The person in question is gracious and helpful. I got a response personally, and I am VERY grateful for that . Had I not re-posted, I would not have had an answer for upper management regarding our endpoint security in the new environment at all, and they would need to stew on that for the next 72 hours.

Thank you for re-instating the post, I believe it might now be posted multiple times on CheckMates, possibly more than twice with the re-instatement. If so, I will clean that up and leave behind only one post when I have a a chance on Monday.

Thank you for your help, and your understanding.

Chris.

[Chris_Butler]

Hey Val,

I thought my last reply already dispelled any notion that I have any expectations for instantaneous remediation of spam flagging, as I explained, you just saw someone at wits end in an unfortunate sequence of events. Further, I now know that there is not a 24//7 moderator done in shifts  No need to "hope in future" when it comes to me, as I already mentioned that I will act on your guidance in future,  I only need to be told once: I am already on board 😁

My apologies for you being on the receiving end of that chain of unfortunate events 🙂

With all that said, I would ask that perhaps someone in web dev can look at the possibility of putting in the option of recovering a post that is marked as spam and giving the end user the chance to revise and re-post it?  That would certainly lighten your workload at least a bit. Perhaps the spam filter could be run at form submission time?

Ok, I will stop beating a dead horse now, Thanks for all your help again Val

Chris.

[PhoneBoy]

One thing I've noticed that triggers the spam filters more often than not is editing a message more than a couple times.
Regardless, we can restore messages flagged as Spam easily enough.
If something seems amiss, feel free to contact Val or I.

[Chris_Butler]

Ah, good to know, thanks!

I am not sure if it is coincidence, but the spam flag seemed to happen contemporaneously with attempting to share it to email recipients using the CheckMates share feature. 

Could just be timing though, not sure, but for certain, I will do the same thing I did with the home-grown knowledge base / ticketing system my former consulting group used, which is to work out the body text offline in notepad or another editor and cut an paste it when it is complete and looks ready to submit. Only took a couple of times of posting to a session that timed out 15 minutes prior with nary a popup to warn ya.

The CheckMates form also does the same thing if you are pulled away from posting something for a decent amount of time. You post and it takes you to a login screen. Thankfully, whatever the method used for the form allows you to use the back button on the browser to get back to the former state of the edit window and get to the copy as it looked before hittng post.

Good to hear from you again, Dameon, be well 🙂