WMI Permission denied - From this months Windows Update

[nzmatto1]

Hi, 

We have a number or R81.10 gateways which are still using AD lookups and we have the workaround in place to permit this to still work as per: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

The next Microsoft date relating to this is supposed to be March 2023, however with this months patches going in on the domain controllers we have noticed our firewalls receiving the error WMI permission Denied when attempting to authenticate against the servers. Rolling back the patches on the AD server has fixed the issue. 

Is anyone else facing this and aside from moving to AD Collector is there a fix for it?  

https://community.checkpoint.com/t5/Security-Gateways/Move-from-Identity-Awareness-AD-Query-to-ID-Co...

Thanks.

[George_Casper]

Similar situation.  We have the proper Jumbo installed since June to address compatibility with DCOM hardening fully enabled and working fine until Windows AD controllers updated with October patches.  Not seeing the tell tale event 10036 messages in AD event viewer that would be associated with the DCOM hardening issue.  Rechecked WMI permissions and don't see any issues, just getting WMI Permission Denied on the gateways.   Opened a TAC case and waiting for a response.  

[PhoneBoy · Accepted solution]

After applying this month's patches, did you check to see if the registry key specified in kb5004442 is still set appropriately?
In any case, our official recommendation is to use Identity Collector (mentioned in the SK you linked).

[George_Casper]

Identity Collector is the way to go.   Just fixed one site with it, working on the rest.   Unofficial from TAC, delete if not appropriate to post here and accept my apologies in advance:  TAC found a commonality with a few of us that called in about the same thing, a ported hotfix on top of r80.40 jumbo 158 (bug introduced in Jumbo 156 that caused VPN timeouts after 2 hours sk178891). This SK also affects r81 & r81.10.  Seems it is a combination problem with the hotfix & the Microsoft October patches.    Interesting that this hotfix is now incorporated in Jumbo 180 so wondering if it could haunt those that are still using AD query when it comes time to install Jumbo 180.

[nzmatto1]

Yes, as per George_Casper, we have the registry changes and hardening done, my experience is identical to his. I am now experiencing this at two sites. I expect other people will come across this as the patch cycle works through.

[George_Casper]

P.S.  The identity collector was a breeze to install/config. Wish I did it months ago.

[FTZ]

Identity Collector is a great solution for Gaia firewalls, But what is the solution for the SMB Gaia embedded firewall? Identity collector is not supported. We tested it anyway, but remote access didn't work after we activated Identity collector on the gateway.

[PhoneBoy]

SMB appliances running R80.20.35 and above managed with R81.20 management will support Identity Collector.
I believe a future version of R81.10.x will support Identity Collector for locally managed gateways.

[neo3313]

Rolling back the patches on the AD server has fixed the issue.

[BG]

Same situation here. R80.40 gateways with jumbo 158(no other hotfixes are installed as mentioned before). After october updates connection to domain controllers are broken with the message "WMI permission error [ntstatus = 0x80041003]". Uninstalling the update solves the problem.

[G_W_Albrecht]

sk176148: Check Point response to CVE-2021-26414 - "Windows DCOM Server Security Feature Bypass"

[BG]

Applied windows patch was not KB5004442 so I believe the problem is not related described in sk176148. Applied patch was KB5018411 which was released this month for windows server 2016. Also I don't see any events with ID 10036 or "bad credentials or firewall blocks DCOM traffic [ntstatus = 0xc0000022]" error message using adlog tool. BTW uninstalling patch KB5018411 solves the issue.

[George_Casper · Accepted solution]

Same here, no signs of trouble with DCOM hardening.  Though may be related to DCOM or anything for that matter, the symptoms don't match at all in any way shape or form to either Microsoft's or Checkpoint's DCOM Hardening guidance.   Spent hours on the phone with TAC on Friday trying to troubleshoot it.    I'm sure someone will figure it out and create a hotfix or something else, but in the mean time just install Identity Collector and your problem will be solved in 20 minutes and it works.   While nice to have a dedicated VM, you can stand it up on most any VM in a pinch and then move it to a dedicated VM later.   It's surprisingly simple and not sure why I had a mental block about doing it 6 months ago but should have.   Below is an unofficial quick cheat sheet on the install steps.  Its a 33MB installer that only takes a minute to install and about 20 to configure.  You can do some fancy stuff in there later on, looks like there's integration with Cisco ICE, Aruba and some other stuff.  Will have fun sometime later with that but out of the box vanilla config is quick and easy to replace AD Query.

On the gateway object in smart console select Identity Awareness. Then select Identity Collector check box. Hit the green + arrow and add a host object with the IP of the machine that will have IDC installed on it. It will generate a shared secret, save this secret. Install policy.

Install IDC from this sk, sk134312. After it is installed click the * (blue star for new object). Create a Domain, name it whatever you want. The Identity Collector requires an AD user that belongs to the default Event Log Readers group. Add that user and click test, then after a success click okay. 

Click the blue star * again. Active Directory > Fetch Automatically > Provide the DC IP. Click Fetch > OK.

Next create a query pool (in the top left). Give the query pool a name and click the check box in the top left to select the Domain Controller.

Next create a Gateway object. I would name it the same name in Smart Console. Put in the IP address of the gateway in Smart Console. Apply that saved shared secret, add the query pool. Click test then trust. Click OK.

After you confirm it's all connected uncheck Active Directory Query on your Gateway object in Smart Console. Install policy.

On the CLI on your gateway you can verify connectivity with this command.

# pdp connections idc

P.S.  If you have a domain controller that identity connector won't connect to and you can ping it and looks good otherwise, on the DC itself, check Windows Firewall & make sure to Allow a Program Through the Firewall "Remote Event Log Management" and Domain network is Checked On

[BG]

Thank you very much for the installation&configuration steps. I also didn't want to use Identity Collector as it means two more agents, two more vms, two more things to maintain. But since Check Point recommends it and now this problem I have no choice 😅

[Greg_Harbers]

The Identity collector Technical overview recommends a 16 Gb of RAM and 12 core machines, and a maximum of 35 DCs per collector. In additon, to achive resiliency the recommendation is to have two separate windows machines configured identically.

One of our clients has in the order of 200+ DC. This would mean around  12 windows machines are needed to support this environment.

What has the real world experience been? what spec Windows machines are people using and how many DCs per collector have people configured? are people running the collector directly on DCs?

Thanks

[PhoneBoy]

I haven't heard of anyone running Identity Collector on their AD server, FWIW.

R81.20 has some pretty significant changes "under the hood" that will improve scalability and resiliency.
If I understand correctly, it should reduce the number of needed Identity Collector instances.

[Greg_Harbers]

Hi Dameon,

Thanks for the reply but you dont really answer the question. What is the really world experience with the resouce requirements for the identity collector? and will we need 10 plus instances of the collector to support an environment of 200+ Domain controllers if resilency is required? An additonal question, what, if any, thought has Check Point put into managing the collector versions? is there any ability to maintain the installed versions of the collector?

Thanks

[PhoneBoy]

You're correct in your assumptions you'll need that many Identity Collector instances.
My understanding is that real world experience bears these limits out.

To clarify my statements around R81.20, these two bullets from the upcoming release tell the tale:

• Improved resiliency, scalability, and stability for PDPs and Identity Brokers. Additional threads handle authentication and authorization flows.
• During a PDP failure, a PEP Identity Awareness Gateway can recover its identity database from connected PDP Gateways.

This should translate into needing less Identity Collector instances.

Not sure what you mean by "managing the collector versions."
We always have the latest one here and provide a change log for past versions: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 
Earlier versions can likely be obtained from the TAC if necessary.

[Greg_Harbers]

Once again, thanks for the reply. 

to expand on my question re managing collector versions, I mean that should we have 10 + servers out there running the Identity collector, when a new version is released, is there a centralised method by which we can update all 10 servers? I am not sure how often new versions are released, but looking at the version history table, it appears it is now on the 15th release.

at the bottom of sk108235, there is a link to sk178086 - "How to Upgrade Indentity Collector version", clicking on this link I get "Sorry, this solution is deleted and can only be viewed by Check Point employees", What is in this article that CP do not want us to know?

[PhoneBoy]

A centralized method to upgrade the servers?
Not that I'm aware of, but maybe @Royi_Priov (or one of his team) can comment on that as well as the upgrade procedure, which may be different than what is documented in the deleted sk.

[cir007]

Hello,

AFAIK in sk176148 it says:

To apply the Microsoft hardening and continue using AD Query and Identity Logging, you must install a hotfix.

The hotfix is included in Jumbo Hotfix Accumulators for these supported versions of Security Gateways / Security Management / Multi-Domain Servers:

• Jumbo Hotfix Accumulator for R81.10 starting from Take 55
• Jumbo Hotfix Accumulator for R81 starting from Take 60
• Jumbo Hotfix Accumulator for R80.40 starting from Take 158
• Jumbo Hotfix Accumulator for R80.30 starting from Take 251
• Jumbo Hotfix Accumulator for R80.20 starting from Take 208

I'm moving all of my customers to IC, but on some accounts, where this can't be done right away, I applied the JHF and it solved the issue. This was on R81.10 and R80.40 GWs. And yes, if you are in doubt, if this is the time to star using IC the answer is definitely yes - 15min and you are set. However Can CP please clarify the statement regarding JHF.

Br J

[Greg_Harbers]

Hi Dameon, 

Has, or will, Checkpoint officially announced that ADQuery is dead and is not longer supported? 

Thanks

Greg

[_Val_]

Why should we? It is Microsoft who is hardening the server side. 

Identity Collector is the recommended way, however, ADQuery will be supported to all the customers using it, until the very last one.

[Greg_Harbers]

Given that TAC are floundering with the issue that we are seeing across multiple clients and others posting to this topic are also experiencing, while it is easy to blame Microsoft we can't tell our customer not to patch their domain controllers.
Yes, I agree that Identity collector may be the way foward but for customers with many sites, many firewalls and many domain controllers, it is not as simple as loading IC onto a server and job done. This needs a design, planning, testing etc. 

Therefore, the question is, does Check Point acknowledge that after the installation of the October 2022 windows patches, ADQuery is no longer functional and is anyone assigned with coming up with a fix?

[_Val_]

Nobody "blames" Microsoft, there are security reasons to do the hardening. 

The acknowledgment of ADQuery becoming problematic after MS patching is done in November 2021, together with guidance to the customers on how to deal with it. Please look into sk176148, which is already mentioned above. The issue was also discussed in the community multiple times.

Hope this helps. If not, please let me know about any further clarification you may require.

[Greg_Harbers]

Hi Val,

To confirm a few points which I think others who have posted here have also said the same thing...

We do not believe this is  the same issue as described in sk176148 for the following reasons:

• The gateway devices are all running the required JHF versions or SMB Firmware versions
• After the release and installation of KB5004442 on the domain controllers in June 2022, ADQuery was operating correctly.
• As described in sk176148, the response from a gateway that is not running the required JHF or firmware version is “bad credentials or firewall blocks DCOM traffic [ntstatus = 0xc0000022]”. The error we are seeing is “WMI permissions error”. This started after the installation of the October 2022 Windows Server 2019 Cumulative update (KB5018419)
• By making the IA service account a domain admin, the gateway is able to connect to the domain controller.
• All of our customers that are using a service account that is a member of domain admins are unaffected, Those customer who are using a service account with the permissions as defined in sk93938 are affected.
• By adding or removing the account from domain admins and running the command "adlog a control reconf" we can clear or get the wmi permissions error.

based on this, is Check Point open to exploring if this can be overcome by granting additional permissions to the service account, or must it now be a domain admin to operate?

Thanks

[PhoneBoy]

Our official recommendation is to use Identity Collector and has been since Microsoft started tightening down WMI in response to CVE-2021-26414.
That is documented here: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Having said that, there is no official announcement about the death of ADQuery.
As Microsoft continues to lock down WMI, I would expect further issues to occur.

[David_Evans]

We have been given this as a possible fix from Microsoft.    We are just starting testing.

KB5020439

https://support.microsoft.com/en-us/topic/october-18-2022-kb5020439-os-build-14393-5429-out-of-band-...

[David_Evans]

This had no affect on our WMI issue.

[nzmatto1]

Same here. We deployed KB5020439 on one DC, and tested the access, still getting the same failure. I'm also trying to push the team forward with an Identity Collector instead.