Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
nzmatto1
Contributor
Jump to solution

WMI Permission denied - From this months Windows Update

Hi, 

We have a number or R81.10 gateways which are still using AD lookups and we have the workaround in place to permit this to still work as per: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

The next Microsoft date relating to this is supposed to be March 2023, however with this months patches going in on the domain controllers we have noticed our firewalls receiving the error WMI permission Denied when attempting to authenticate against the servers. Rolling back the patches on the AD server has fixed the issue. 

Is anyone else facing this and aside from moving to AD Collector is there a fix for it?  

https://community.checkpoint.com/t5/Security-Gateways/Move-from-Identity-Awareness-AD-Query-to-ID-Co...

Thanks.

0 Kudos
3 Solutions

Accepted Solutions
PhoneBoy
Admin
Admin

After applying this month's patches, did you check to see if the registry key specified in kb5004442 is still set appropriately?
In any case, our official recommendation is to use Identity Collector (mentioned in the SK you linked).

View solution in original post

0 Kudos
George_Casper
Contributor

Same here, no signs of trouble with DCOM hardening.  Though may be related to DCOM or anything for that matter, the symptoms don't match at all in any way shape or form to either Microsoft's or Checkpoint's DCOM Hardening guidance.   Spent hours on the phone with TAC on Friday trying to troubleshoot it.    I'm sure someone will figure it out and create a hotfix or something else, but in the mean time just install Identity Collector and your problem will be solved in 20 minutes and it works.   While nice to have a dedicated VM, you can stand it up on most any VM in a pinch and then move it to a dedicated VM later.   It's surprisingly simple and not sure why I had a mental block about doing it 6 months ago but should have.   Below is an unofficial quick cheat sheet on the install steps.  Its a 33MB installer that only takes a minute to install and about 20 to configure.  You can do some fancy stuff in there later on, looks like there's integration with Cisco ICE, Aruba and some other stuff.  Will have fun sometime later with that but out of the box vanilla config is quick and easy to replace AD Query.

 

On the gateway object in smart console select Identity Awareness. Then select Identity Collector check box. Hit the green + arrow and add a host object with the IP of the machine that will have IDC installed on it. It will generate a shared secret, save this secret. Install policy.

Install IDC from this sk, sk134312. After it is installed click the * (blue star for new object). Create a Domain, name it whatever you want. The Identity Collector requires an AD user that belongs to the default Event Log Readers group. Add that user and click test, then after a success click okay. 

Click the blue star * again. Active Directory > Fetch Automatically > Provide the DC IP. Click Fetch > OK.

Next create a query pool (in the top left). Give the query pool a name and click the check box in the top left to select the Domain Controller.

Next create a Gateway object. I would name it the same name in Smart Console. Put in the IP address of the gateway in Smart Console. Apply that saved shared secret, add the query pool. Click test then trust. Click OK.

After you confirm it's all connected uncheck Active Directory Query on your Gateway object in Smart Console. Install policy.

On the CLI on your gateway you can verify connectivity with this command.

# pdp connections idc

P.S.  If you have a domain controller that identity connector won't connect to and you can ping it and looks good otherwise, on the DC itself, check Windows Firewall & make sure to Allow a Program Through the Firewall "Remote Event Log Management" and Domain network is Checked On

 

View solution in original post

Liel_Shaish
Employee
Employee

Hi,

Thanks for the sharing this experience. 
After investigating the issue together with Microsoft, its related to a security hardening Microsoft had introduced in the October 2022 update.
As part of the hardening (not the DCOM which is described in sk176148), they changed the read privileges that affect the GW query to the DC.
In case ADQuery is configured with an admin user, there is no issue. but in case ADQuery is configured with a non admin user (sk93938) the query will fail with WMI error. We are looking on a way to adjust the default query to work in all cases.

Current suggestion is to change the query to the reduced query (sk104900).
**please note the reduced query will not read security events on specific DC which are forwarded from other DCs.
Identity Collector is not affected by this update.

Thanks,
Liel Shaish
Group Manager, Identity Awareness R&D

View solution in original post

33 Replies
George_Casper
Contributor

Similar situation.  We have the proper Jumbo installed since June to address compatibility with DCOM hardening fully enabled and working fine until Windows AD controllers updated with October patches.  Not seeing the tell tale event 10036 messages in AD event viewer that would be associated with the DCOM hardening issue.  Rechecked WMI permissions and don't see any issues, just getting WMI Permission Denied on the gateways.   Opened a TAC case and waiting for a response.  

0 Kudos
PhoneBoy
Admin
Admin

After applying this month's patches, did you check to see if the registry key specified in kb5004442 is still set appropriately?
In any case, our official recommendation is to use Identity Collector (mentioned in the SK you linked).

0 Kudos
George_Casper
Contributor

Identity Collector is the way to go.   Just fixed one site with it, working on the rest.   Unofficial from TAC, delete if not appropriate to post here and accept my apologies in advance:  TAC found a commonality with a few of us that called in about the same thing, a ported hotfix on top of r80.40 jumbo 158 (bug introduced in Jumbo 156 that caused VPN timeouts after 2 hours sk178891). This SK also affects r81 & r81.10.  Seems it is a combination problem with the hotfix & the Microsoft October patches.    Interesting that this hotfix is now incorporated in Jumbo 180 so wondering if it could haunt those that are still using AD query when it comes time to install Jumbo 180.

0 Kudos
nzmatto1
Contributor

Yes, as per George_Casper, we have the registry changes and hardening done, my experience is identical to his. I am now experiencing this at two sites. I expect other people will come across this as the patch cycle works through.

0 Kudos
George_Casper
Contributor

P.S.  The identity collector was a breeze to install/config. Wish I did it months ago.

0 Kudos
(2)
FTZ
Explorer

Identity Collector is a great solution for Gaia firewalls, But what is the solution for the SMB Gaia embedded firewall? Identity collector is not supported. We tested it anyway, but remote access didn't work after we activated Identity collector on the gateway.

(1)
PhoneBoy
Admin
Admin

SMB appliances running R80.20.35 and above managed with R81.20 management will support Identity Collector.
I believe a future version of R81.10.x will support Identity Collector for locally managed gateways.

0 Kudos
neo3313
Explorer

Rolling back the patches on the AD server has fixed the issue.

0 Kudos
BG
Participant

Same situation here. R80.40 gateways with jumbo 158(no other hotfixes are installed as mentioned before). After october updates connection to domain controllers are broken with the message "WMI permission error [ntstatus = 0x80041003]". Uninstalling the update solves the problem.

0 Kudos
G_W_Albrecht
Legend
Legend
0 Kudos
BG
Participant

Applied windows patch was not KB5004442 so I believe the problem is not related described in sk176148. Applied patch was KB5018411 which was released this month for windows server 2016. Also I don't see any events with ID 10036 or "bad credentials or firewall blocks DCOM traffic [ntstatus = 0xc0000022]" error message using adlog tool. BTW uninstalling patch KB5018411 solves the issue.

George_Casper
Contributor

Same here, no signs of trouble with DCOM hardening.  Though may be related to DCOM or anything for that matter, the symptoms don't match at all in any way shape or form to either Microsoft's or Checkpoint's DCOM Hardening guidance.   Spent hours on the phone with TAC on Friday trying to troubleshoot it.    I'm sure someone will figure it out and create a hotfix or something else, but in the mean time just install Identity Collector and your problem will be solved in 20 minutes and it works.   While nice to have a dedicated VM, you can stand it up on most any VM in a pinch and then move it to a dedicated VM later.   It's surprisingly simple and not sure why I had a mental block about doing it 6 months ago but should have.   Below is an unofficial quick cheat sheet on the install steps.  Its a 33MB installer that only takes a minute to install and about 20 to configure.  You can do some fancy stuff in there later on, looks like there's integration with Cisco ICE, Aruba and some other stuff.  Will have fun sometime later with that but out of the box vanilla config is quick and easy to replace AD Query.

 

On the gateway object in smart console select Identity Awareness. Then select Identity Collector check box. Hit the green + arrow and add a host object with the IP of the machine that will have IDC installed on it. It will generate a shared secret, save this secret. Install policy.

Install IDC from this sk, sk134312. After it is installed click the * (blue star for new object). Create a Domain, name it whatever you want. The Identity Collector requires an AD user that belongs to the default Event Log Readers group. Add that user and click test, then after a success click okay. 

Click the blue star * again. Active Directory > Fetch Automatically > Provide the DC IP. Click Fetch > OK.

Next create a query pool (in the top left). Give the query pool a name and click the check box in the top left to select the Domain Controller.

Next create a Gateway object. I would name it the same name in Smart Console. Put in the IP address of the gateway in Smart Console. Apply that saved shared secret, add the query pool. Click test then trust. Click OK.

After you confirm it's all connected uncheck Active Directory Query on your Gateway object in Smart Console. Install policy.

On the CLI on your gateway you can verify connectivity with this command.

# pdp connections idc

P.S.  If you have a domain controller that identity connector won't connect to and you can ping it and looks good otherwise, on the DC itself, check Windows Firewall & make sure to Allow a Program Through the Firewall "Remote Event Log Management" and Domain network is Checked On

 

BG
Participant

Thank you very much for the installation&configuration steps. I also didn't want to use Identity Collector as it means two more agents, two more vms, two more things to maintain. But since Check Point recommends it and now this problem I have no choice 😅

0 Kudos
Greg_Harbers
Collaborator

The Identity collector Technical overview recommends a 16 Gb of RAM and 12 core machines, and a maximum of 35 DCs per collector. In additon, to achive resiliency the recommendation is to have two separate windows machines configured identically.

One of our clients has in the order of 200+ DC. This would mean around  12 windows machines are needed to support this environment.

What has the real world experience been? what spec Windows machines are people using and how many DCs per collector have people configured? are people running the collector directly on DCs?

Thanks

0 Kudos
PhoneBoy
Admin
Admin

I haven't heard of anyone running Identity Collector on their AD server, FWIW.

R81.20 has some pretty significant changes "under the hood" that will improve scalability and resiliency.
If I understand correctly, it should reduce the number of needed Identity Collector instances.

0 Kudos
Greg_Harbers
Collaborator

Hi Dameon,

Thanks for the reply but you dont really answer the question. What is the really world experience with the resouce requirements for the identity collector? and will we need 10 plus instances of the collector to support an environment of 200+ Domain controllers if resilency is required? An additonal question, what, if any, thought has Check Point put into managing the collector versions? is there any ability to maintain the installed versions of the collector?

Thanks

 

0 Kudos
PhoneBoy
Admin
Admin

You're correct in your assumptions you'll need that many Identity Collector instances.
My understanding is that real world experience bears these limits out.

To clarify my statements around R81.20, these two bullets from the upcoming release tell the tale:

  • Improved resiliency, scalability, and stability for PDPs and Identity Brokers. Additional threads handle authentication and authorization flows.
  • During a PDP failure, a PEP Identity Awareness Gateway can recover its identity database from connected PDP Gateways.

This should translate into needing less Identity Collector instances.

Not sure what you mean by "managing the collector versions."
We always have the latest one here and provide a change log for past versions: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 
Earlier versions can likely be obtained from the TAC if necessary.

0 Kudos
Greg_Harbers
Collaborator

Once again, thanks for the reply. 

to expand on my question re managing collector versions, I mean that should we have 10 + servers out there running the Identity collector, when a new version is released, is there a centralised method by which we can update all 10 servers? I am not sure how often new versions are released, but looking at the version history table, it appears it is now on the 15th release.

at the bottom of sk108235, there is a link to sk178086 - "How to Upgrade Indentity Collector version", clicking on this link I get "Sorry, this solution is deleted and can only be viewed by Check Point employees", What is in this article that CP do not want us to know?

0 Kudos
PhoneBoy
Admin
Admin

A centralized method to upgrade the servers?
Not that I'm aware of, but maybe @Royi_Priov (or one of his team) can comment on that as well as the upgrade procedure, which may be different than what is documented in the deleted sk.

0 Kudos
jb1
Contributor
Contributor

Hello,

 

AFAIK in sk176148 it says:

 

To apply the Microsoft hardening and continue using AD Query and Identity Logging, you must install a hotfix.

 

The hotfix is included in Jumbo Hotfix Accumulators for these supported versions of Security Gateways / Security Management / Multi-Domain Servers:

 

I'm moving all of my customers to IC, but on some accounts, where this can't be done right away, I applied the JHF and it solved the issue. This was on R81.10 and R80.40 GWs. And yes, if you are in doubt, if this is the time to star using IC the answer is definitely yes - 15min and you are set. However Can CP please clarify the statement regarding JHF.

 

Br J

 

 

0 Kudos
Greg_Harbers
Collaborator

Hi Dameon, 

Has, or will, Checkpoint officially announced that ADQuery is dead and is not longer supported? 

Thanks

Greg

0 Kudos
_Val_
Admin
Admin

Why should we? It is Microsoft who is hardening the server side. 

Identity Collector is the recommended way, however, ADQuery will be supported to all the customers using it, until the very last one.

0 Kudos
Greg_Harbers
Collaborator

Given that TAC are floundering with the issue that we are seeing across multiple clients and others posting to this topic are also experiencing, while it is easy to blame Microsoft we can't tell our customer not to patch their domain controllers.
Yes, I agree that Identity collector may be the way foward but for customers with many sites, many firewalls and many domain controllers, it is not as simple as loading IC onto a server and job done. This needs a design, planning, testing etc. 

Therefore, the question is, does Check Point acknowledge that after the installation of the October 2022 windows patches, ADQuery is no longer functional and is anyone assigned with coming up with a fix?

0 Kudos
_Val_
Admin
Admin

Nobody "blames" Microsoft, there are security reasons to do the hardening. 

The acknowledgment of ADQuery becoming problematic after MS patching is done in November 2021, together with guidance to the customers on how to deal with it. Please look into sk176148, which is already mentioned above. The issue was also discussed in the community multiple times.

Hope this helps. If not, please let me know about any further clarification you may require.

0 Kudos
Greg_Harbers
Collaborator

Hi Val,

To confirm a few points which I think others who have posted here have also said the same thing...

We do not believe this is  the same issue as described in sk176148 for the following reasons:

  • The gateway devices are all running the required JHF versions or SMB Firmware versions
  • After the release and installation of KB5004442 on the domain controllers in June 2022, ADQuery was operating correctly.
  • As described in sk176148, the response from a gateway that is not running the required JHF or firmware version is “bad credentials or firewall blocks DCOM traffic [ntstatus = 0xc0000022]”. The error we are seeing is “WMI permissions error”. This started after the installation of the October 2022 Windows Server 2019 Cumulative update (KB5018419)
  • By making the IA service account a domain admin, the gateway is able to connect to the domain controller.
  • All of our customers that are using a service account that is a member of domain admins are unaffected, Those customer who are using a service account with the permissions as defined in sk93938 are affected.
  • By adding or removing the account from domain admins and running the command "adlog a control reconf" we can clear or get the wmi permissions error.

based on this, is Check Point open to exploring if this can be overcome by granting additional permissions to the service account, or must it now be a domain admin to operate?

Thanks

PhoneBoy
Admin
Admin

Our official recommendation is to use Identity Collector and has been since Microsoft started tightening down WMI in response to CVE-2021-26414.
That is documented here: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Having said that, there is no official announcement about the death of ADQuery.
As Microsoft continues to lock down WMI, I would expect further issues to occur.

0 Kudos
David_Evans
Contributor

We have been given this as a possible fix from Microsoft.    We are just starting testing.

KB5020439



https://support.microsoft.com/en-us/topic/october-18-2022-kb5020439-os-build-14393-5429-out-of-band-...

David_Evans
Contributor

This had no affect on our WMI issue.

0 Kudos
nzmatto1
Contributor

Same here. We deployed KB5020439 on one DC, and tested the access, still getting the same failure. I'm also trying to push the team forward with an Identity Collector instead. 

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events