R82 Mgmt Server Updated to HF-60 Cannot Renew Cluster/Gateway IPSEC Certs

[D_Riddleberger]

Error received is 'Failed Creating Certificate. General Problem in Certificate Authority'

Confirmed same for multiple clusters and/or gateways and already tried a cpstop/start and reboot on the Mgmt VM.

<attached>

 

[Vincent_Bacher]

Did not see that because not yet upgraded to R82.10 but first step I would do to check the ICA status using 

cpstat ca -f all

[D_Riddleberger]

 

Thanks Vincent_Bacher. This cpstat ca -f all output does not look normal as there are not any Valid Certs or SIC Certs listed. Also this is R82 and not R82.10

[Vincent_Bacher]

Sorry my bad, R82 of course. 
We did not face this issue.

 At this point I would prefer creating a tac sr before researching and maybe trying to recreate the ica without confirmation or proposal by tac.

[genisis__]

Have you rolled back HFA60? and then tried?  Did you take a snapshot prior to installing HFA60?  I'm using R82 with HFA44, and have not experienced any issues like this.

Whats lead to this?  I feel we missing some historic information here.  If this has just happened after updating to Jumbo HFA60 then logically restoring your snapshot or backup should get you back to a working system, fast.  If you don't have these then you are going down the TAC road and hope they can resolve this, or you may have to potentially recreate the ICA and renew all the certs (hope it does not come to that!).

Also see below for the ICA tool, but be careful - take a snapshot and backup before doing anything.  I would certainly suggest raising a TAC case.
The ICA Management Tool

[JozkoMrkvicka]

Isnt internal_ca certificate on management somehow corrupted or expired ?

[Vincent_Bacher]

If I remember right there is a web based tool for ica management and there is something in the admin guide.

But I haven't dared to try it yet, even if it's just to look. When CA comes into the z I'm too cautious and would definitely create an SR and ask Tac for help.

[JozkoMrkvicka]

The ICA Management Tool is that one. I never used it, as setting it up using user's certificate is too complex.

You can still use SmartConsole to check status of internal_ca. Go to the affected domain, open Object Explorer -> Servers -> Trusted CA -> double click on "internal_ca" -> Local Security Management Server -> View...

[Vincent_Bacher]

Thanks, too long ago i had the need for checking that.
But in this case i would not touch anything without CP assistance to be honest.

[the_rock]

Hey Dan,

Honestly, here is the best advice/suggestion I can give. Since this is the management, PLEASE be careful what you do. I recommend generation snapshot (if you can) before anything. 

Since renewal is failing, you can try cprestart or rbeooting the mgmt and test again. What does cpstat mg show?

[genisis__]

We need to understand if this all started happening since going to JHFA60.  If so then the question we also need to know is if backups and snapshots where take prior to implementing the new Jumbo. 
If so then the correct course of action at this point would be snapshot restore or restore the backup.
If this was not taken then TAC need to be involved to determine the next steps, and the ICA tool can be used to at least look at the ICA, but would certainly feel more comfortable if TAC where leading that investigation.

[D_Riddleberger]

 

 

Thanks everyone. Here are the updates

1) The problem was identified a couple of weeks after HF-60 update, previous version was HF-44 

2) Environment has 500+ CP sites, and there are multiple policy changes every day, so a Snapshot revert is not an option at this time. But 'yes' we have the HF-44 Snapshot

3) We were able to lab this up and IPSEC Cert renewals appear to be working in a lab with HF-60. So, it is unclear if HF-60 is the actual problem or there is some other ''unknown' problem and/or corruption with the ICA

4) Updated cpstat mg and cpstat ca -f all (attached)

5) A TAC case has been opened, more to come

[the_rock]

@D_Riddleberger 

My lab output, for the context:

[Expert@CP-MANAGEMENT:0]# cpstat mg

Product Name: Check Point Security Management Server
Major version: 6
Minor version: 0
Build number: 998000009
Is started: 1
Active status: active
ICA status: 0
Status: The Internal Certificate Authority (ICA) certificate is valid until Jan 19 03:14:07 2038 GMT

Connected clients
----------------------------------------------------
|Client type |Administrator|Host |Database lock|
----------------------------------------------------
|SmartConsole|admin |EVE-WIN11|false |
----------------------------------------------------

[Expert@CP-MANAGEMENT:0]#

[D_Riddleberger]

Thanks Andy, I just updated my prev post with updated screenshots of the cpstat mg and cpstat ca -f all. Still unclear if this was caused by the HF-60 update or other unknown ICA corruption.

[the_rock]

As precaution, I would definitely open TAC case. Now, if you can uninstall jumbo 60 and reboot, might be a good way to see if that jumbo take caused an issue or not. ABSOLUTELY do generate at least backup before trying that and also snapshot, if possible.

[D_Riddleberger]

2/10/26 (13:30 EST)

Andy and All,

I was able to recover a migrate export in our lab, update to HF-60 and successfully renew 'a given gateway's' IPSEC VPN Cert. This is looking more like ICA corruption vs anything outright with the HF-60 update. More to come....

[the_rock]

Awesome work, Dan.

[Duane_Toler]

I had a different version of the same bug, but in R82 JHF 39.  Updating to JHF-60 (at that time it was the ongoing/latest JHF, not recommended then). JHF 60 fixed that bug.

The bug then was a timeout error in the gen-pki-cert-req function (as noted in the JHF release notes).  If you're on JHF 60 and have this issue, then this is new.  Looks like you worked it out, but I'm sure TAC/R&D would love to get it fixed.

Good luck and keep us updated!

[D_Riddleberger · Accepted solution]

Update 3/12/26

Issue is now fully resolved and has been stable for about two weeks.

The resolution was to update from R82 HF-60 (Recommended) to R82 HF-73 (Latest).

Summarizing that the HF-73 update process itself either corrected or rebuilt whatever was corrupted/non-functioning in the database.

Another piece of the puzzle we figured out was, that we could revoke a given gateway cert manually using cli and then successfully renew the cert in Smart Console. Thus, speculating that it was the underlining 'revoke'' function of the renewal feature in SmartConsole that was failing. 

Example: cli - >cpca_client revoke_cert -n "CN=FW-Gateway-Name VPN Certificate"  

99.9% of this work was performed and learned by trial and error in a lab environment. 

So, if you don't have a lab, build one !

And lastly.... 

"Please keep your arms and legs inside the ride at all times..."

[the_rock]

Excellent work, Dan!

[Duane_Toler]

That is great news!  Glad you were able to get past it!  Thanks for sharing the cpca_client path as well; no doubt that will be helpful to someone else in the future.