R80.20 upgrade - CPM stuck on initilialisation

[Matt_Elkington]

I'm attempting to migrate an existing R77.30 openserver to a new R80.20 openserver.

The upgrade process itself appears to perform correctly and everything is fine up until the initial cpstart, then cpm never comes out of initialisation.

I have done a migrate export using th R80.20 tool and imported into a newly build R80.20, plus I have done a migrate export with the R77.30 tools, imported into a newly built R77.30 then upgraded. 

Both display the same behaviour once they try to start under R80.20, so the behaviour is repeatable, which suggests that there is an issue with the export as opposed to the upgrade process.

Looks like SIC isn't working correctly as cpm.elg has a whole bunch of errors with:

Caused by: com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file

Followed by

Caused by: java.security.cert.CertificateException: Signature algorithm mismatch

Currently this is on a non-production box so I can mess about however much I like as long as I get a procedure that will allow me to migrate the actual platform when I'm ready.

I have a case open with TAC already, but thought this way might glean faster results (as already I've had them tell me that "you need to ensure you used the correct ISO to upgrade", so I feel it's going to be one of those cases...)

R80.20 JHF Take 47

[PhoneBoy]

Just to confirm, you can reproduce this with a migrate export/import?
Are you using real hardware or VMs for testing?
The lack of a SIC cert doesn't sound good, but it also sounds like it should be fixable.

[Matt_Elkington]

I'm currently using VMs for testing, but that's only because the import into the real hardware doesn't work.

Source is R77.30 open server on real hardware.

First destination was R80.20 open server on real hardware, didn't work.

Second destination was R80.20 VM, didn't work.

Third destination was R77.30 VM, this came up with no issues.

Did an inplace CPUse upgrade, verified with no issue, upgrade completed with no errors.  CPM never gets past initialisation.

This is first error that turns up in the cpm.elg

08/04/19 18:13:55,499  INFO export.internal.ExclusionClasses [main]: This is new version
08/04/19 18:13:55,529  INFO web_services.internal.WsPublisher [main]: Publishing web services on <a href="https://0.0.0.0:19009/cpmws/" target="_blank">https://0.0.0.0:19009/cpmws/</a>...
08/04/19 18:13:55,793 ERROR utils.runtime.CpAssert$DefaultAssertionErrorHandler [main]: AssertionError has been caught: failed to load SIC cert file
08/04/19 18:13:55,796 ERROR infrastructure.logging.CpAssertionErrorExceptionLoggerHandler [main]: incident [7f88dde7-410a-4ce5-b034-6b16ce64f4ef]:
failed to load SIC cert file

followed by

08/04/19 18:13:55,797 ERROR infrastructure.logging.DefaultExceptionLoggerHandlerImpl [main]: incident [0629715a-d653-4aeb-9edd-2f4446f0c731]:
java.security.cert.CertificateException: Signature algorithm mismatch

I'm about to go and have a poke, becasue I'm wondering if the ICA certs are too small?  The original management server has been about for a long time, so it's entirely possible the key length is 1024 bit (don;t know if that would affect R77.30, but R80.x seems to be 2048 bit minimum). 

[Jeff_Williams1]

I had a variant of this exact same problem trying to upgrade a non-production management server from R77.30 to R80.10 with very similar behavior. How old is your root cert on your management server, and is it a SHA1 cert? Was your 77.30 management server also an upgrade from an older version? It was determined in my case that I needed to re-sign the root certificate after the upgrade, which required a hotfix and some instructions, then CPM started working.

[Matt_Elkington]

This Is what I am thinking the issue may be as well.

The management server itself has been multiply upgraded over years, customer says probably from as far back as NGX. The root cert is a 1024 bit RSA, so I highly suspect that this has something to do with the issue. (Issue date was 2001!!)

As part of my testing I tried going to R80.10 and got the same issue.

I was trying to do a "fwm sic_reset" on my VM, but I'm running into the old issues with "There are IKE Certificates that were generated by the internal Certificate Authority." except they don't exist, if you do a list of valid IKE certs you get nothing back.

I've checked and there are no valid certs in the ICA for VPN. In fact because this is a test box I have deleted and/or revoked every single cert apart from the ones for the gateway I'm interested in and the managers.

[Jeff_Williams1]

Yeah, at least for R80.10, with the patch they gave me you wouldn't need to do the sic_reset. Might want to ask TAC about other customers having this issue with R80.10, and maybe they can send you the same files and instructions they sent me, then you can upgrade to R80.20 after that, or maybe they can port the fix to R80.20 and the jumbo you're wanting to use and you can go directly to that version. It sounds like you are having my exact problem, they may have just changed the wording of the error in the log.

[Matt_Elkington]

I did a test upgrade to R80.10 and got the same issue. The sic_reset was more of a test to see if I could force it to generate a new root CA cert with 2048 bit signature, and see if it would hen successfully upgrade.

TAC have mentioned that they have found two similar cases, particularly when I mentioned the thought behind the ICA, but they wanted more files (which annoyingly would have already been in the cpinfo they have...) before they'll release the information.

We'll see what they have for me tomorrow morning.

[Matt_Elkington]

Well it looks like the TAC supplied solution is to manually destroy the ICA ad then reset SIC to everything, which is far from ideal...

I think I will pressure them for a better solution, maybe resigning the root cert

[Matt_Elkington]

I don't suppose you could send me the SR number for that case?

TAC gave me a "solution" from R&D whereby I can manually reset the ICA to rebuild it, however this does mean that SIC is totally gone and I will have to reset SIC to all gateways.

If there is a better way to do this than destroying the ICA and recreating I would like to do that.

This does make me think though... If there is no actual mechanism to renew the ICA root cert then that could spell trouble in the future. The ICA Root cert is valid for 20 years, the one I'm working on was done in 2001, so would have expired in 2 years anyway even if I hadn't hit this current problem with the upgrade... I shudder to think how it'll behave when the root certificate that signs everything is no longer valid...