Management Backup / Restore failes

[Masek]

I try to get a R82 Management from one VM infrastructure to another.

What did I do:

1. Updated my management to R82 (backup/restore using R81.20 because I couldn't get the version aligned)
2. Checked everything is working after the update
3. Created a backup and exported it
4. Shut down the old management
5. Setup a new clean R82 installation
6. Imported the backup (without errors)

Backup succeed, restore succeeds, but the management doesn't come up.

HCP reports two relevant errors:

1. This is my primary suspect:
SIC Error

2. There is a weird one:

Download-Server unreachable

The suspect to be the SIC is reinforced by the following debug in "cpm.elg":

com.checkpoint.infrastructure.utils.runtime.CpAssertionError: Uncaught exception org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'wsPublisher' defined in class path resource [com/checkpoint/management/web_services/internal/ws-internal-config.xml]: Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.cxf.transport.http_jetty.JettyHTTPServerEngineFactory': Cannot create inner bean 'httpj:engine#4b4fa9c2' of type [org.apache.cxf.transport.http_jetty.spring.JettyHTTPServerEngineBeanDefinitionParser$SpringJettyHTTPServerEngine] while setting bean property 'enginesList' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'httpj:engine#4b4fa9c2': Cannot create inner bean '(inner bean)#1795c09a' of type [org.apache.cxf.configuration.jsse.TLSServerParametersConfig] while setting bean property 'tlsServerParameters'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#1795c09a': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.cxf.configuration.jsse.TLSServerParametersConfig]: Constructor threw exception; nested exception is com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file in thread Thread[main,5,main]
	at com.checkpoint.infrastructure.utils.runtime.CpAssert$DefaultAssertionErrorCreator.createAssertionError(CpAssert.java:2)
	at com.checkpoint.infrastructure.utils.runtime.CpAssert.doFail(CpAssert.java:47)
	at com.checkpoint.infrastructure.utils.runtime.CpAssert.fail(CpAssert.java:53)
	at com.checkpoint.management.dleserver.internal.DefaultExceptionHandler.uncaughtException(DefaultExceptionHandler.java:8)
	at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:863)
	at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:861)
	at java.lang.Thread.uncaughtException(Thread.java:1353)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'wsPublisher' defined in class path resource [com/checkpoint/management/web_services/internal/ws-internal-config.xml]: Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.cxf.transport.http_jetty.JettyHTTPServerEngineFactory': Cannot create inner bean 'httpj:engine#4b4fa9c2' of type [org.apache.cxf.transport.http_jetty.spring.JettyHTTPServerEngineBeanDefinitionParser$SpringJettyHTTPServerEngine] while setting bean property 'enginesList' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'httpj:engine#4b4fa9c2': Cannot create inner bean '(inner bean)#1795c09a' of type [org.apache.cxf.configuration.jsse.TLSServerParametersConfig] while setting bean property 'tlsServerParameters'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#1795c09a': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.cxf.configuration.jsse.TLSServerParametersConfig]: Constructor threw exception; nested exception is com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1804)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:144)
	at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:95)
	at com.checkpoint.infrastructure.spring.IgnoringDuplicateBeansClassPathXmlApplicationContext.<init>(IgnoringDuplicateBeansClassPathXmlApplicationContext.java:1)
	at com.checkpoint.management.cpm.Cpm.initSpringContext(Cpm.java:91)
	at com.checkpoint.management.cpm.Cpm.main(Cpm.java:187)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.cxf.transport.http_jetty.JettyHTTPServerEngineFactory': Cannot create inner bean 'httpj:engine#4b4fa9c2' of type [org.apache.cxf.transport.http_jetty.spring.JettyHTTPServerEngineBeanDefinitionParser$SpringJettyHTTPServerEngine] while setting bean property 'enginesList' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'httpj:engine#4b4fa9c2': Cannot create inner bean '(inner bean)#1795c09a' of type [org.apache.cxf.configuration.jsse.TLSServerParametersConfig] while setting bean property 'tlsServerParameters'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#1795c09a': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.cxf.configuration.jsse.TLSServerParametersConfig]: Constructor threw exception; nested exception is com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:389)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:127)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:428)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:173)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1707)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1452)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1160)
	at org.apache.cxf.bus.spring.SpringBeanLocator.getBeanOfType(SpringBeanLocator.java:123)
	at org.apache.cxf.bus.extension.ExtensionManagerBus.getExtension(ExtensionManagerBus.java:215)
	at org.apache.cxf.transport.http_jetty.JettyDestinationFactory.createDestination(JettyDestinationFactory.java:36)
	at org.apache.cxf.transport.http.HTTPTransportFactory.getDestination(HTTPTransportFactory.java:278)
	at org.apache.cxf.binding.soap.SoapTransportFactory.getDestination(SoapTransportFactory.java:135)
	at org.apache.cxf.endpoint.ServerImpl.initDestination(ServerImpl.java:85)
	at org.apache.cxf.endpoint.ServerImpl.<init>(ServerImpl.java:64)
	at org.apache.cxf.frontend.ServerFactoryBean.create(ServerFactoryBean.java:182)
	at org.apache.cxf.jaxws.JaxWsServerFactoryBean.create(JaxWsServerFactoryBean.java:211)
	at com.checkpoint.management.web_services.internal.WsPublisher.init_aroundBody0(WsPublisher.java:105)
	at com.checkpoint.management.web_services.internal.WsPublisher$AjcClosure1.run(WsPublisher.java:1)
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
	at com.checkpoint.management.ngm_mgmt_aspects.PerformanceTestAspect.aroundPerformanceTest(PerformanceTestAspect.java:33)
	at com.checkpoint.management.web_services.internal.WsPublisher.init(WsPublisher.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1930)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1872)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
	... 14 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'httpj:engine#4b4fa9c2': Cannot create inner bean '(inner bean)#1795c09a' of type [org.apache.cxf.configuration.jsse.TLSServerParametersConfig] while setting bean property 'tlsServerParameters'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#1795c09a': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.cxf.configuration.jsse.TLSServerParametersConfig]: Constructor threw exception; nested exception is com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:389)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:134)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1707)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1452)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:374)
	... 47 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#1795c09a': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.cxf.configuration.jsse.TLSServerParametersConfig]: Constructor threw exception; nested exception is com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:374)
	... 53 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.cxf.configuration.jsse.TLSServerParametersConfig]: Constructor threw exception; nested exception is com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:224)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311)
	... 59 more
Caused by: com.checkpoint.infrastructure.utils.runtime.CpAssertionError: failed to load SIC cert file
	at com.checkpoint.infrastructure.utils.runtime.CpAssert$DefaultAssertionErrorCreator.createAssertionError(CpAssert.java:2)
	at com.checkpoint.infrastructure.utils.runtime.CpAssert.doFail(CpAssert.java:47)
	at com.checkpoint.infrastructure.utils.runtime.CpAssert.fail(CpAssert.java:53)
	at com.checkpoint.management.web_services.internal.sic.SicCertManager.loadSicCertKeyStore(SicCertManager.java:85)
	at com.checkpoint.management.web_services.internal.sic.SicCertManager.getSicCertKeyStore(SicCertManager.java:42)
	at com.checkpoint.management.web_services.internal.sic.SicKeyManagerFactorySpi.engineInit(SicKeyManagerFactorySpi.java:10)
	at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:23)
	at org.apache.cxf.configuration.jsse.TLSParameterJaxBUtils.getKeyManagers(TLSParameterJaxBUtils.java:296)
	at org.apache.cxf.configuration.jsse.TLSServerParametersConfig.<init>(TLSServerParametersConfig.java:77)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:83)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:437)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211)
	... 61 more
Caused by: java.io.IOException: Integrity check failed: java.security.UnrecoverableKeyException: Failed PKCS12 integrity checking
	at com.ibm.crypto.provider.PKCS12KeyStoreOracle.engineLoad(Unknown Source)
	at java.security.KeyStore.load(KeyStore.java:1460)
	at com.checkpoint.management.web_services.internal.sic.SicCertManager.loadSicCertKeyStore(SicCertManager.java:46)
	... 71 more
Caused by: java.security.UnrecoverableKeyException: Failed PKCS12 integrity checking

[PhoneBoy · Accepted solution]

migrate_server should be used instead.
See: https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_Installation_and_Upgrade_Guide/Con... 

[Masek]

Will try that one

But what worries me: I used this to evaluate my Recovery procedures and failed.

[Masek]

Should I make my backup via migrate_server export in the future?

[the_rock]

I would follow below sk, it gives best options. 

Andy

https://support.checkpoint.com/results/sk/sk108902

[e1pex]

@Masek Did you choose to continue on PhoneBoy's recommendation to use migrate_server or have you continued to try to solve the restore issue?

Since I'm actually doing a recovery validation myself I believe we are facing the same issue. I get similar errors in the cpm.elg log but on a R81.20 backup/restore operation to a newly deployed vm running R81.20 with matching JHF number.

I've managed to narrow it down to be some issue causing "$CPDIR/registry/HKLM_registry.data" not being restored correctly which leads to the system trying to load the wrong SIC-file (among other things).

If you have access to the problematic box a "less $CPDIR/registry/HKLM_registry.data | grep MySICName" will probably return the name of the new/temporary box SIC and not the one you are trying to restore which can be viewed with "keytool -list -v -keystore $CPDIR/conf/sic_cert.p12 | grep Owner:" on the same test box.

The really odd thing is that the issue causing the problem seems to be intermittent, since I can redo the restore with the same backup file to the same VM (rolled back to a clean state of course) and sometimes get it to work with like a 50/50 chance of success.

This is how far I've gotten in my troubleshooting and I believe the next step will be to log a case with the support to take this futher.

[Masek]

The approach with migrate_server worked. I am thinking of using migrate_server for backup purposes in the future.

I have some ideas what has gone wrong. My primary suspect is that I used a different hostname. When using migrate_server, I prepared the destination system more carefully (e.g. copying the configuration via clish before importing).

I have ideas on how to improve the process. If I have the time to do it, I'll post a followup here.

[the_rock]

I believe when you do migrate_server, IP and hostname would stay unchanged, but the license would be tied to an "old" server IP address.

Andy

[the_rock]

I see what @PhoneBoy is saying and I agree, BUT, here is what I would try for now, lets see if we can help you fix this in broken state. So, commands I would verify:

cpwd_admin list

api status

$FWDIR/scripts/./cpm_status.sh

At the end of the day, if api does NOT start or come up even after cprestart or reboot, and we cant figure out why, it might be "toast", sorry : - (

Andy

[Masek]

cpm_status.sh said something like "failed to start". The system doesn't exist any more, so I cannot give the output of the other two.

If the next attempt fails, I'll run those commands.

[the_rock]

You need to know where you are going my friend, hehe : - )

You are going towards making this work and we can help!

Just follow migrate_server that @PhoneBoy mentioned. Here are some things to remember about it, as people may forget this...when it comes th that process, IP and hostname do NOT change, but license does, as it would be tied to the license on your old server, not new one.

Hope that helps.

Btw, if api said failed to start, yea, smart console would never load without it. If that happens again (hope not), I would also tru those other commands as well. I am not aware of any process to debug API, but will check.

Andy

[JozkoMrkvicka]

Do you have proper license on freshly installed R82 ? Was IP changed between new and old MGMT ?

[Masek]

The license was installed (checked that right away).

The IP address was not changed.

[GHaider]

for a "complete" backup solution i would recommend ALL of the following:

• normal GAIA backup
• migrate_server export
• exported snapshot

and i really mean every 3...  did a restore-test on R80.40 endpoint server a while back and ran into https://support.checkpoint.com/results/sk/sk168062  (endpoint db missing in gaia backup) ...so since that i always have all 3 backups 🙂

[Masek]

I luckily do not have an endpoint management on my Check Point installation (Management Server plus  3600)  at home 😀. But it is a bit frustrating that backup/restore is a challenge even for such a trivial installation as mine. If you skip the logs, my backup should fit into 100KB. A snapshot export is 300GB.