MDS Upgrade failure R77.30 to R80.40

[devinspike]

We are trying to upgrade 2 MDSs from R77.30 to R80.40 in our lab environment.  When we upgrade the primary MDS it will upgrade just fine, but when we try and upgrade the secondary, it will fail after the reboot when trying to import the CMAs.

The CMA that fails seems to be random.  We checked the import_mds log in /opt/CPInstLog/ and every time its a random CMA that fails.  It is always a CMA that should be active on the secondary MDS.

We have a lab with 2 MDSs setup, one is the primary and one is a secondary.   Each MDS has 6 CMAs, 3 are active on the primary MDS and 3 are active on the secondary MDS.  These MDSs are also in the same subnet so there should be no issues with them communicating with one another.

We are trying to replicate things we have encountered when upgrading firewalls in production.  Such as adding cluster members that aren't in a cluster, and adding non-ascii characters in the comments for gateways.  One of the CMAs is managing an R75 Splat device, which is not supported by R80.40, we just changed the version in smartdashboard as opposed to upgrading the gateway first, figured this should be fine as the CMA will think it can support it.

We fixed all of these issues before the upgrade process and ensured that all of the verifier tools found no errors.
And we ensured to follow the procedure for upgrading high availability MDSs that are below R80.10, but that still leads to the same error.

Anyone have any ideas?

[Tal_Paz-Fridman]

Hi

Can you run da_cli collect_logs and send me the file? I will consult with various R&D owners.

Thanks

Tal

tfridman@checkpoint.com

[devinspike]

I have run the requested command to collect the logs on both the Primary that succeeds when upgrading, and the Secondary which fails at upgrading. 

I have marked the files with 'Primary' and 'Secondary'

[Tal_Paz-Fridman]

Thank you. I have forwarded the files and will update with new information.

[devinspike]

I also just wanted to note that in those logs the upgrade is failing on the "R75_Splat_Management" CMA, and we have tried upgrading the IPS database for all of the CMAs prior to upgrading.

[John_Fleming]

a little more info on this. FYI these are pure lab boxes so we didn't want to open a ticket on this to bother support. Everything is in GNS3. We did this to run the upgrade process just to see what happens. 

from the failed CMA on MDS2 I see the follow which does not show up int the upgrade log on MDS2 CMAs that didn't fail.

[4193 4065348352]@MDS2[7 Aug 17:09:31] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: Fault returned from remote server: SOAP 1.1 fault: SOAP-ENV:Server[no subcode]
"An internal error has occurred."
Detail:

[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: ERROR: Failed to update 300 objects:
[4193 4065348352]@MDS2[7 Aug 17:09:35] =================LIST START=================
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{92AEF8FE-0C56-461C-8FFA-A7C7204C90A9}', Name: 'IPSSettings'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{493E259B-4687-4D42-AC9F-3FDA209689EE}', Name: 'IPSGlobalEnvironmentSettings'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{8B6939AC-5462-4468-88E2-430DA59D5B2C}', Name: 'PortScanSensitivityLevels'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{742171AF-6E7E-4BA2-B396-C3676C563145}', Name: 'ApplicationEnginesList'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{AB8A5214-EC23-4D9A-B1B8-F529E03717FE}', Name: 'ProductInterSpectCziv'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{F221AA2D-0918-4AA6-AC4F-4107A526524D}', Name: 'ProductConnectra2'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{60967354-2671-4EDA-AF34-784FA62C1672}', Name: 'ProductConnectraNGX'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{20CFCDE9-2952-4CE5-B623-43EA119C8BE6}', Name: 'ProductConnectraNGXR61'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{ADB0F414-D876-4DA1-AD74-7B7C60D55241}', Name: 'ProductInterSpectDalya'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{47E3A957-9B26-4C36-8A3F-1D012840EBDB}', Name: 'APPS'

etc 280ish more lines..

[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{931FC5A5-204E-D148-B476-EE2824C8EA09}', Name: 'A540660CE-5B1D-42BB-A5F8-4D5436547A84'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{CF1193A1-E512-4A43-9B0B-DE2F579EC2B5}', Name: 'ABDFEBAF3-0FCC-4BC5-8B5F-D53A322F3530'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{3C5B477A-C10E-CA47-B45E-F3DBBAABA142}', Name: 'A512259B8-9548-44BD-8946-DE19C701A380'
[4193 4065348352]@MDS2[7 Aug 17:09:35] ==================LIST END==================
2020.08.07_17:09:35 - Failed to update remote server with object 'A512259B8-9548-44BD-8946-DE19C701A380'
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - failed to write object to file
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveObject] ERROR: Failed to update the DB. Object 'A512259B8-9548-44BD-8946-DE19C701A380' (Table 'asm' / Class 'simple_pattern_definition') will not be added to the DB.
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveObject] ERROR: CPMI Error: 0x8003001D (Could not access file for write operation). Update error: An internal error has occurred..
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveTable] Failed to add object 'A512259B8-9548-44BD-8946-DE19C701A380' to table 'asm'.
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...

etc

193 4065348352]@MDS2[7 Aug 17:09:37] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:37] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:37] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'asm'. Saved '1123' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] ERROR: Failed to add 1 objects to table 'asm' in the DB. The resulting DB will be corrupted.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'asm_params'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'asm_params'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'atlas_gateway_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'atlas_gateway_properties'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'atlas_general_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'atlas_general_properties'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'authentication_objects'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'authentication_objects'. Saved '7' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'ce_policies'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CTableMgr::GetTable: no such table ce_policies
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::GetCkpTable] WARNING: Failed to get table 'ce_policies' with error 0x80004005 (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'ce_policies'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CTableMgr::GetTable: no such table ce_policies
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::GetCkpTable] WARNING: Failed to get table 'ce_policies' with error 0x80004005 (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'ce_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'ce_properties'. Saved '0' objects.

more errors..

[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand start ====
[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand fwset header (
:AdminInfo (
:chkpf_uid ("{9D0609EB-B8B3-2548-AE04-C9365C9B12A8}")
:ClassName (replace_uids_command)
)
)

[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand full fwset (
:AdminInfo (
:chkpf_uid ("{9D0609EB-B8B3-2548-AE04-C9365C9B12A8}")
:ClassName (replace_uids_command)
)
)

[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand end ====
[4193 4065348352]@MDS2[7 Aug 17:10:14] [CUpgCPMIInterface::Save] ERROR: Failed to update some of the tables. The resulting DB will be corrupted.
[4193 4065348352]@MDS2[7 Aug 17:10:14] [CUpgradeMgr::PerformCPMIUpdate] WARNING: Failed to save the DB.
[4193 4065348352]@MDS2[7 Aug 17:10:14] CNgmProxyImpl::init: this = 0xd8a98d0
[4193 4065348352]@MDS2[7 Aug 17:10:14] CNgmProxyImpl::init: this = 0xd9e86a0
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CUpgradeMgr::PerformUpgrade] ERROR: Failed to perform cpmi update
[4193 4065348352]@MDS2[7 Aug 17:10:15] [writeUpgradeResult] The path to result file is: '/opt/CPmds-R80.40/customers/Splat_75_Management_Server/CPsuite-R80.40/fw1/log/upgrade_result'
[4193 4065348352]@MDS2[7 Aug 17:10:15] [writeUpgradeResult] Wrote the following to the result file: '80004005'
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] cpdb ended with result '0x80004005' (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 9762a232-094a-4c46-b128-13926594aac3 to post_objects
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 9762a232-094a-4c46-b128-13926594aac3 to finalize_domain
[4193 4065348352]@MDS2[7 Aug 17:10:42] [main] FATAL ERROR: Operation failed.
[4193 4065348352]@MDS2[7 Aug 17:10:42] [UpgradeToRenaissanceInfra::shutdownJavaUpgradeServer] Sending shutdown request to java upgrade server
(END)

[John_Fleming]

BTW this is a in place CPUSE upgrade.

[Tal_Paz-Fridman]

Taking this offline with John and Devin.

Will update when we know the root cause.

Tal

[Tal_Paz-Fridman · Accepted solution]

Update:

This was the the relevant error in cpm_for_cpdb:

07/08/20 17:09:34,990 ERROR internal.operation.OperationSvcImpl [qtp-1915696195-258]: caught exception "Tried to persist object ac6e56be-d026-46d7-9c9a-39c941824f98 with domain a0bbbc99-adef-4ef8-bb6d-cebcebcebceb while active domain is 9762a232-094a-4c46-b128-13926594aac3" from class java.lang.IllegalArgumentException

It was fixed in R80.40 JHF 38 (PRJ-11517). We suggested they install JHF take 67 (latest GA according to sk165456) on the Primary. Then install a clean secondary MDS with R80.40 + JHF 67 and perform Advanced Upgrade from the R77.30 MDS to use the fix during the import.

After applying the procedure issue is resolved and they were able to upgrade and sync the Secondary MDS.

Thanks

Tal