Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
devinspike
Participant
Jump to solution

MDS Upgrade failure R77.30 to R80.40

We are trying to upgrade 2 MDSs from R77.30 to R80.40 in our lab environment.  When we upgrade the primary MDS it will upgrade just fine, but when we try and upgrade the secondary, it will fail after the reboot when trying to import the CMAs.

The CMA that fails seems to be random.  We checked the import_mds log in /opt/CPInstLog/ and every time its a random CMA that fails.  It is always a CMA that should be active on the secondary MDS.

 

We have a lab with 2 MDSs setup, one is the primary and one is a secondary.   Each MDS has 6 CMAs, 3 are active on the primary MDS and 3 are active on the secondary MDS.  These MDSs are also in the same subnet so there should be no issues with them communicating with one another.

We are trying to replicate things we have encountered when upgrading firewalls in production.  Such as adding cluster members that aren't in a cluster, and adding non-ascii characters in the comments for gateways.  One of the CMAs is managing an R75 Splat device, which is not supported by R80.40, we just changed the version in smartdashboard as opposed to upgrading the gateway first, figured this should be fine as the CMA will think it can support it.

We fixed all of these issues before the upgrade process and ensured that all of the verifier tools found no errors.
And we ensured to follow the procedure for upgrading high availability MDSs that are below R80.10, but that still leads to the same error.

 

Anyone have any ideas?

1 Solution

Accepted Solutions
Tal_Paz-Fridman
Employee
Employee

Update:

This was the the relevant error in cpm_for_cpdb:

07/08/20 17:09:34,990 ERROR internal.operation.OperationSvcImpl [qtp-1915696195-258]: caught exception "Tried to persist object ac6e56be-d026-46d7-9c9a-39c941824f98 with domain a0bbbc99-adef-4ef8-bb6d-cebcebcebceb while active domain is 9762a232-094a-4c46-b128-13926594aac3" from class java.lang.IllegalArgumentException

 

It was fixed in R80.40 JHF 38 (PRJ-11517). We suggested they install JHF take 67 (latest GA according to sk165456) on the Primary. Then install a clean secondary MDS with R80.40 + JHF 67 and perform Advanced Upgrade from the R77.30 MDS to use the fix during the import.

 

After applying the procedure issue is resolved and they were able to upgrade and sync the Secondary MDS.

 

Thanks

Tal

View solution in original post

8 Replies
Tal_Paz-Fridman
Employee
Employee

Hi

Can you run da_cli collect_logs and send me the file? I will consult with various R&D owners.

Thanks

Tal

tfridman@checkpoint.com

devinspike
Participant

I have run the requested command to collect the logs on both the Primary that succeeds when upgrading, and the Secondary which fails at upgrading. 

I have marked the files with 'Primary' and 'Secondary'

Tal_Paz-Fridman
Employee
Employee

Thank you. I have forwarded the files and will update with new information.

0 Kudos
devinspike
Participant

I also just wanted to note that in those logs the upgrade is failing on the "R75_Splat_Management" CMA, and we have tried upgrading the IPS database for all of the CMAs prior to upgrading.

John_Fleming
Advisor

a little more info on this. FYI these are pure lab boxes so we didn't want to open a ticket on this to bother support. Everything is in GNS3. We did this to run the upgrade process just to see what happens. 

from the failed CMA on MDS2 I see the follow which does not show up int the upgrade log on MDS2 CMAs that didn't fail.

 

[4193 4065348352]@MDS2[7 Aug 17:09:31] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: Fault returned from remote server: SOAP 1.1 fault: SOAP-ENV:Server[no subcode]
"An internal error has occurred."
Detail:

[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: ERROR: Failed to update 300 objects:
[4193 4065348352]@MDS2[7 Aug 17:09:35] =================LIST START=================
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{92AEF8FE-0C56-461C-8FFA-A7C7204C90A9}', Name: 'IPSSettings'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{493E259B-4687-4D42-AC9F-3FDA209689EE}', Name: 'IPSGlobalEnvironmentSettings'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{8B6939AC-5462-4468-88E2-430DA59D5B2C}', Name: 'PortScanSensitivityLevels'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{742171AF-6E7E-4BA2-B396-C3676C563145}', Name: 'ApplicationEnginesList'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{AB8A5214-EC23-4D9A-B1B8-F529E03717FE}', Name: 'ProductInterSpectCziv'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{F221AA2D-0918-4AA6-AC4F-4107A526524D}', Name: 'ProductConnectra2'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{60967354-2671-4EDA-AF34-784FA62C1672}', Name: 'ProductConnectraNGX'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{20CFCDE9-2952-4CE5-B623-43EA119C8BE6}', Name: 'ProductConnectraNGXR61'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{ADB0F414-D876-4DA1-AD74-7B7C60D55241}', Name: 'ProductInterSpectDalya'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{47E3A957-9B26-4C36-8A3F-1D012840EBDB}', Name: 'APPS'

etc 280ish more lines..

[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{931FC5A5-204E-D148-B476-EE2824C8EA09}', Name: 'A540660CE-5B1D-42BB-A5F8-4D5436547A84'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{CF1193A1-E512-4A43-9B0B-DE2F579EC2B5}', Name: 'ABDFEBAF3-0FCC-4BC5-8B5F-D53A322F3530'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{3C5B477A-C10E-CA47-B45E-F3DBBAABA142}', Name: 'A512259B8-9548-44BD-8946-DE19C701A380'
[4193 4065348352]@MDS2[7 Aug 17:09:35] ==================LIST END==================
2020.08.07_17:09:35 - Failed to update remote server with object 'A512259B8-9548-44BD-8946-DE19C701A380'
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - failed to write object to file
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveObject] ERROR: Failed to update the DB. Object 'A512259B8-9548-44BD-8946-DE19C701A380' (Table 'asm' / Class 'simple_pattern_definition') will not be added to the DB.
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveObject] ERROR: CPMI Error: 0x8003001D (Could not access file for write operation). Update error: An internal error has occurred..
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveTable] Failed to add object 'A512259B8-9548-44BD-8946-DE19C701A380' to table 'asm'.
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...

etc

193 4065348352]@MDS2[7 Aug 17:09:37] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:37] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:37] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'asm'. Saved '1123' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] ERROR: Failed to add 1 objects to table 'asm' in the DB. The resulting DB will be corrupted.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'asm_params'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'asm_params'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'atlas_gateway_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'atlas_gateway_properties'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'atlas_general_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'atlas_general_properties'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'authentication_objects'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'authentication_objects'. Saved '7' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'ce_policies'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CTableMgr::GetTable: no such table ce_policies
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::GetCkpTable] WARNING: Failed to get table 'ce_policies' with error 0x80004005 (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'ce_policies'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CTableMgr::GetTable: no such table ce_policies
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::GetCkpTable] WARNING: Failed to get table 'ce_policies' with error 0x80004005 (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'ce_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'ce_properties'. Saved '0' objects.

more errors..

[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand start ====
[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand fwset header (
:AdminInfo (
:chkpf_uid ("{9D0609EB-B8B3-2548-AE04-C9365C9B12A8}")
:ClassName (replace_uids_command)
)
)

[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand full fwset (
:AdminInfo (
:chkpf_uid ("{9D0609EB-B8B3-2548-AE04-C9365C9B12A8}")
:ClassName (replace_uids_command)
)
)

[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand end ====
[4193 4065348352]@MDS2[7 Aug 17:10:14] [CUpgCPMIInterface::Save] ERROR: Failed to update some of the tables. The resulting DB will be corrupted.
[4193 4065348352]@MDS2[7 Aug 17:10:14] [CUpgradeMgr::PerformCPMIUpdate] WARNING: Failed to save the DB.
[4193 4065348352]@MDS2[7 Aug 17:10:14] CNgmProxyImpl::init: this = 0xd8a98d0
[4193 4065348352]@MDS2[7 Aug 17:10:14] CNgmProxyImpl::init: this = 0xd9e86a0
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CUpgradeMgr::PerformUpgrade] ERROR: Failed to perform cpmi update
[4193 4065348352]@MDS2[7 Aug 17:10:15] [writeUpgradeResult] The path to result file is: '/opt/CPmds-R80.40/customers/Splat_75_Management_Server/CPsuite-R80.40/fw1/log/upgrade_result'
[4193 4065348352]@MDS2[7 Aug 17:10:15] [writeUpgradeResult] Wrote the following to the result file: '80004005'
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] cpdb ended with result '0x80004005' (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 9762a232-094a-4c46-b128-13926594aac3 to post_objects
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 9762a232-094a-4c46-b128-13926594aac3 to finalize_domain
[4193 4065348352]@MDS2[7 Aug 17:10:42] [main] FATAL ERROR: Operation failed.
[4193 4065348352]@MDS2[7 Aug 17:10:42] [UpgradeToRenaissanceInfra::shutdownJavaUpgradeServer] Sending shutdown request to java upgrade server
(END)

John_Fleming
Advisor

BTW this is a in place CPUSE upgrade.

Tal_Paz-Fridman
Employee
Employee

Taking this offline with John and Devin.

Will update when we know the root cause.

Tal

0 Kudos
Tal_Paz-Fridman
Employee
Employee

Update:

This was the the relevant error in cpm_for_cpdb:

07/08/20 17:09:34,990 ERROR internal.operation.OperationSvcImpl [qtp-1915696195-258]: caught exception "Tried to persist object ac6e56be-d026-46d7-9c9a-39c941824f98 with domain a0bbbc99-adef-4ef8-bb6d-cebcebcebceb while active domain is 9762a232-094a-4c46-b128-13926594aac3" from class java.lang.IllegalArgumentException

 

It was fixed in R80.40 JHF 38 (PRJ-11517). We suggested they install JHF take 67 (latest GA according to sk165456) on the Primary. Then install a clean secondary MDS with R80.40 + JHF 67 and perform Advanced Upgrade from the R77.30 MDS to use the fix during the import.

 

After applying the procedure issue is resolved and they were able to upgrade and sync the Secondary MDS.

 

Thanks

Tal

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events