a little more info on this. FYI these are pure lab boxes so we didn't want to open a ticket on this to bother support. Everything is in GNS3. We did this to run the upgrade process just to see what happens.
from the failed CMA on MDS2 I see the follow which does not show up int the upgrade log on MDS2 CMAs that didn't fail.
[4193 4065348352]@MDS2[7 Aug 17:09:31] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: Fault returned from remote server: SOAP 1.1 fault: SOAP-ENV:Server[no subcode]
"An internal error has occurred."
Detail:
[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: ERROR: Failed to update 300 objects:
[4193 4065348352]@MDS2[7 Aug 17:09:35] =================LIST START=================
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{92AEF8FE-0C56-461C-8FFA-A7C7204C90A9}', Name: 'IPSSettings'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{493E259B-4687-4D42-AC9F-3FDA209689EE}', Name: 'IPSGlobalEnvironmentSettings'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{8B6939AC-5462-4468-88E2-430DA59D5B2C}', Name: 'PortScanSensitivityLevels'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{742171AF-6E7E-4BA2-B396-C3676C563145}', Name: 'ApplicationEnginesList'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{AB8A5214-EC23-4D9A-B1B8-F529E03717FE}', Name: 'ProductInterSpectCziv'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{F221AA2D-0918-4AA6-AC4F-4107A526524D}', Name: 'ProductConnectra2'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{60967354-2671-4EDA-AF34-784FA62C1672}', Name: 'ProductConnectraNGX'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{20CFCDE9-2952-4CE5-B623-43EA119C8BE6}', Name: 'ProductConnectraNGXR61'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{ADB0F414-D876-4DA1-AD74-7B7C60D55241}', Name: 'ProductInterSpectDalya'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{47E3A957-9B26-4C36-8A3F-1D012840EBDB}', Name: 'APPS'
etc 280ish more lines..
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{931FC5A5-204E-D148-B476-EE2824C8EA09}', Name: 'A540660CE-5B1D-42BB-A5F8-4D5436547A84'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{CF1193A1-E512-4A43-9B0B-DE2F579EC2B5}', Name: 'ABDFEBAF3-0FCC-4BC5-8B5F-D53A322F3530'
[4193 4065348352]@MDS2[7 Aug 17:09:35] Uid: '{3C5B477A-C10E-CA47-B45E-F3DBBAABA142}', Name: 'A512259B8-9548-44BD-8946-DE19C701A380'
[4193 4065348352]@MDS2[7 Aug 17:09:35] ==================LIST END==================
2020.08.07_17:09:35 - Failed to update remote server with object 'A512259B8-9548-44BD-8946-DE19C701A380'
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - failed to write object to file
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveObject] ERROR: Failed to update the DB. Object 'A512259B8-9548-44BD-8946-DE19C701A380' (Table 'asm' / Class 'simple_pattern_definition') will not be added to the DB.
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveObject] ERROR: CPMI Error: 0x8003001D (Could not access file for write operation). Update error: An internal error has occurred..
[4193 4065348352]@MDS2[7 Aug 17:09:35] [CUpgCPMIInterface::SaveTable] Failed to add object 'A512259B8-9548-44BD-8946-DE19C701A380' to table 'asm'.
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:35] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
etc
193 4065348352]@MDS2[7 Aug 17:09:37] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:37] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:37] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'asm'. Saved '1123' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] ERROR: Failed to add 1 objects to table 'asm' in the DB. The resulting DB will be corrupted.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'asm_params'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'asm_params'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'atlas_gateway_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'atlas_gateway_properties'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'atlas_general_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'atlas_general_properties'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'authentication_objects'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[4193 4065348352]@MDS2[7 Aug 17:09:38] CCkpDbObjectsCWsRemoteImpl::Write: session = Vq21v9asmpGs2gJDP150Csl1qJMFWOiP_-gr1LLbSRE
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'authentication_objects'. Saved '7' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'ce_policies'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CTableMgr::GetTable: no such table ce_policies
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::GetCkpTable] WARNING: Failed to get table 'ce_policies' with error 0x80004005 (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'ce_policies'. Saved '0' objects.
[4193 4065348352]@MDS2[7 Aug 17:09:38] CTableMgr::GetTable: no such table ce_policies
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::GetCkpTable] WARNING: Failed to get table 'ce_policies' with error 0x80004005 (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Started saving table 'ce_properties'.
[4193 4065348352]@MDS2[7 Aug 17:09:38] [CUpgCPMIInterface::SaveTable] Ended saving table 'ce_properties'. Saved '0' objects.
more errors..
[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand start ====
[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand fwset header (
:AdminInfo (
:chkpf_uid ("{9D0609EB-B8B3-2548-AE04-C9365C9B12A8}")
:ClassName (replace_uids_command)
)
)
[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand full fwset (
:AdminInfo (
:chkpf_uid ("{9D0609EB-B8B3-2548-AE04-C9365C9B12A8}")
:ClassName (replace_uids_command)
)
)
[4193 4065348352]@MDS2[7 Aug 17:10:09] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand end ====
[4193 4065348352]@MDS2[7 Aug 17:10:14] [CUpgCPMIInterface::Save] ERROR: Failed to update some of the tables. The resulting DB will be corrupted.
[4193 4065348352]@MDS2[7 Aug 17:10:14] [CUpgradeMgr::PerformCPMIUpdate] WARNING: Failed to save the DB.
[4193 4065348352]@MDS2[7 Aug 17:10:14] CNgmProxyImpl::init: this = 0xd8a98d0
[4193 4065348352]@MDS2[7 Aug 17:10:14] CNgmProxyImpl::init: this = 0xd9e86a0
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CUpgradeMgr::PerformUpgrade] ERROR: Failed to perform cpmi update
[4193 4065348352]@MDS2[7 Aug 17:10:15] [writeUpgradeResult] The path to result file is: '/opt/CPmds-R80.40/customers/Splat_75_Management_Server/CPsuite-R80.40/fw1/log/upgrade_result'
[4193 4065348352]@MDS2[7 Aug 17:10:15] [writeUpgradeResult] Wrote the following to the result file: '80004005'
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] cpdb ended with result '0x80004005' (Unspecified error).
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 9762a232-094a-4c46-b128-13926594aac3 to post_objects
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[4193 4065348352]@MDS2[7 Aug 17:10:15] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 9762a232-094a-4c46-b128-13926594aac3 to finalize_domain
[4193 4065348352]@MDS2[7 Aug 17:10:42] [main] FATAL ERROR: Operation failed.
[4193 4065348352]@MDS2[7 Aug 17:10:42] [UpgradeToRenaissanceInfra::shutdownJavaUpgradeServer] Sending shutdown request to java upgrade server
(END)