AnsweredAssumed Answered

MDS upgrade failing: R77.30 to R80.10

Question asked by Jeff Ladd on Aug 9, 2018
Latest reply on Aug 10, 2018 by Jeff Ladd

I have been trying to get my MDS upgraded from R77.30 to R80.10. We run through the upgrade in a VmWare environment to test the upgrade prior to upgrading production, so this is still in my test environment.  My configuration is:  R77.30 T286, 4 CMA’s – 1 of which contains a VSX cluster.  Global policy is assigned to two CMA’s, and has only 1 object defined (SmartEvent server), no enrollment in global IPS (IPS is configured on 3 CMA’s just not as part of the global policy).

 

April timeframe I uploaded a cpinfo to the Checkpoint verification service and they responded with 1 error and a few warnings (“DNS Protocol” used in a group was the error, the warnings referenced renaming services and App Control groups that were changing, LTE Services, OPSEC changing, table.def changes, Threat Prevention permissions profile conflict) – nothing major.   I restored an MDS backup into my test env., fixed what I could from the verification service and successfully upgraded to R80.10 via CPUSE.  Did this multiple times, with multiple different mds backups.  And as work would have it, the production upgrade got delayed.  When I got back to testing the upgrade again, I resubmitted to the verification service and they now told me that my environment was not compatible with R80.10 and they would contact me shortly – this was on May 15th.  In the mean time I attempted the CPUSE upgrade again and it failed, also tried the mds_export/mds_import upgrade and that failed as well.  From the logs, it looked like the ips import script was failing, but nothing specific.  As I waited on Checkpoint’s verification service response, I SK surfed, found some things - though none seemed to help.

 

Still no response from the Checkpoint, July 2nd I uploaded another cpinfo to the verification service and they again responded that my environment is not compatible and they would contact me.  Instead of waiting, I opened a ticket with Checkpoint.  July 5th the verification service opened a ticket as well,  so now I have two tickets opened.  Since July 5 the verification service ticket has been bounced around with no worthwhile feedback and the newest tech person just finished (Aug 8th) importing our mds backup into their system and ran the pre-upgrade.

 

My other support ticket has proved more insightful and there have been a few things suggested/attempted:

- Disable endpoint protection (which is not and never has been enabled)

- More disk space (not an issue)

- Specify a destination for the export different from where the command was run

- Newer export tools than what is in the T462 ISO

- IPS DB corrupt or outdated, to resolve update IPS on R77.30 prior to upgrade

- Use mds_export from ISO and mds_import to a Clean Install of R80.10

- Update IPS DB on all CMA’s, even the ones that are not configured for IPS and also in the Global policy   *** The import got farther this time, but still failed.

- Provided fw1_wrapper_HOTFIX_R80_10_HF_BASE_717_GA_FULL.tgz to install on clean install of R80.10  ***  This failed install verification

- Provided R80.10 Take91 and fw1_wrapper_HOTFIX_R80_10_JHF_T91_304_GA_FULL.tgz to install on clean install of R80.10 prior to import

 

As the import goes, one CMA is successfully imported then the next three fail.  The CMA that imports successfully is the VSX cluster that also has IPS configured.

 

 

Import operation started at: Wed Aug 8 06:59:41 EDT 2018

 

Multi-Domain Server databases - Success

Import operation for Multi-Domain Server finished at: Wed Aug 8 07:34:15 EDT 2018

Domain Management Server datacenter-mgmt database - Success

Import operation for datacenter-mgmt finished at: Wed Aug 8 07:49:59 EDT 2018

Domain Management Server intfw-mgmt database - Failure

Domain Management Server pci-mgmt database - Failure

Domain Management Server guest-mgmt database – Failure

 

The migrate log for the failing CMA’s looks like this:

 

[8 Aug  7:50:28] ...<-- NotCondition::IsConditionHolds

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] Second condition: 1

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] 'and' condition holds

[8 Aug  7:50:28] ..<-- AndCondition::IsConditionHolds

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] Second condition: 1

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] 'and' condition holds

[8 Aug  7:50:28] .<-- AndCondition::IsConditionHolds

[8 Aug  7:50:28] [ConditionalExecutor::exec] Condition holds, executing activity

[8 Aug  7:50:28] .--> CommandRunner::exec

[8 Aug  7:50:28] ..--> UpgradeMacroReplacer::Instance

[8 Aug  7:50:28] ..<-- UpgradeMacroReplacer::Instance

[8 Aug  7:50:28] ..--> CanonicalizePath

[8 Aug  7:50:28] [CanonicalizePath] Canonicalizing path '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/bin/upgrade_phase -d 4c44b3f3-ccf0-ec45-a4b9-f8be01b364ee -s started'

[8 Aug  7:50:28] [CanonicalizePath] Resulting path: '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/bin/upgrade_phase -d 4c44b3f3-ccf0-ec45-a4b9-f8be01b364ee -s started'

[8 Aug  7:50:28] ..<-- CanonicalizePath

[8 Aug  7:50:28] ..--> ExecCommandGetOutput

[8 Aug  7:50:28] [ExecCommandGetOutput] Going to execute command: '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/bin/upgrade_phase -d 4c44b3f3-ccf0-ec45-a4b9-f8be01b364ee -s started'

[8 Aug  7:50:28] [ExecCommandGetOutput] ERR: Command completed with error code 4

[8 Aug  7:50:28] ..<-- ExecCommandGetOutput

[8 Aug  7:50:28] [CommandRunner::exec] Command's output:

-------------------------------------

Failed to upgrade phase

-------------------------------------

[8 Aug  7:50:28] [CommandRunner::exec] ERR: Command execution had failed

[8 Aug  7:50:28] .<-- CommandRunner::exec

[8 Aug  7:50:28] <-- ConditionalExecutor::exec

[8 Aug  7:50:28] [ActivitiesManager::exec] ERR: Activity 'ConditionalExecutor' failed

[8 Aug  7:50:28] [ProgressUpdater::UpdateProgressToGaia] Progress Updated to '56.4103

[8 Aug  7:50:28] [ActivitiesManager::exec] WRN: Activities execution finished with errors

[8 Aug  7:50:28] [ActivitiesManager::exec] WRN: Activities 'ConditionalExecutor' have failed

[8 Aug  7:50:28] [ActivitiesManager::exec] Designated exit code is 1

[8 Aug  7:50:28] --> CleanupManager::Instance

[8 Aug  7:50:28] <-- CleanupManager::Instance

[8 Aug  7:50:28] --> CleanupManager::DoCleanup

[8 Aug  7:50:28] [CleanupManager::DoCleanup] Starting to perform cleanup

[8 Aug  7:50:28] .--> DirCleaner::exec

[8 Aug  7:50:28] [DirCleaner::exec] Going to remove directory '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/tmp/migrate/'

[8 Aug  7:50:29] .<-- DirCleaner::exec

[8 Aug  7:50:29] [CleanupManager::DoCleanup] Completed the cleanup

[8 Aug  7:50:29] <-- CleanupManager::DoCleanup

 

 

 

At this point I am looking for options, advice, recommendations - any help would be greatly appreciated.

 

Thanks in advance.

Outcomes