MDS upgrade failing: R77.30 to R80.10

[Jeff_Ladd]

I have been trying to get my MDS upgraded from R77.30 to R80.10. We run through the upgrade in a VmWare environment to test the upgrade prior to upgrading production, so this is still in my test environment.  My configuration is:  R77.30 T286, 4 CMA’s – 1 of which contains a VSX cluster.  Global policy is assigned to two CMA’s, and has only 1 object defined (SmartEvent server), no enrollment in global IPS (IPS is configured on 3 CMA’s just not as part of the global policy).

April timeframe I uploaded a cpinfo to the Checkpoint verification service and they responded with 1 error and a few warnings (“DNS Protocol” used in a group was the error, the warnings referenced renaming services and App Control groups that were changing, LTE Services, OPSEC changing, table.def changes, Threat Prevention permissions profile conflict) – nothing major.   I restored an MDS backup into my test env., fixed what I could from the verification service and successfully upgraded to R80.10 via CPUSE.  Did this multiple times, with multiple different mds backups.  And as work would have it, the production upgrade got delayed.  When I got back to testing the upgrade again, I resubmitted to the verification service and they now told me that my environment was not compatible with R80.10 and they would contact me shortly – this was on May 15^th.  In the mean time I attempted the CPUSE upgrade again and it failed, also tried the mds_export/mds_import upgrade and that failed as well.  From the logs, it looked like the ips import script was failing, but nothing specific.  As I waited on Checkpoint’s verification service response, I SK surfed, found some things - though none seemed to help.

Still no response from the Checkpoint, July 2^nd I uploaded another cpinfo to the verification service and they again responded that my environment is not compatible and they would contact me.  Instead of waiting, I opened a ticket with Checkpoint.  July 5^th the verification service opened a ticket as well,  so now I have two tickets opened.  Since July 5 the verification service ticket has been bounced around with no worthwhile feedback and the newest tech person just finished (Aug 8^th) importing our mds backup into their system and ran the pre-upgrade.

My other support ticket has proved more insightful and there have been a few things suggested/attempted:

- Disable endpoint protection (which is not and never has been enabled)

- More disk space (not an issue)

- Specify a destination for the export different from where the command was run

- Newer export tools than what is in the T462 ISO

- IPS DB corrupt or outdated, to resolve update IPS on R77.30 prior to upgrade

- Use mds_export from ISO and mds_import to a Clean Install of R80.10

- Update IPS DB on all CMA’s, even the ones that are not configured for IPS and also in the Global policy   *** The import got farther this time, but still failed.

- Provided fw1_wrapper_HOTFIX_R80_10_HF_BASE_717_GA_FULL.tgz to install on clean install of R80.10  ***  This failed install verification

- Provided R80.10 Take91 and fw1_wrapper_HOTFIX_R80_10_JHF_T91_304_GA_FULL.tgz to install on clean install of R80.10 prior to import

As the import goes, one CMA is successfully imported then the next three fail.  The CMA that imports successfully is the VSX cluster that also has IPS configured.

Import operation started at: Wed Aug 8 06:59:41 EDT 2018

Multi-Domain Server databases - Success

Import operation for Multi-Domain Server finished at: Wed Aug 8 07:34:15 EDT 2018

Domain Management Server datacenter-mgmt database - Success

Import operation for datacenter-mgmt finished at: Wed Aug 8 07:49:59 EDT 2018

Domain Management Server intfw-mgmt database - Failure

Domain Management Server pci-mgmt database - Failure

Domain Management Server guest-mgmt database – Failure

The migrate log for the failing CMA’s looks like this:

[8 Aug  7:50:28] ...<-- NotCondition::IsConditionHolds

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] Second condition: 1

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] 'and' condition holds

[8 Aug  7:50:28] ..<-- AndCondition::IsConditionHolds

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] Second condition: 1

[8 Aug  7:50:28] [AndCondition::IsConditionHolds] 'and' condition holds

[8 Aug  7:50:28] .<-- AndCondition::IsConditionHolds

[8 Aug  7:50:28] [ConditionalExecutor::exec] Condition holds, executing activity

[8 Aug  7:50:28] .--> CommandRunner::exec

[8 Aug  7:50:28] ..--> UpgradeMacroReplacer::Instance

[8 Aug  7:50:28] ..<-- UpgradeMacroReplacer::Instance

[8 Aug  7:50:28] ..--> CanonicalizePath

[8 Aug  7:50:28] [CanonicalizePath] Canonicalizing path '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/bin/upgrade_phase -d 4c44b3f3-ccf0-ec45-a4b9-f8be01b364ee -s started'

[8 Aug  7:50:28] [CanonicalizePath] Resulting path: '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/bin/upgrade_phase -d 4c44b3f3-ccf0-ec45-a4b9-f8be01b364ee -s started'

[8 Aug  7:50:28] ..<-- CanonicalizePath

[8 Aug  7:50:28] ..--> ExecCommandGetOutput

[8 Aug  7:50:28] [ExecCommandGetOutput] Going to execute command: '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/bin/upgrade_phase -d 4c44b3f3-ccf0-ec45-a4b9-f8be01b364ee -s started'

[8 Aug  7:50:28] [ExecCommandGetOutput] ERR: Command completed with error code 4

[8 Aug  7:50:28] ..<-- ExecCommandGetOutput

[8 Aug  7:50:28] [CommandRunner::exec] Command's output:

-------------------------------------

Failed to upgrade phase

-------------------------------------

[8 Aug  7:50:28] [CommandRunner::exec] ERR: Command execution had failed

[8 Aug  7:50:28] .<-- CommandRunner::exec

[8 Aug  7:50:28] <-- ConditionalExecutor::exec

[8 Aug  7:50:28] [ActivitiesManager::exec] ERR: Activity 'ConditionalExecutor' failed

[8 Aug  7:50:28] [ProgressUpdater::UpdateProgressToGaia] Progress Updated to '56.4103

[8 Aug  7:50:28] [ActivitiesManager::exec] WRN: Activities execution finished with errors

[8 Aug  7:50:28] [ActivitiesManager::exec] WRN: Activities 'ConditionalExecutor' have failed

[8 Aug  7:50:28] [ActivitiesManager::exec] Designated exit code is 1

[8 Aug  7:50:28] --> CleanupManager::Instance

[8 Aug  7:50:28] <-- CleanupManager::Instance

[8 Aug  7:50:28] --> CleanupManager::DoCleanup

[8 Aug  7:50:28] [CleanupManager::DoCleanup] Starting to perform cleanup

[8 Aug  7:50:28] .--> DirCleaner::exec

[8 Aug  7:50:28] [DirCleaner::exec] Going to remove directory '/opt/CPmds-R80/customers/intfw-mgmt/CPsuite-R80/fw1/tmp/migrate/'

[8 Aug  7:50:29] .<-- DirCleaner::exec

[8 Aug  7:50:29] [CleanupManager::DoCleanup] Completed the cleanup

[8 Aug  7:50:29] <-- CleanupManager::DoCleanup

At this point I am looking for options, advice, recommendations - any help would be greatly appreciated.

Thanks in advance.

[Yonatan_Philip]

HI Jeff, 

It sounds as if the overall experience has been rather poor. Not one of our finer moments.

I'm not going to try and make any excuses for two reasons - first off I wasn't part of the process and don't know what did or didn't happen behind the scene, and second, it shouldn't matter. Regardless of if it was or wasn't justified, we should have done better. Period.(re

What I suggest we try to do going forward is this - Contact me directly at (removed). It looks as if you already have an export that you are trying to upgrade. The output you posted doesn't contain the relevant info, but Instead of giving you the run-around and asking for various logs and debugs, I suggest that you upload your database to me directly and I'll test it in my lab, see what's what, do whatever debug is needed on my own and hopefully just come back with a working procedure.

I've already left for the weekend, but I'll set up a secure FTP account first thing next week where you can upload your database.

I know you already uploaded your database for the verification service, but I don't have access to that system and can't just go and take a customer DB without his explicit consent.

If this sounds like a plan, just send me an email, and if you prefer to go with a different route, just let me know either here or privately through my email and we'll figure out how to move forward to resolve the issue.

HTH 

 Yonatan  

[Jeff_Ladd]

Hi Yonatan,

Thank you for your response. I do appreciate the offer and will reach out to you directly.

Jeff

[Vladimir]

Jeff,

Can you tell me if you have ended-up successfully upgrading your environment and what the root causes of the issues you have experienced were?

I am looking a number of similar projects in a near future and would like to know if there are any gotchas I should be looking for.

Thank you,

Vladimir