MDS Upgrade failing from R80.10 to R80.30 (solved)

[HeikoAnkenbrand]

The upgrade of an MDS server hangs at this point for more than 24 hours.

There are also no CMA's created under:

/opt/CPmds-R80.30/customers/

What can you do as next step?
TAC case?

[JozkoMrkvicka]

Hi Heiko,

PUV (Pre-Upgrade Verifier) was run before the upgrade? No errors were found?
Is there enough RAM on that machine?

I faced a similar situation for upgrading R77.30 MDS to R80.x
In my case, it took around 3 hours to complete the upgrade, or revert back to R77.30 latest snapshot (in case of some errors).
During the upgrading, I was also not able to check any CMA using "mdsstat". I was checking status of the upgrade via clish command "show installer package <NUMBER>" where was mentioned path for upgrade log file.

I did tail -f on that file to see if there is any real upgrade progress, together with "top" command.

Here is example for R80.30 Jumbo Take 19 (last line)

[HeikoAnkenbrand]

Hi @JozkoMrkvicka 

Pre-Upgrade Verifier was run before with no errors!

Yes, enough RAM in that machine:
HP DL380 G9 with 64GB (ESX), 2 x CPU with 8 Cores
VM 32GB, 12 Cores

Now comes the classical way:-)
- Create cpinfo
- Restore R80.10 snapshot
- Open TAC case.

[JozkoMrkvicka]

Just for fun, I reverted back my R77.30 LAB MDS and started upgrade to R80.20.

Here is the status after reboot (installation stuck at 58%):

Here is the last line in Installation log so far:

[2019-08-24 - 23:43:31][4829 4829]:Importing MDS configuration to destination.

There is also a detailed log file, located in /var/log/. In my case, the detailed log file is here:
/var/log/install_Major_R80.20_Mgmt_1_detailed.log

Click to Expand

--------------- Installing MDS ---------------
Installation StartedNo Multi-Domain Security Management is detected.

The Multi-Domain Security Management installation includes several
infrastructure packages. These packages will be installed now.

There are no packages dependent on:
Check Point CPinfo.

**************************************
Check Point CPinfo uninstall complete.
**************************************

- Installing package <CPcpfc-R80.20-00> ...

- Installing package <CPida-R80.20-00> ...

- Installing package <CPmgmt-R80.20-00> ...

- Installing package <CPNGXCMP-R80.20-00> ...

- Installing package <CPEdgecmp-R80.20-00> ...

- Installing package <CPSG80CMP-R80.20-00> ...

- Installing package <CPSG80R75CMP-R80.20-00> ...

- Installing package <CPSFWR77CMP-R80.20-00> ...

- Installing package <CPR71CMP-R80.20-00> ...

- Installing package <CPR75CMP-R80.20-00> ...

- Installing package <CPR7520CMP-R80.20-00> ...

- Installing package <CPR7540CMP-R80.20-00> ...

- Installing package <CPR76CMP-R80.20-00> ...

- Installing package <CPR77CMP-R80.20-00> ...

- Installing package <CPmds-R80.20-00> ...

- Installing package <CPrt-R80.20-00> ...

- Installing package <CPSmartLog-R80.20-00> ...

- Installing package <CPinfo-10-00> ...

- Installing package <CPvsec-R80.20-00> ...

- Installing package <CPdiag-R80.20-00> ...
Preparing Directories and Registries
Performing post install operations
Installing R80.20 Components
Automatically collecting random data to be used in
various cryptographic operations.

[....................]

Automatic collection of random data is done.

Internal Certificate Authority created successfully
Certificate was created successfully
Certificate was created successfully
Trying to contact Certificate Authority. It might take a while...
192.168.135.10 was successfully set to the Internal CA
Certificate was created successfully
Certificate Authority initialization ended successfully
cpridstop: cprid watchdog stopped
cpridstop: cprid stopped
cpridstart: Starting cprid
[1] 19796
Setting FQDN to: 192.168.135.10
do_dns: Executing "$CPDIR/bin/cp_conf ca fqdn 192.168.135.10" in order to set FQDN, round: 0
do_dns: after Executing "$CPDIR/bin/cp_conf ca fqdn 192.168.135.10" status: 0, round: 0
/bin/ln: failed to create symbolic link '/opt/CPSmartLog-R80.20/data': File exists
Running auto configuration
Starting column profile upgrade...

Iterating over '/opt/CPSmartLog-R80.20/data/users_settings' folder
Column profile upgrade Ended.
Starting Multi-Domain Server...

A log file was created: /opt/CPInstLog/mds_setup_08_24_23_39.log

--------------- Importing MDS settings ---------------
Reading configuration of imported Multi-Domain Server.
Export tool version matches import tool version. Proceeding.

Your Multi-Domain Server should NOT be running while you import.
mds_import.sh will now stop the Multi-Domain Server.
Do you want to continue [yes/no] ? Silent update finished (1566683370694 ). Continue with migrate.
Stopping Multi-Domain Server

Stop Search Infrastructure...
Stopping RFL ...
cpwd_admin:
successful Detach operation
Stopping Solr ...
cpwd_admin:
successful Detach operation
Stop SmartView ...
Stopping SmartView ...
cpwd_admin:
successful Detach operation
Stop Log Indexer...
cpwd_admin:
Process INDEXER (pid=21254) stopped with command "kill 21254". Exit code 0.
Stop SmartLog Server...
cpwd_admin:
Process SMARTLOG_SERVER terminated
evstop: Stopping product - SmartEvent Server
evstop: Stopping product - SmartEvent Correlation Unit
Check Point SmartEvent Correlation Unit is not running
cpwd_admin:
Process FWM terminated
cpwd_admin:
Process FWD terminated
Stopping CPM Server ...
cpwd_admin:
Process CPD terminated
cpwd_admin: cpWatchDog killed
Multi-Domain Server stopped
Starting CPM only
Starting cpWatchDog
Starting CPM Server ...
[1] 29185
CPM Server is running.
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is running and ready
CPM server started
----------------------------------------
--- Starting Import Procedure ---
----------------------------------------

Importing Multi-Domain Server data
Upgrading Databases:
Importing Multi-Domain Server Databases.

Another log file is here: /opt/CPInstLog/mds_setup_08_24_23_39.log

[HeikoAnkenbrand]

I think that's the problem with my MDS. The CMA's are not imported.

[JozkoMrkvicka]

but in case there is some error during the import, the installer should revert back to the latest snapshot which was created before upgrade itself.

[HeikoAnkenbrand]

After the 58%, nothing happens with my MDS 24 hours.

I don't see any more log entries here.

I installed the snapshot manually now.

[JozkoMrkvicka]

After 40 minutes installer stuck at 58%, I see that some CMAs are being created:

[HeikoAnkenbrand]

# watch "clish -c 'show installer package 2'"

# tail -f /opt/CPInstLog//install_Major_R80.30_Mgmt_T200.log

[HeikoAnkenbrand]

@JozkoMrkvicka thanks for the support. I'll open a ticket.

[HeikoAnkenbrand]

There is no CMA created:-(

[JozkoMrkvicka]

Heh, I just noticed that you are upgrading to R80.30, not R80.20 as I am upgrading now 😄
I will check in the morning if R80.30 will be smooth in my case (upgrade from R77.30).

Following files should be created during the MDS major upgrade and can be used for troubleshooting:
/var/log/install_Major_*
/opt/CPInstLog//install_Major_*
/opt/CPInstLog/mds_setup_*

[JozkoMrkvicka]

PS: I see you are using the latest CPUSE Deployment Agent, version 1731. I am on 1669. Not saying it is the cause, but nowadays, not everything what is newest is better than older 🙂

[HeikoAnkenbrand]

The cpuse agent  1731 had already caused other problems:-)

I also tested an older version. Same problem, though.

[HeikoAnkenbrand]

I need to get some sleep. Thanks again for the support.

I will try the following tomorrow in the LAB:

R80.10

1) # mds_backup -l                               (Backup without log's)         
2) --> clean install R80.10

3) --> install latest JHFA
4) # mds_restore <backupfile.tgz>
5) --> install cpuse agent 1731
6) --> cpuse upgrade to R80.30

[Lari_Luoma]

Are you doing an in-place upgrade?

Best practice and recommendation is import/export and clean install.

1. pre-upgrade verifier
2. mds_export in R80.10 using R80.30 upgrade tools
3. Clean installation of R80.30
4. First time wizard
5. Jumbo hotfix installation
6. mds_import

[HeikoAnkenbrand]

Hi

@Lari_Luoma 

I tried that:
Upgrading one Multi-Domain Server from R80.10 and lower with CPUSE

After eight hours, nothing has changed:-(

I'll try the other way now:

1. pre-upgrade verifier
2. mds_export in R80.10 using R80.30 upgrade tools
3. Clean installation of R80.30
4. First time wizard
5. Jumbo hotfix installation
6. mds_import

[JozkoMrkvicka]

Hi Heiko,

Can you let us know what environment you are running ? This is only a single MDS (Primary) with no HA and no Logservers ? How many CMAs? Isn't database too big ? Do you have enough disk space on that device ? License is correctly attached on MDS level and on CMAs level ? Are you using Global Policy ?

I see you are referring to upgrade procedure for R80.20, not for R80.30.

I would suggest going through this one:

Upgrading one Multi-Domain Server from R80.20, R80.10, and lower with Advanced Upgrade

[Tal_Paz-Fridman]

Hi Heiko

Is this a very large database?

FYI @Ofer_Barzvi and @Boaz_Orshav 

Thanks

Tal

[HeikoAnkenbrand]

Hi @Tal_Paz-Fridman 

No it is a small datadase.

2 x MDS
4 x Domain
4 x CMA
3 x Backup CMA

[HeikoAnkenbrand]

Hi @Lari_Luoma and @Tal_Paz-Fridman 

I've also tried away 2 now.

1. pre-upgrade verifier
2. mds_export in R80.10 using R80.30 upgrade tools
3. Clean installation of R80.30
4. First time wizard
5. Jumbo hotfix installation
6. mds_import

Unfortunately, it doesn't work either.

[HeikoAnkenbrand · Accepted solution]

Hi @Tal_Paz-Fridman, @Lari_Luoma 

Sorry, here are the detail way:

R80.10 MDS:

# mdsstop
# mdsenv
# mkdir /mnt/cdrom
# mount /dev/dvd /mnt/cdrom               -> VMWare R80.30 ISO
# cd /mnt/cdrom/linux/p1_install/
# ./mds_setup

# ./mds_setup

--> Per WinSCP download  /var/log/exported_mds.25Aug2019-100343.tgz

R80.30 MDS:

--> Fresh install R80.30
--> Install latest JHFA 19
--> VMWare Snapshot 😀
--> Per WinSCP upload  /var/log/exported_mds.25Aug2019-100343.tgz
# yes | nohup $MDSDIR/scripts/mds_import.sh /var/log/exported_mds.25Aug2019-100343.tgz

And here it gets stuck again:-(

[JozkoMrkvicka]

go for lunch or coffee and get back in 1 hour to see some progress within importing 🙂
I started upgrade from R77.30 to R80.30 using CPUSE. It started to import process at 2019-08-25 - 11:34:00 and still no progress in my case till now (12:30). But I see java processes are using 300% of CPU, so something is doing 😄

Would be great to have WORKING progress bar where end user can see real progress of import/upgrade. Not just "Upgrade is still running. Log in to the Status and Actions page to see the progress." where constantly 58% is showing (even after 1 hour) ...

EDIT:
After 70 minutes, my upgrade is done.

[2019-08-25 - 11:34:00][4882 4882]:Importing MDS configuration to destination.
.
.
.

[2019-08-25 - 12:45:13][4882 4882]:[HIGH MSG_SANITY_TEST_SUCCEEDED]: Self Test for Check_Point_R80.30_T200_Fresh_Install_and_Upgrade_Security_Management.tgz succeeded.

[HeikoAnkenbrand]

Hi @Tal_Paz-Fridman, @Lari_Luoma 

If I start the import with -x, I see the errors. There is no detailed information about the error and the log file contains only that output:-(

# $MDSDIR/scripts/mds_import.sh -x /var/log/exported_mds.25Aug2019-100343.tgz

After 48 hours I'm opening a TAC ticket now.

[Aim_Checkoway]

Hi @HeikoAnkenbrand,

Use the mds_import.sh -c option to delete the CMA‘s.

[HeikoAnkenbrand]

THX @Aim_Checkoway,

The -c option doesn't help.

[Maarten_Sjouw]

Heiko,
I did a test upgrade of an upgrade of R77.30 to R80.30, with clean install and using mds_setup. When I started the the import it was around 23:00 the next morning at 7:00 I checked and it was only just finished, you should check the nohup file created in the directory you are in when starting the import from another SSH session. The first 2 hours it was just working and after that it started working on the 30 CMA's we had on that MDS.

[HeikoAnkenbrand · Accepted solution]

Hi @Tal_Paz-Fridman, @Lari_Luoma 

I could solve the issue😀.

When I specify the migration tools of R80.30, the export and import works fine.

Maybe the R80.30 Maigration Tools  from support center are newer than on the ISO image?

Best Regards

Heiko

[Lari_Luoma]

@HeikoAnkenbrand This is likely. I always download the latest tools from the User Center when possible.

[Tommy_Forrest]

I'm having the same issue.

I mds_backup'ed our primary and backup 80.10 MDS (physical) servers.

In VMWare Workstation, I created 2 new VM's and did a fresh install of 80.10 take 479.  I mds_import'ed the files.  Brought the systems up (after a slight modification to adjust the leading interface).

The primary MDS in VMWare Workstation upgraded to 80.30 just fine.  Took about an hour or so.  I was able to log in and install the database on all 6 CMA's.

Unfortunately, my backup MDS in VMWare Workstation is hanging at 58%.  Is this something I can expect in my production environment?

I've blown the VM away several times and started over from scratch (big thanks goes out to a weekly mandatory workstation reboot policy) and each time this issue occurs.

What's the solution short of a clean install (a clean install will not be an option for this upgrade)?

In my case, my VM's do not have access to the internet.  So I'm using files downloaded directly from UserCenter.

Also, anyone have any recommendations on testing a MLM and SmartEvent stand alone server upgrade?