Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
HeikoAnkenbrand
Champion Champion
Champion
Jump to solution

MDS Upgrade failing from R80.10 to R80.30 (solved)

The upgrade of an MDS server hangs at this point for more than 24 hours.

 

MDS_4.JPG

There are also no CMA's created under:

/opt/CPmds-R80.30/customers/

What can you do as next step?
TAC case?

 

➜ CCSM Elite, CCME, CCTE
2 Solutions

Accepted Solutions
HeikoAnkenbrand
Champion Champion
Champion

Hi @Tal_Paz-Fridman, @Lari_Luoma 

Sorry, here are the detail way:

R80.10 MDS:

# mdsstop
# mdsenv
# mkdir /mnt/cdrom
# mount /dev/dvd /mnt/cdrom               -> VMWare R80.30 ISO
# cd /mnt/cdrom/linux/p1_install/
# ./mds_setup

clipboard_image_0.png

clipboard_image_1.png

# ./mds_setup

clipboard_image_2.png

clipboard_image_3.png

clipboard_image_4.png

clipboard_image_9.png

--> Per WinSCP download  /var/log/exported_mds.25Aug2019-100343.tgz

 

R80.30 MDS:

-->
Fresh install R80.30
--> Install latest JHFA 19
--> VMWare Snapshot 😀
--> Per WinSCP upload  /var/log/exported_mds.25Aug2019-100343.tgz
# yes | nohup $MDSDIR/scripts/mds_import.sh /var/log/exported_mds.25Aug2019-100343.tgz

clipboard_image_0.png

And here it gets stuck again:-(

 

➜ CCSM Elite, CCME, CCTE

View solution in original post

HeikoAnkenbrand
Champion Champion
Champion

Hi @Tal_Paz-Fridman, @Lari_Luoma 

I could solve the issue😀.

When I specify the migration tools of R80.30, the export and import works fine.

clipboard_image_2.png

Maybe the R80.30 Maigration Tools  from support center are newer than on the ISO image?

Best Regards

Heiko

 
➜ CCSM Elite, CCME, CCTE

View solution in original post

29 Replies
JozkoMrkvicka
Mentor
Mentor

Hi Heiko,

PUV (Pre-Upgrade Verifier) was run before the upgrade? No errors were found?
Is there enough RAM on that machine?

I faced a similar situation for upgrading R77.30 MDS to R80.x
In my case, it took around 3 hours to complete the upgrade, or revert back to R77.30 latest snapshot (in case of some errors).
During the upgrading, I was also not able to check any CMA using "mdsstat". I was checking status of the upgrade via clish command "show installer package <NUMBER>" where was mentioned path for upgrade log file.

I did tail -f on that file to see if there is any real upgrade progress, together with "top" command.

Here is example for R80.30 Jumbo Take 19 (last line)

image.png

Kind regards,
Jozko Mrkvicka
HeikoAnkenbrand
Champion Champion
Champion

Hi @JozkoMrkvicka 

Pre-Upgrade Verifier was run before with no errors!

Yes, enough RAM in that machine:
HP DL380 G9 with 64GB (ESX), 2 x CPU with 8 Cores
VM 32GB, 12 Cores

Now comes the classical way:-)
- Create cpinfo
- Restore R80.10 snapshot
- Open TAC case.

➜ CCSM Elite, CCME, CCTE
JozkoMrkvicka
Mentor
Mentor

Just for fun, I reverted back my R77.30 LAB MDS and started upgrade to R80.20.

Here is the status after reboot (installation stuck at 58%):image.png

 

Here is the last line in Installation log so far:

[2019-08-24 - 23:43:31][4829 4829]:Importing MDS configuration to destination.

There is also a detailed log file, located in /var/log/. In my case, the detailed log file is here:
/var/log/install_Major_R80.20_Mgmt_1_detailed.log

Click to Expand

--------------- Installing MDS ---------------
Installation StartedNo Multi-Domain Security Management is detected.

The Multi-Domain Security Management installation includes several
infrastructure packages. These packages will be installed now.

There are no packages dependent on:
Check Point CPinfo.

**************************************
Check Point CPinfo uninstall complete.
**************************************

- Installing package <CPcpfc-R80.20-00> ...

- Installing package <CPida-R80.20-00> ...

- Installing package <CPmgmt-R80.20-00> ...

- Installing package <CPNGXCMP-R80.20-00> ...

- Installing package <CPEdgecmp-R80.20-00> ...

- Installing package <CPSG80CMP-R80.20-00> ...

- Installing package <CPSG80R75CMP-R80.20-00> ...

- Installing package <CPSFWR77CMP-R80.20-00> ...

- Installing package <CPR71CMP-R80.20-00> ...

- Installing package <CPR75CMP-R80.20-00> ...

- Installing package <CPR7520CMP-R80.20-00> ...

- Installing package <CPR7540CMP-R80.20-00> ...

- Installing package <CPR76CMP-R80.20-00> ...

- Installing package <CPR77CMP-R80.20-00> ...

- Installing package <CPmds-R80.20-00> ...

- Installing package <CPrt-R80.20-00> ...

- Installing package <CPSmartLog-R80.20-00> ...

- Installing package <CPinfo-10-00> ...

- Installing package <CPvsec-R80.20-00> ...

- Installing package <CPdiag-R80.20-00> ...
Preparing Directories and Registries
Performing post install operations
Installing R80.20 Components
Automatically collecting random data to be used in
various cryptographic operations.

[....................]

Automatic collection of random data is done.

Internal Certificate Authority created successfully
Certificate was created successfully
Certificate was created successfully
Trying to contact Certificate Authority. It might take a while...
192.168.135.10 was successfully set to the Internal CA
Certificate was created successfully
Certificate Authority initialization ended successfully
cpridstop: cprid watchdog stopped
cpridstop: cprid stopped
cpridstart: Starting cprid
[1] 19796
Setting FQDN to: 192.168.135.10
do_dns: Executing "$CPDIR/bin/cp_conf ca fqdn 192.168.135.10" in order to set FQDN, round: 0
do_dns: after Executing "$CPDIR/bin/cp_conf ca fqdn 192.168.135.10" status: 0, round: 0
/bin/ln: failed to create symbolic link '/opt/CPSmartLog-R80.20/data': File exists
Running auto configuration
Starting column profile upgrade...

Iterating over '/opt/CPSmartLog-R80.20/data/users_settings' folder
Column profile upgrade Ended.
Starting Multi-Domain Server...

A log file was created: /opt/CPInstLog/mds_setup_08_24_23_39.log

--------------- Importing MDS settings ---------------
Reading configuration of imported Multi-Domain Server.
Export tool version matches import tool version. Proceeding.

Your Multi-Domain Server should NOT be running while you import.
mds_import.sh will now stop the Multi-Domain Server.
Do you want to continue [yes/no] ? Silent update finished (1566683370694 ). Continue with migrate.
Stopping Multi-Domain Server

Stop Search Infrastructure...
Stopping RFL ...
cpwd_admin:
successful Detach operation
Stopping Solr ...
cpwd_admin:
successful Detach operation
Stop SmartView ...
Stopping SmartView ...
cpwd_admin:
successful Detach operation
Stop Log Indexer...
cpwd_admin:
Process INDEXER (pid=21254) stopped with command "kill 21254". Exit code 0.
Stop SmartLog Server...
cpwd_admin:
Process SMARTLOG_SERVER terminated
evstop: Stopping product - SmartEvent Server
evstop: Stopping product - SmartEvent Correlation Unit
Check Point SmartEvent Correlation Unit is not running
cpwd_admin:
Process FWM terminated
cpwd_admin:
Process FWD terminated
Stopping CPM Server ...
cpwd_admin:
Process CPD terminated
cpwd_admin: cpWatchDog killed
Multi-Domain Server stopped
Starting CPM only
Starting cpWatchDog
Starting CPM Server ...
[1] 29185
CPM Server is running.
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is running and ready
CPM server started
----------------------------------------
--- Starting Import Procedure ---
----------------------------------------

Importing Multi-Domain Server data
Upgrading Databases:
Importing Multi-Domain Server Databases.

Another log file is here: /opt/CPInstLog/mds_setup_08_24_23_39.log

Kind regards,
Jozko Mrkvicka
HeikoAnkenbrand
Champion Champion
Champion

I think that's the problem with my MDS. The CMA's are not imported.

 

➜ CCSM Elite, CCME, CCTE
JozkoMrkvicka
Mentor
Mentor

but in case there is some error during the import, the installer should revert back to the latest snapshot which was created before upgrade itself.

image.png

Kind regards,
Jozko Mrkvicka
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

After the 58%, nothing happens with my MDS 24 hours.

I don't see any more log entries here.

I installed the snapshot manually now.

➜ CCSM Elite, CCME, CCTE
0 Kudos
JozkoMrkvicka
Mentor
Mentor

After 40 minutes installer stuck at 58%, I see that some CMAs are being created:

image.png

Kind regards,
Jozko Mrkvicka
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

 

# watch "clish -c 'show installer package 2'"

MDS_5.PNG

# tail -f /opt/CPInstLog//install_Major_R80.30_Mgmt_T200.log

MDS_6.PNG

➜ CCSM Elite, CCME, CCTE
HeikoAnkenbrand
Champion Champion
Champion

@JozkoMrkvicka thanks for the support. I'll open a ticket.

➜ CCSM Elite, CCME, CCTE
HeikoAnkenbrand
Champion Champion
Champion

There is no CMA created:-(

MDS_7.PNG

➜ CCSM Elite, CCME, CCTE
0 Kudos
JozkoMrkvicka
Mentor
Mentor

Heh, I just noticed that you are upgrading to R80.30, not R80.20 as I am upgrading now 😄
I will check in the morning if R80.30 will be smooth in my case (upgrade from R77.30).

Following files should be created during the MDS major upgrade and can be used for troubleshooting:
/var/log/install_Major_*
/opt/CPInstLog//install_Major_*
/opt/CPInstLog/mds_setup_*

 

Kind regards,
Jozko Mrkvicka
0 Kudos
JozkoMrkvicka
Mentor
Mentor

PS: I see you are using the latest CPUSE Deployment Agent, version 1731. I am on 1669. Not saying it is the cause, but nowadays, not everything what is newest is better than older 🙂

Kind regards,
Jozko Mrkvicka
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

The cpuse agent  1731 had already caused other problems:-)

I also tested an older version. Same problem, though.



 

 

➜ CCSM Elite, CCME, CCTE
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion
I need to get some sleep. Thanks again for the support.
 
I will try the following tomorrow in the LAB:
 
R80.10
1) # mds_backup -l                               (Backup without log's)         
2) --> clean install R80.10
3) --> install latest JHFA
4) # mds_restore <backupfile.tgz>
5) --> install cpuse agent 1731
6) --> cpuse upgrade to R80.30
➜ CCSM Elite, CCME, CCTE
0 Kudos
Lari_Luoma
Ambassador Ambassador
Ambassador

Are you doing an in-place upgrade?

Best practice and recommendation is import/export and clean install.

  1. pre-upgrade verifier
  2. mds_export in R80.10 using R80.30 upgrade tools
  3. Clean installation of R80.30
  4. First time wizard
  5. Jumbo hotfix installation
  6. mds_import

 

 

0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Hi

@Lari_Luoma 

I tried that:
Upgrading one Multi-Domain Server from R80.10 and lower with CPUSE

After eight hours, nothing has changed:-(

MDS_8.PNG

 

I'll try the other way now:

  1. pre-upgrade verifier
  2. mds_export in R80.10 using R80.30 upgrade tools
  3. Clean installation of R80.30
  4. First time wizard
  5. Jumbo hotfix installation
  6. mds_import

 

➜ CCSM Elite, CCME, CCTE
0 Kudos
JozkoMrkvicka
Mentor
Mentor

Hi Heiko,

Can you let us know what environment you are running ? This is only a single MDS (Primary) with no HA and no Logservers ? How many CMAs? Isn't database too big ? Do you have enough disk space on that device ? License is correctly attached on MDS level and on CMAs level ? Are you using Global Policy ?

I see you are referring to upgrade procedure for R80.20, not for R80.30.

I would suggest going through this one:

Upgrading one Multi-Domain Server from R80.20, R80.10, and lower with Advanced Upgrade

Kind regards,
Jozko Mrkvicka
0 Kudos
Tal_Paz-Fridman
Employee
Employee

Hi Heiko

Is this a very large database?

FYI @Ofer_Barzvi and @Boaz_Orshav 

Thanks

Tal

HeikoAnkenbrand
Champion Champion
Champion

Hi @Tal_Paz-Fridman 

No it is a small datadase.

2 x MDS
4 x Domain
4 x CMA
3 x Backup CMA

 

➜ CCSM Elite, CCME, CCTE
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Hi @Lari_Luoma and @Tal_Paz-Fridman 

I've also tried away 2 now.

  1. pre-upgrade verifier
  2. mds_export in R80.10 using R80.30 upgrade tools
  3. Clean installation of R80.30
  4. First time wizard
  5. Jumbo hotfix installation
  6. mds_import

Unfortunately, it doesn't work either.

 

 

➜ CCSM Elite, CCME, CCTE
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Hi @Tal_Paz-Fridman, @Lari_Luoma 

Sorry, here are the detail way:

R80.10 MDS:

# mdsstop
# mdsenv
# mkdir /mnt/cdrom
# mount /dev/dvd /mnt/cdrom               -> VMWare R80.30 ISO
# cd /mnt/cdrom/linux/p1_install/
# ./mds_setup

clipboard_image_0.png

clipboard_image_1.png

# ./mds_setup

clipboard_image_2.png

clipboard_image_3.png

clipboard_image_4.png

clipboard_image_9.png

--> Per WinSCP download  /var/log/exported_mds.25Aug2019-100343.tgz

 

R80.30 MDS:

-->
Fresh install R80.30
--> Install latest JHFA 19
--> VMWare Snapshot 😀
--> Per WinSCP upload  /var/log/exported_mds.25Aug2019-100343.tgz
# yes | nohup $MDSDIR/scripts/mds_import.sh /var/log/exported_mds.25Aug2019-100343.tgz

clipboard_image_0.png

And here it gets stuck again:-(

 

➜ CCSM Elite, CCME, CCTE
JozkoMrkvicka
Mentor
Mentor

go for lunch or coffee and get back in 1 hour to see some progress within importing 🙂
I started upgrade from R77.30 to R80.30 using CPUSE. It started to import process at 2019-08-25 - 11:34:00 and still no progress in my case till now (12:30). But I see java processes are using 300% of CPU, so something is doing 😄

Would be great to have WORKING progress bar where end user can see real progress of import/upgrade. Not just "Upgrade is still running. Log in to the Status and Actions page to see the progress." where constantly 58% is showing (even after 1 hour) ...

image.png

 

EDIT:
After 70 minutes, my upgrade is done.

[2019-08-25 - 11:34:00][4882 4882]:Importing MDS configuration to destination.
.
.
.

[2019-08-25 - 12:45:13][4882 4882]:[HIGH MSG_SANITY_TEST_SUCCEEDED]: Self Test for Check_Point_R80.30_T200_Fresh_Install_and_Upgrade_Security_Management.tgz succeeded.

Kind regards,
Jozko Mrkvicka
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Hi @Tal_Paz-Fridman, @Lari_Luoma 

If I start the import with -x, I see the errors. There is no detailed information about the error and the log file contains only that output:-(

# $MDSDIR/scripts/mds_import.sh -x /var/log/exported_mds.25Aug2019-100343.tgz

clipboard_image_0.png

After 48 hours I'm opening a TAC ticket now.

 

➜ CCSM Elite, CCME, CCTE
0 Kudos
Aim_Checkoway
Explorer

Hi @HeikoAnkenbrand,

Use the mds_import.sh -c option to delete the CMA‘s.

0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

THX @Aim_Checkoway,

The -c option doesn't help.

➜ CCSM Elite, CCME, CCTE
0 Kudos
Maarten_Sjouw
Champion
Champion
Heiko,
I did a test upgrade of an upgrade of R77.30 to R80.30, with clean install and using mds_setup. When I started the the import it was around 23:00 the next morning at 7:00 I checked and it was only just finished, you should check the nohup file created in the directory you are in when starting the import from another SSH session. The first 2 hours it was just working and after that it started working on the 30 CMA's we had on that MDS.
Regards, Maarten
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Hi @Tal_Paz-Fridman, @Lari_Luoma 

I could solve the issue😀.

When I specify the migration tools of R80.30, the export and import works fine.

clipboard_image_2.png

Maybe the R80.30 Maigration Tools  from support center are newer than on the ISO image?

Best Regards

Heiko

 
➜ CCSM Elite, CCME, CCTE
Lari_Luoma
Ambassador Ambassador
Ambassador

@HeikoAnkenbrand This is likely. I always download the latest tools from the User Center when possible.

Tommy_Forrest
Advisor

I'm having the same issue.

 

I mds_backup'ed our primary and backup 80.10 MDS (physical) servers.

In VMWare Workstation, I created 2 new VM's and did a fresh install of 80.10 take 479.  I mds_import'ed the files.  Brought the systems up (after a slight modification to adjust the leading interface).

The primary MDS in VMWare Workstation upgraded to 80.30 just fine.  Took about an hour or so.  I was able to log in and install the database on all 6 CMA's.

Unfortunately, my backup MDS in VMWare Workstation is hanging at 58%.  Is this something I can expect in my production environment?

I've blown the VM away several times and started over from scratch (big thanks goes out to a weekly mandatory workstation reboot policy) and each time this issue occurs.

What's the solution short of a clean install (a clean install will not be an option for this upgrade)?

In my case, my VM's do not have access to the internet.  So I'm using files downloaded directly from UserCenter.

 

Also, anyone have any recommendations on testing a MLM and SmartEvent stand alone server upgrade?

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events