cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post
Highlighted

MDS Upgrade failing from R80.10 to R80.30 (solved)

Jump to solution

The upgrade of an MDS server hangs at this point for more than 24 hours.

 

MDS_4.JPG

There are also no CMA's created under:

/opt/CPmds-R80.30/customers/

What can you do as next step?
TAC case?

 

2 Solutions

Accepted Solutions

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @Tal_Paz-Fridman, @Lari_Luoma 

Sorry, here are the detail way:

R80.10 MDS:

# mdsstop
# mdsenv
# mkdir /mnt/cdrom
# mount /dev/dvd /mnt/cdrom               -> VMWare R80.30 ISO
# cd /mnt/cdrom/linux/p1_install/
# ./mds_setup

clipboard_image_0.png

clipboard_image_1.png

# ./mds_setup

clipboard_image_2.png

clipboard_image_3.png

clipboard_image_4.png

clipboard_image_9.png

--> Per WinSCP download  /var/log/exported_mds.25Aug2019-100343.tgz

 

R80.30 MDS:

-->
Fresh install R80.30
--> Install latest JHFA 19
--> VMWare Snapshot 😀
--> Per WinSCP upload  /var/log/exported_mds.25Aug2019-100343.tgz
# yes | nohup $MDSDIR/scripts/mds_import.sh /var/log/exported_mds.25Aug2019-100343.tgz

clipboard_image_0.png

And here it gets stuck again:-(

 

Tags (1)

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @Tal_Paz-Fridman, @Lari_Luoma 

I could solve the issue😀.

When I specify the migration tools of R80.30, the export and import works fine.

clipboard_image_2.png

Maybe the R80.30 Maigration Tools  from support center are newer than on the ISO image?

Best Regards

Heiko

 
Tags (1)
0 Kudos
28 Replies
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi Heiko,

PUV (Pre-Upgrade Verifier) was run before the upgrade? No errors were found?
Is there enough RAM on that machine?

I faced a similar situation for upgrading R77.30 MDS to R80.x
In my case, it took around 3 hours to complete the upgrade, or revert back to R77.30 latest snapshot (in case of some errors).
During the upgrading, I was also not able to check any CMA using "mdsstat". I was checking status of the upgrade via clish command "show installer package <NUMBER>" where was mentioned path for upgrade log file.

I did tail -f on that file to see if there is any real upgrade progress, together with "top" command.

Here is example for R80.30 Jumbo Take 19 (last line)

image.png

Kind regards,
Jozko Mrkvicka

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @JozkoMrkvicka 

Pre-Upgrade Verifier was run before with no errors!

Yes, enough RAM in that machine:
HP DL380 G9 with 64GB (ESX), 2 x CPU with 8 Cores
VM 32GB, 12 Cores

Now comes the classical way:-)
- Create cpinfo
- Restore R80.10 snapshot
- Open TAC case.

0 Kudos
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Just for fun, I reverted back my R77.30 LAB MDS and started upgrade to R80.20.

Here is the status after reboot (installation stuck at 58%):image.png

 

Here is the last line in Installation log so far:

[2019-08-24 - 23:43:31][4829 4829]:Importing MDS configuration to destination.

There is also a detailed log file, located in /var/log/. In my case, the detailed log file is here:
/var/log/install_Major_R80.20_Mgmt_1_detailed.log

Spoiler

--------------- Installing MDS ---------------
Installation StartedNo Multi-Domain Security Management is detected.

The Multi-Domain Security Management installation includes several
infrastructure packages. These packages will be installed now.

There are no packages dependent on:
Check Point CPinfo.

**************************************
Check Point CPinfo uninstall complete.
**************************************

- Installing package <CPcpfc-R80.20-00> ...

- Installing package <CPida-R80.20-00> ...

- Installing package <CPmgmt-R80.20-00> ...

- Installing package <CPNGXCMP-R80.20-00> ...

- Installing package <CPEdgecmp-R80.20-00> ...

- Installing package <CPSG80CMP-R80.20-00> ...

- Installing package <CPSG80R75CMP-R80.20-00> ...

- Installing package <CPSFWR77CMP-R80.20-00> ...

- Installing package <CPR71CMP-R80.20-00> ...

- Installing package <CPR75CMP-R80.20-00> ...

- Installing package <CPR7520CMP-R80.20-00> ...

- Installing package <CPR7540CMP-R80.20-00> ...

- Installing package <CPR76CMP-R80.20-00> ...

- Installing package <CPR77CMP-R80.20-00> ...

- Installing package <CPmds-R80.20-00> ...

- Installing package <CPrt-R80.20-00> ...

- Installing package <CPSmartLog-R80.20-00> ...

- Installing package <CPinfo-10-00> ...

- Installing package <CPvsec-R80.20-00> ...

- Installing package <CPdiag-R80.20-00> ...
Preparing Directories and Registries
Performing post install operations
Installing R80.20 Components
Automatically collecting random data to be used in
various cryptographic operations.

[....................]

Automatic collection of random data is done.

Internal Certificate Authority created successfully
Certificate was created successfully
Certificate was created successfully
Trying to contact Certificate Authority. It might take a while...
192.168.135.10 was successfully set to the Internal CA
Certificate was created successfully
Certificate Authority initialization ended successfully
cpridstop: cprid watchdog stopped
cpridstop: cprid stopped
cpridstart: Starting cprid
[1] 19796
Setting FQDN to: 192.168.135.10
do_dns: Executing "$CPDIR/bin/cp_conf ca fqdn 192.168.135.10" in order to set FQDN, round: 0
do_dns: after Executing "$CPDIR/bin/cp_conf ca fqdn 192.168.135.10" status: 0, round: 0
/bin/ln: failed to create symbolic link '/opt/CPSmartLog-R80.20/data': File exists
Running auto configuration
Starting column profile upgrade...

Iterating over '/opt/CPSmartLog-R80.20/data/users_settings' folder
Column profile upgrade Ended.
Starting Multi-Domain Server...

A log file was created: /opt/CPInstLog/mds_setup_08_24_23_39.log

--------------- Importing MDS settings ---------------
Reading configuration of imported Multi-Domain Server.
Export tool version matches import tool version. Proceeding.

Your Multi-Domain Server should NOT be running while you import.
mds_import.sh will now stop the Multi-Domain Server.
Do you want to continue [yes/no] ? Silent update finished (1566683370694 ). Continue with migrate.
Stopping Multi-Domain Server

Stop Search Infrastructure...
Stopping RFL ...
cpwd_admin:
successful Detach operation
Stopping Solr ...
cpwd_admin:
successful Detach operation
Stop SmartView ...
Stopping SmartView ...
cpwd_admin:
successful Detach operation
Stop Log Indexer...
cpwd_admin:
Process INDEXER (pid=21254) stopped with command "kill 21254". Exit code 0.
Stop SmartLog Server...
cpwd_admin:
Process SMARTLOG_SERVER terminated
evstop: Stopping product - SmartEvent Server
evstop: Stopping product - SmartEvent Correlation Unit
Check Point SmartEvent Correlation Unit is not running
cpwd_admin:
Process FWM terminated
cpwd_admin:
Process FWD terminated
Stopping CPM Server ...
cpwd_admin:
Process CPD terminated
cpwd_admin: cpWatchDog killed
Multi-Domain Server stopped
Starting CPM only
Starting cpWatchDog
Starting CPM Server ...
[1] 29185
CPM Server is running.
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is running and ready
CPM server started
----------------------------------------
--- Starting Import Procedure ---
----------------------------------------

Importing Multi-Domain Server data
Upgrading Databases:
Importing Multi-Domain Server Databases.

Another log file is here: /opt/CPInstLog/mds_setup_08_24_23_39.log

Kind regards,
Jozko Mrkvicka

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

I think that's the problem with my MDS. The CMA's are not imported.

 

Tags (1)
0 Kudos
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

but in case there is some error during the import, the installer should revert back to the latest snapshot which was created before upgrade itself.

image.png

Kind regards,
Jozko Mrkvicka
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

After the 58%, nothing happens with my MDS 24 hours.

I don't see any more log entries here.

I installed the snapshot manually now.

Tags (1)
0 Kudos
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

After 40 minutes installer stuck at 58%, I see that some CMAs are being created:

image.png

Kind regards,
Jozko Mrkvicka
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

 

# watch "clish -c 'show installer package 2'"

MDS_5.PNG

# tail -f /opt/CPInstLog//install_Major_R80.30_Mgmt_T200.log

MDS_6.PNG

Tags (1)
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

@JozkoMrkvicka thanks for the support. I'll open a ticket.

Tags (1)

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

There is no CMA created:-(

MDS_7.PNG

Tags (1)
0 Kudos
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Heh, I just noticed that you are upgrading to R80.30, not R80.20 as I am upgrading now 😄
I will check in the morning if R80.30 will be smooth in my case (upgrade from R77.30).

Following files should be created during the MDS major upgrade and can be used for troubleshooting:
/var/log/install_Major_*
/opt/CPInstLog//install_Major_*
/opt/CPInstLog/mds_setup_*

 

Kind regards,
Jozko Mrkvicka
0 Kudos
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

PS: I see you are using the latest CPUSE Deployment Agent, version 1731. I am on 1669. Not saying it is the cause, but nowadays, not everything what is newest is better than older 🙂

Kind regards,
Jozko Mrkvicka
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

The cpuse agent  1731 had already caused other problems:-)

I also tested an older version. Same problem, though.



 

 

Tags (1)
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution
I need to get some sleep. Thanks again for the support.
 
I will try the following tomorrow in the LAB:
 
R80.10
1) # mds_backup -l                               (Backup without log's)         
2) --> clean install R80.10
3) --> install latest JHFA
4) # mds_restore <backupfile.tgz>
5) --> install cpuse agent 1731
6) --> cpuse upgrade to R80.30
Tags (1)
0 Kudos
Employee+
Employee+

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Are you doing an in-place upgrade?

Best practice and recommendation is import/export and clean install.

  1. pre-upgrade verifier
  2. mds_export in R80.10 using R80.30 upgrade tools
  3. Clean installation of R80.30
  4. First time wizard
  5. Jumbo hotfix installation
  6. mds_import

 

 

0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi

@Lari_Luoma 

I tried that:
Upgrading one Multi-Domain Server from R80.10 and lower with CPUSE

After eight hours, nothing has changed:-(

MDS_8.PNG

 

I'll try the other way now:

  1. pre-upgrade verifier
  2. mds_export in R80.10 using R80.30 upgrade tools
  3. Clean installation of R80.30
  4. First time wizard
  5. Jumbo hotfix installation
  6. mds_import

 

Tags (1)
0 Kudos
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi Heiko,

Can you let us know what environment you are running ? This is only a single MDS (Primary) with no HA and no Logservers ? How many CMAs? Isn't database too big ? Do you have enough disk space on that device ? License is correctly attached on MDS level and on CMAs level ? Are you using Global Policy ?

I see you are referring to upgrade procedure for R80.20, not for R80.30.

I would suggest going through this one:

Upgrading one Multi-Domain Server from R80.20, R80.10, and lower with Advanced Upgrade

Kind regards,
Jozko Mrkvicka
0 Kudos
Employee++
Employee++

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi Heiko

Is this a very large database?

FYI @Ofer_Barzvi and @Boaz_Orshav 

Thanks

Tal

0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @Tal_Paz-Fridman 

No it is a small datadase.

2 x MDS
4 x Domain
4 x CMA
3 x Backup CMA

 

Tags (1)
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @Lari_Luoma and @Tal_Paz-Fridman 

I've also tried away 2 now.

  1. pre-upgrade verifier
  2. mds_export in R80.10 using R80.30 upgrade tools
  3. Clean installation of R80.30
  4. First time wizard
  5. Jumbo hotfix installation
  6. mds_import

Unfortunately, it doesn't work either.

 

 

Tags (1)
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @Tal_Paz-Fridman, @Lari_Luoma 

Sorry, here are the detail way:

R80.10 MDS:

# mdsstop
# mdsenv
# mkdir /mnt/cdrom
# mount /dev/dvd /mnt/cdrom               -> VMWare R80.30 ISO
# cd /mnt/cdrom/linux/p1_install/
# ./mds_setup

clipboard_image_0.png

clipboard_image_1.png

# ./mds_setup

clipboard_image_2.png

clipboard_image_3.png

clipboard_image_4.png

clipboard_image_9.png

--> Per WinSCP download  /var/log/exported_mds.25Aug2019-100343.tgz

 

R80.30 MDS:

-->
Fresh install R80.30
--> Install latest JHFA 19
--> VMWare Snapshot 😀
--> Per WinSCP upload  /var/log/exported_mds.25Aug2019-100343.tgz
# yes | nohup $MDSDIR/scripts/mds_import.sh /var/log/exported_mds.25Aug2019-100343.tgz

clipboard_image_0.png

And here it gets stuck again:-(

 

Tags (1)
JozkoMrkvicka
Platinum

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

go for lunch or coffee and get back in 1 hour to see some progress within importing 🙂
I started upgrade from R77.30 to R80.30 using CPUSE. It started to import process at 2019-08-25 - 11:34:00 and still no progress in my case till now (12:30). But I see java processes are using 300% of CPU, so something is doing 😄

Would be great to have WORKING progress bar where end user can see real progress of import/upgrade. Not just "Upgrade is still running. Log in to the Status and Actions page to see the progress." where constantly 58% is showing (even after 1 hour) ...

image.png

 

EDIT:
After 70 minutes, my upgrade is done.

[2019-08-25 - 11:34:00][4882 4882]:Importing MDS configuration to destination.
.
.
.

[2019-08-25 - 12:45:13][4882 4882]:[HIGH MSG_SANITY_TEST_SUCCEEDED]: Self Test for Check_Point_R80.30_T200_Fresh_Install_and_Upgrade_Security_Management.tgz succeeded.

Kind regards,
Jozko Mrkvicka
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @Tal_Paz-Fridman, @Lari_Luoma 

If I start the import with -x, I see the errors. There is no detailed information about the error and the log file contains only that output:-(

# $MDSDIR/scripts/mds_import.sh -x /var/log/exported_mds.25Aug2019-100343.tgz

clipboard_image_0.png

After 48 hours I'm opening a TAC ticket now.

 

Tags (1)
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @HeikoAnkenbrand,

Use the mds_import.sh -c option to delete the CMA‘s.

0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

THX @Aim_Checkoway,

The -c option doesn't help.

Tags (1)
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution
Heiko,
I did a test upgrade of an upgrade of R77.30 to R80.30, with clean install and using mds_setup. When I started the the import it was around 23:00 the next morning at 7:00 I checked and it was only just finished, you should check the nohup file created in the directory you are in when starting the import from another SSH session. The first 2 hours it was just working and after that it started working on the 30 CMA's we had on that MDS.
Regards, Maarten
0 Kudos

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

Hi @Tal_Paz-Fridman, @Lari_Luoma 

I could solve the issue😀.

When I specify the migration tools of R80.30, the export and import works fine.

clipboard_image_2.png

Maybe the R80.30 Maigration Tools  from support center are newer than on the ISO image?

Best Regards

Heiko

 
Tags (1)
0 Kudos
Employee+
Employee+

Re: MDS Upgrade failing from R80.10 to R80.30

Jump to solution

@HeikoAnkenbrand This is likely. I always download the latest tools from the User Center when possible.