Installation & Upgrade problems from R77.x to R80.x

[Oliver_Fink]

We did several update projects from different versions of R77.x to R80.x. Some went good, some went bad. I want to share our experiences and to ask what your experiences are and how you solved problems. Maybe we can learn from each other.

Customer 1: R77.30 to R80.20.M1 & R80.10

Environment:

• 2 security gateway clusters (HA)
  • 21800
  • 13800
• 2 log servers
  • Smart-1 3500
• 1 security management server
  • Open Server in VMware

Problems:

The upgrade worked without problems. We had problems to run special scripts for the customer. They are not related to the upgrade but to the bad documentation of SmartEvent internals. It seems that parts of this documentation have not been updated for years.

Customer 2: R77.30 to R80.20

Environment:

• 1 security gateway
  • Open Server
• 2 remote office VPN gateways
  • 1120
  • 1140
• 1 security management server
  • Open Server

Problems:

The upgrade was done without problems. But the customer discovered problems with one VPN connection. Mails could not be send anymore while other communication worked (RDP). This site was connected with a DSL router and the 1120 behind. The problem was with the MSS clamping used that worked seamlessly before. The following SKs were involved:

• sk101219 - New VPN features in R77.20 and later
• sk111412 - 'MSS clamping' value not enforced on interface for VPN traffic
• sk112094 - MSS value is not applied to IPsec VPN traffic, although MSS Adjustment (Clamping) for IPs...

We finally got the VPN running again.

Customer 3: R77.30 to R80.20

Environment:

• 1 security gateway cluster
  • 4600
• 1 security management server
  • Smart-1 504, migrated to Open Server in VMware

Problems:

We did the update because the customer changed his ISP. The new provider implemented an internet connection with backup via LTE. This is transparent to the firewall. As a result of this the external IPs of the gateway cluster changed to private address space, something from the 10.0.0.0/8 range. The official IPs reside at the ISP's data center and are NATed there. Such, we needed NAT traversal on incoming and outgoing VPN connections.

We changed the configuration and did the upgrade. Using only R80.20 security management and R77.30 gateway cluster we did not experience problems. But still outgoing VPN connections with NAT-T were not possible, because you need R80.x for that. For that we also upgraded the gateways. We tried with CPUSE upgrade. During this process the gateways lost their SIC names and certificates. We had no faith that nothing else went wrong besides that. So we decided to do a fresh install with from a USB stick with ISOmorphic.

After activating the new R80.20 gateways we had two serious problems:

1. Massive problems on SMTP connections from the customers MX at a service provider to the internal Trend Micro gateway. We saw massive log entries of "First packet isn't SYN". They got less after we fixed problem #2 but did not vanish totally. We will have a closer look on this in 2019.
2. Site-to-site and Remote Access VPNs did not work anymore. We thought we did something wrong with the configuration. But that made no sense because also VPNs working incoming with R77.30 gateways did not work anymore. Anyway, we were not sure that we tested enough. Finally we involved TAC. The supporter gathered some information and said that he was going to analyze them. I saw some occurrences of "dropped by fwpslglue_chain Reason: PSL Reject: internal - reject enabled" when he executed "fw ctl zdebug drop". I should have issued the command by myself, so shame on me. I put the debug message into Google search and the first entry lead me to sk109777: Traffic is dropped with log: "PSL Reject: internal - reject enabled" after migration to a .... The solution was sk33328: How to clear $FWDIR/state/ directory to resolve policy corruption issues. This meant that we had to execute a cpstop on security management server and all gateway cluster nodes at the same time – this means a total outage of the firewall cluster. No fun for the customer, but this fixed all our VPN problems a once.

Customer 4: R77.20 to R80.20

Environment:

• 2 security gateway clusters
  • Open Server
• 1 security management server
  • Open Server

Problems:

Due to service window and other time restrictions of the customer we did the upgrade for the management server some weeks before the first gateway cluster. Everything went fine and the R77.20 gateways worked very well and without any problem with the security management server.

After that we upgraded the first cluster. First we experienced no serious problems. The customer recognized that he could not reach every interface from his monitoring host. While searching for the reason I also stumbled upon "dropped by fwpslglue_chain Reason: PSL Reject: internal - reject enabled" when doing "fw ctl zdebug drop" (Please honor this: I am able to learn. ;-). Finally, together with TAC we discovered that the reason was a difference in anti-spoofing behaviour.

We still see the debug messages from "fw ctl zdebug drop" but the customer did not identify any issue by now. This does not say that he will not identify them tomorrow. I do not feel really well with the case that the gateway cluster silently drops packages and  sk109777: Traffic is dropped with log: "PSL Reject: internal - reject enabled" after migration to a ... explicitly saying:

Cause

A corruption to the policy files happened during the migration.

I identified that the connections dropped are high port to high port or high port to port 135. This leads to the conclusion that Windows RPC is affected. For that a service request is open at TAC.

I want to mention that TAC and sk109777 are saying different things. The sk109777 states that policy files are corrupted, TAC says that this is one possibility but there may be other reasons. Besides: sk33328: How to clear $FWDIR/state/ directory to resolve policy corruption issues was executed twice and did not change anything.

Customer 5: R77.30 VSX to R80.20

Environment:

• 2 node VSX VSLS cluster
  • Open Server
• security management server
  • Open Server

Problems:

Upgrade of management server to R80.20 went without problems. When upgrading the VSX cluster nodes the first node also shows "dropped by fwpslglue_chain Reason: PSL Reject: internal - reject enabled" when doing "fw ctl zdebug drop". My colleague said that the customer experienced massive traffic problems. The problem is, that you cannot switch the VSX cluster object back to R77.30 once you have set it to R80.20, he stated. Always think of snapshotting before doing an upgrade.  🙂

At the moment we have a working VSX cluster node with R77.30 and a dysfunctional one with R80.20 for debugging purposes with the TAC where a service request is open. The customer is not very happy about running a cluster with only one working node when paying for 2 for good reasons. I think he is right.

Customer 6: R77.30 to R80.20

Environment:

• 2 node security cluster
  • 13500 HPP
• 1 SandBlast Appliance
  • TE250X
• 2 security management servers
  • Smart-1 210
  • Open Server on VSX

Problems:

We have problems to even import the files from "migrate export" from R77.30 to a freshly installed management server with R80.20. A service request is open at TAC. We had to delete one object from the policy and got a script that did 2 SQLite deletions. We are still working to check if this fixes the problem.

In pre-upgrade verifier we also see warnings for blades we never had activated in the customers environment. We suspect that their origin is an export from an MDM of the former service partner of the customer. 

Customer 7: R8.10 fresh installation

Environment:

• 1 security gateway cluster
  • 4200
• 1 security management server
  • Open Server VMware

Problems:

Rules were ordered by traffic flow directions with layers that implement detailed rules for that directions. Randomly rules in a sub-layer do not work and do not log. The workaround is to put them into the top layer instead. But this cannot be the permanent solution. Service request had not been opened yet.

During lab installation we wanted to disable the policy on the gateway because a connection to a specific interface was not possible. So we did an "fw unloadlocal". After that a connection was still not possible. We discovered that the anti-spoofing settings still stuck to this interface. That seems weird for me.

Customer 8: SmartMove from Cisco ASA to R80.20

Environment:

• security gateway cluster
  • 5800
• security management server
  • Open Server

Problems:

No problems at all. Even if we used an older version of ASA software than supported we were able to import a policy, NAT made some troubles. We decided not to use the ASA policy structure and copy-pasted rules from one policy package to another, did new NAT rules and used the imported objects with the weird names to modify rules and groups. Given the poor potential of ASA for meaningful naming SmartMove did a really great job helping us to migrate the customer. A big praise to Check Point for that.

Customer 9: R77.30 VSX with VSLS & MDM to R80.20

Environment:

• 4 node VSX VSLS cluster
  • Open Server
• many GAiA and embedded GAiA firewalls
  • not covered by this text
• 2 multi-domain security management servers
  • Open Server

Problems:

We are at the beginning of the project. My colleague encounters funny messages after "migrate import" in the lab environment. After importing SmartConsole complains about objects with leading or trailing spaces. We are wondering why this is not checked with the pre-upgrade verifier.

Due to the complexity of this installation and the problems we experienced and still experience I suggested to my colleague to postpone this migration until we – and even Check Point! – understand better which problems arise through migration. I am not quite sure if management and sales have the same view on it like me.  😉

Some more words

At one customer I forgot to insert the new CPM port (19009) into the policy before upgrading the management server. Stupid me! I accessed it through the firewall cluster – no problem in my test lab at all. In the past I was able to get through a firewall with SSH forwarding to the firewall gateway and connecting to localhost. Seems that this is not possible anymore to avoid man-in-the-middle attacks. Not good for me in this case but a smart move anyway. (We had to patch a cable to the management network. Customer accepted this with a smile. Luckily.)

The policy verifier is a lot more efficient than before. This might take some time after upgrading until you get overlapping rules fixed and can push policy again – depending on the existing policy design and accuracy. I appreciate that better policy verifier. But if you forget to insert port 19009 into your policy (see above) and need to push the policy soon, this could be unnerving.

Conclusions

I am not a friend of beating the supporters at the TAC. They do a difficult job and often covered our asses in the past. But I have to state that my impression is that they are heavy struggling with the problems arising from updates to R80.20. In one service request I felt they were playing games with me to gain time. I do not know how high the load in TAC is due to upgrade problems. But I heard of a partner meeting in Tel Aviv where also other partners complained massively about problems when upgrading to R80.20.

Our problem is the massive lack of information. Check Point must have learned from the problems by now. I cannot believe that no list of Dos, Don'ts and Caveats exist in Tel Aviv. But partners and customers do not know about it. And it seems that even the TAC does not get informed in a manner that they are able to help fast and effectively. This would be a good time to publish a Best Practices for Upgrades to R80.20 primer by Check Point. Maybe there is one. Then I would like to get known to it.

I would suggest that Check Point modifies the pre-upgrade verifier to recognize illegal spaces in object names and to offer an option to eliminate them automatically during "migrate export". This is no rocket science. The spaces should not have made it into the policy in the past and nobody should have to remove them by hand today.

What I have learned from all these updates is, that you gain nothing by upgrading the security management server to R80.20 first and test it with R77.x gateways. We do not experience any serious problem in this combination. All our problems started when it came to R80.x gateways. This said, I even suspect the problem to reside within the security management server itself in many cases, but to get effective only with the new gateways.

At the moment, I have no confidence in the upgrade procedures for VSX and MDM – even in combination! We had so much trouble with simple environments that I do not dare to go to complex ones. 

I would like to know about experiences from other upgrades, problems and solutions. As long as Check Point seems to put a non-disclosure policy on upgrade problems and does not deliver help and solutions, CheckMates is the best place to help each other. I am neither angry nor frustrated. Do not get me wrong. I work with Check Point every day. This is my job and I do it with love. But I think it is time for Check Point to speak with us about problems with and solutions for R80.20 upgrades. I would appreciate that very much – to deliver a better performance to Check Point's and our customers.

[_Val_]

Oliver Fink‌, thanks a lot for this elaborate collection of cases. We have asked relevant departments to look into all mentioned issues.

[_Val_]

Oliver Fink‌, thanks to nice more for this info. You should be hearing from several CP specialists shortly, please do not be surprised. We want to do as much as possible to analyze each of your cases and SRs to make sure we address them properly on multiple levels of QA, development and TAC

[Oliver_Fink]

Hi, Valeri.

That sounds good and I appreciate this very much. But I am on my last hours for this year and out of office to make a customer happy at the moment. 

It would be fine if we can fix some things in 2019 together with Check Point.

[_Val_]

Whenever you are ready 🙂

[Oliver_Fink]

We had partial success with customer 6. Got a shell script from TAC that deletes two entries from sqlite. After that and some pre-upgrade verifier cleanup we were able to import to R80.20 security management server in lB. The fun will continue 2019 with a real life upgrade for that customer. No good idea to start this just before Christmas holidays. 

[Alex_Lam1]

Hey Oliver

We have the same/similar kind of the issue of the customer 6 & 9.

Pretty frustrating really.

I have the TAC case opened already but felt like they are buying times like you said.

I've gone through the cpm_for_cpdb_xxxxx log and found couple of things. Need to deserialize object (which doesn't exist) and object that is " null"a s the script of the "mds import" of the R80.20 or R80 should be meant to fix this kind of the issue. 

I have also uploaded the exported DB to TAC and also requested tier 3 engineer to investigate this issue to be fixed.

I wonder if the TAC give you a custom script to run to fix the object(s).

If you could share what TAC has told you how to fix the a non-exist object that would be great!

But if it is a custom script - I wonder how many custom scripts they are writing or going to write for each failures 😕

Alex

[Oliver_Fink]

TAC send us a small shell script that is customized for special object identifiers:

#!/bin/bash

#
# check that we are on Multi-Domain
#
if [ "$MDSDIR" != "" ]; then
 #
 # We are in MDS, verify that we are under the mdsenv of a CMA
 #
 if [ "$MDSDIR" = "$FWDIR" ]; then
 2>&1 echo -e "This script must be run from a CMA environment. Run \n\tmdsenv yourCmaName\nand then rerun this command."
 exit 1
 fi
fi

sqlite3 $FWDIR/conf/new_security_rb.sqlite "delete from anti_malware_rulebase_sections where UUID is '{B2CFDA5A-93D2-FE4D-AB90-C68199D52E91}';"
sqlite3 $FWDIR/conf/new_security_rb.sqlite "delete from rulebase_entity_local_instance_table where EntityUid is '{B2CFDA5A-93D2-FE4D-AB90-C68199D52E91}';"
exitCode=$?

if [ $exitCode != 0 ];
then
 2>& echo "Operation did not succeed. Please contact support."
else
 echo "Done"
fi

exit $exitCode

Thus, I assume it is an individualized script based on a template they use for different customers. TAC still has to analyze your data to get known of your object identifiers.

Sorry for the late answer. Your comment got somehow out of my focus.

[JozkoMrkvicka]

Hello,

I am facing a similar issue with upgrade of MLM from R77.30 to R80.20.

My environment (all VMs):
1x R77.30 MDM with 6 CMAs
1x R77.30 MLM with 6 CMAs

MDM and MLM are synced each other. Both machines have installed Take 345 and the deployment agent build 1671.

I want to upgrade MLM from R77.30 to R80.20.  I have downloaded R80.20 upgrade file. "Installer verify" says that upgrade is possible. Once I run "installer upgrade", I see that after reboot it says that MLM is already on R80.20, but the upgrade is still in progress. At some point, the upgrade failed and MLM is reverted back to R77.30.

I checked all files created during upgrade, but I was not able to identify the issue.

There is similar SK about that, but it is not relevant for me, as I want to upgrade from R77.30 to R80.20.

Is there any way how to run upgrade in some kind of "debug" mode ? Or where exactly I am able to check what is causing that upgrade is reverted back to R77.30 ?

Here are some log files:

/opt/CPInstLog//install_Major_R80.20_Mgmt.log:

Click to Expand

[Expert@LOGGER:0]# tail -f /opt/CPInstLog//install_Major_R80.20_Mgmt.log
[2019-05-04 - 21:45:56][6106 13848]:Copying log dir
[2019-05-04 - 21:45:56][6106 13848]:about to copy dir /opt/CPInstLog/ to /mnt/fcd//opt
[2019-05-04 - 21:45:56][6106 13848]:About to execute command: /bin/cp -af /opt/CPInstLog/ /mnt/fcd//opt
[2019-05-04 - 21:47:27][6800 6800]:Setting detailed log to: /var/log/install_Major_R80.20_Mgmt_detailed.log
[2019-05-04 - 21:47:27][6800 6800]:***Post boot action starting... ***
[2019-05-04 - 21:47:27][6800 6800]:------ Installing Products: ------
[2019-05-04 - 21:47:27][6800 6800]:------ Upgrading Products: ------
[2019-05-04 - 21:47:27][6800 6800]:Running post boot management installation
[2019-05-04 - 21:47:27][6800 6800]:Installing products...
[2019-05-04 - 21:47:27][6800 6800]:About to execute command: /sysimg/CPwrapper/linux/p1_install//mds_setup -webui_install >> /var/log/install_Major_R80.20_Mgmt_detailed.log 2>&1
[2019-05-04 - 21:57:18][6800 6800]:/sysimg/CPwrapper/linux/p1_install//mds_setup -webui_install >> /var/log/install_Major_R80.20_Mgmt_detailed.log 2>&1 command summary:
Return code = 0
Output =
[2019-05-04 - 21:57:18][6800 6800]:About to execute command: . /opt/CPda/bin/p1_post_install.sh
[2019-05-04 - 21:57:20][6800 6800]:. /opt/CPda/bin/p1_post_install.sh command summary:
Return code = 0
Output =
[2019-05-04 - 21:57:20][6800 6800]:Importing MDS configuration to destination.
[2019-05-04 - 22:02:12][6800 6800]:Failed on importing MDS settings to destination
[2019-05-04 - 22:02:12][6800 6800]:------ Reverting Back: ------
[2019-05-04 - 22:02:13][6800 6800]:Moving partitions compression requests file to the new partition (/tmp/major_revert)
[2019-05-04 - 22:02:13][6800 6800]:Adding request for compression for AutoSnapShot133
[2019-05-04 - 22:02:13][6800 6800]:About to execute command: flist="Kerntypes* System.map* config* initrd* kernel* vmlinuz*"; cd /boot/; cp -a $flist /boot/AutoSnapShot133/; cp -a /boot/grub/grub.conf /boot/AutoSnapShot133/grub.conf
[2019-05-04 - 22:02:13][6800 6800]:flist="Kerntypes* System.map* config* initrd* kernel* vmlinuz*"; cd /boot/; cp -a $flist /boot/AutoSnapShot133/; cp -a /boot/grub/grub.conf /boot/AutoSnapShot133/grub.conf command summary:
Return code = 0
Output = cp: cannot stat 'Kerntypes*': No such file or directory
cp: cannot stat 'kernel*': No such file or directory

[2019-05-04 - 22:02:13][6800 6800]:About to execute command: flist="Kerntypes* System.map* config* initrd* kernel* vmlinuz*"; cd /tmp/major_revert/boot_bck/; cp -a $flist /boot/; cp -a /tmp/major_revert/boot_bck/grub.conf /boot/grub/grub.conf
[2019-05-04 - 22:02:13][6800 6800]:flist="Kerntypes* System.map* config* initrd* kernel* vmlinuz*"; cd /tmp/major_revert/boot_bck/; cp -a $flist /boot/; cp -a /tmp/major_revert/boot_bck/grub.conf /boot/grub/grub.conf command summary:
Return code = 0
Output = cp: cannot stat 'Kerntypes*': No such file or directory
cp: cannot stat 'kernel*': No such file or directory

[2019-05-04 - 22:02:13][6800 6800]:Copying /var/log from the destination OS
[2019-05-04 - 22:02:13][6800 6800]:about to copy dir /tmp/major_revert/var/log to /var
[2019-05-04 - 22:02:13][6800 6800]:About to execute command: /bin/cp -af /tmp/major_revert/var/log /var
[2019-05-04 - 22:02:13][6800 6800]:/bin/cp -af /tmp/major_revert/var/log /var command summary:
Return code = 0
Output =
[2019-05-04 - 22:02:13][6800 6800]:Copying log dir
[2019-05-04 - 22:02:13][6800 6800]:about to copy dir /opt/CPInstLog/ to /tmp/major_revert//opt
[2019-05-04 - 22:02:13][6800 6800]:About to execute command: /bin/cp -af /opt/CPInstLog/ /tmp/major_revert//opt
[2019-05-04 - 22:02:13][6800 6800]:/bin/cp -af /opt/CPInstLog/ /tmp/major_revert//opt command summary:
Return code = 0
Output =
[2019-05-04 - 22:02:13][6800 6800]:Warning: Could not create major_post_commands file /opt/CPda/bin/post_commands
[2019-05-04 - 22:02:14][6800 6800]:Removing previous request for compression for lv_AutoSnapShot16

Broadcast message from admin@LOGGER (Sat May 4 22:02:29 2019):

The system is going down for reboot NOW!

 /var/log/install_Major_R80.20_Mgmt_detailed.log:

Click to Expand

[Expert@LOGGER:0]# tail -f /var/log/install_Major_R80.20_Mgmt_detailed.log

- Installing package <CPR7540CMP-R80.20-00> ...

- Installing package <CPR76CMP-R80.20-00> ...

- Installing package <CPR77CMP-R80.20-00> ...

- Installing package <CPmds-R80.20-00> ...

- Installing package <CPrt-R80.20-00> ...

- Installing package <CPSmartLog-R80.20-00> ...

- Installing package <CPinfo-10-00> ...

- Installing package <CPvsec-R80.20-00> ...

- Installing package <CPdiag-R80.20-00> ...
Preparing Directories and Registries
Performing post install operations
Installing R80.20 Components
Automatically collecting random data to be used in
various cryptographic operations.

[....................]

Automatic collection of random data is done.

cpridstop: cprid watchdog stopped
cpridstop: cprid stopped
cpridstart: Starting cprid
[1] 17230
/bin/ln: failed to create symbolic link '/opt/CPSmartLog-R80.20/data': File exists
Running auto configuration
Starting column profile upgrade...

Iterating over '/opt/CPSmartLog-R80.20/data/users_settings' folder
Column profile upgrade Ended.
Starting Multi-Domain Server...

A log file was created: /opt/CPInstLog/mds_setup_05_04_21_47.log

--------------- Importing MDS settings ---------------
Reading configuration of imported Multi-Domain Server.
Export tool version matches import tool version. Proceeding.

Your Multi-Domain Server should NOT be running while you import.
mds_import.sh will now stop the Multi-Domain Server.
Do you want to continue [yes/no] ? Stopping Multi-Domain Server

Stop Search Infrastructure...
Stopping RFL ...
cpwd_admin:
successful Detach operation
Stopping Solr ...
cpwd_admin:
successful Detach operation
Stop SmartView ...
Stopping SmartView ...
cpwd_admin:
successful Detach operation
Stop Log Indexer...
cpwd_admin:
Process INDEXER (pid=19039) stopped with command "kill 19039". Exit code 0.
Stop SmartLog Server...
cpwd_admin:
Process SMARTLOG_SERVER terminated
evstop: Stopping product - SmartEvent Server
evstop: Stopping product - SmartEvent Correlation Unit
Check Point SmartEvent Correlation Unit is not running
cpwd_admin:
Process FWM terminated
cpwd_admin:
Process FWD terminated
Stopping CPM Server ...
cpwd_admin:
Process CPD terminated
cpwd_admin: cpWatchDog killed
Multi-Domain Server stopped
Starting CPM only
Starting cpWatchDog
Starting CPM Server ...
[1] 22315
CPM Server is running.
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is during initialization
Waiting for CPM server...
Check Point Security Management Server is running and ready
CPM server started
----------------------------------------
--- Starting Import Procedure ---
----------------------------------------

Importing Multi-Domain Server data
Upgrading Databases:
Importing Multi-Domain Server Databases.
License already exists
Host Expiration Features
192.168.135.111 27May2019 CPSM-C-U CPSB-NPM CPSB-EPM CPSB-LOGS CPSB-GBLP CPSB-COMP-U CK-36C50720103B

Starting Multi-Domain Server only
cpWatchDog is already running
CPM Server already running.
CPM Server is running.
Start Search Infrastructure...
index mode was set to true
startsearch: dbsync does not run on Multi-Domain Security Management
cpwd_admin:
Process SOLR started successfully (pid=26257)
Starting RFL ...
cpwd_admin:
Process RFL started successfully (pid=26286)
Starting SmartView ...
cpwd_admin:
Process SMARTVIEW started successfully (pid=26308)
Start Log Indexer...
cpwd_admin:
Process INDEXER started successfully (pid=26747)
Start SmartLog Server...
cpwd_admin:
Process SMARTLOG_SERVER started successfully (pid=27039)

Multi-Domain Server Started
Execution finished with errors. See log file '/opt/CPshrd-R80.20/log/migrate-2019.05.04_22.01.00.log' for further details

Error: Failed to upgrade Multi-Domain Server Databases

Error: Importing of Multi-Domain Server Databases failed.
DONE.
Stopping CPM Server ...
Stopping Multi-Domain Server

Stop Search Infrastructure...
Stopping RFL ...
cpwd_admin:
successful Detach operation
Stopping Solr ...
cpwd_admin:
successful Detach operation
Stop SmartView ...
Stopping SmartView ...
cpwd_admin:
successful Detach operation
Stop Log Indexer...
cpwd_admin:
Process INDEXER (pid=26747) stopped with command "kill 26747". Exit code 0.
Stop SmartLog Server...
cpwd_admin:
Process SMARTLOG_SERVER terminated
evstop: Stopping product - SmartEvent Server
evstop: Stopping product - SmartEvent Correlation Unit
Check Point SmartEvent Correlation Unit is not running
cpwd_admin:
Process FWM terminated
cpwd_admin:
Process FWD terminated
Stopping CPM Server ...
cpwd_admin:
Process CPD terminated
cpwd_admin: cpWatchDog killed
Multi-Domain Server stopped

Summary of Upgrade operation:

=====================================================================

Import operation started at: Sat May 4 21:57:21 CEST 2019

Multi-Domain Server databases - Failure

=====================================================================

--------------------------------------------------------------------------------
Import operation Failed.
Please see log file /opt/CPInstLog/import_mds.4May2019-215721.log for details
--------------------------------------------------------------------------------
DONE.

Broadcast message from admin@LOGGER (Sat May 4 22:02:29 2019):

The system is going down for reboot NOW!

/opt/CPInstLog/mds_setup_05_04_21_47.log:

Click to Expand

[Expert@LOGGER:0]# tail -f /opt/CPInstLog/mds_setup_05_04_21_47.log
Performing post install operations
Processing postinstall with parameter upgrade
Proceeding with fresh installation
- Registering Global Domain UID to 1e294ce0-367a-11e3-aa6e-0800200c9a66 in registry
- Initializing databases:
get_cust_name: couldn't find /customers/ within fwdir: No such file or directory
get_cust_name: couldn't find /customers/ within fwdir: No such file or directory
get_cust_name: couldn't find /customers/ within fwdir
get_cust_name: couldn't find /customers/ within fwdir: No such file or directory
'/opt/CPsuite-R80.20/fw1/scripts/cpm.sh -s -d -e'
- Setting status interval
Installing R80.20 Components
HFA hooks runner script has been detected.
[ Sat May 4 21:57:05 CEST 2019 ]: ========================================================================
[ Sat May 4 21:57:05 CEST 2019 ]: WELCOME TO HFA HOOKS RUNNER
[ Sat May 4 21:57:05 CEST 2019 ]: ========================================================================
[ Sat May 4 21:57:05 CEST 2019 ]: Executing post_install.sh
[ Sat May 4 21:57:05 CEST 2019 ]: Return code: 0
[ Sat May 4 21:57:05 CEST 2019 ]: Finished!
HFA hooks runner script has completed.
No Multi-Domain Security Management HFA detected.
0
0
0
0
0
setting MDS info to files:
some error message will be displayed for fresh installation of an Multi-Domain Server which is not the first
Failed to get sic name from registry

A log file was created: /opt/CPInstLog/mds_setup_05_04_21_47.log
ExitTrap
ExitTrap> Restoring execution permissions of /opt/CPmds-R80.20/scripts/mdsstart
ExitTrap> Restoring execution permissions of /opt/CPmds-R80.20/scripts/mdsstart_customer

Broadcast message from admin@LOGGER (Sat May 4 22:02:29 2019):

The system is going down for reboot NOW!

[Miri_Ofir]

Hi @JozkoMrkvicka 

Please open SR with TAC and share with me privately the SR number. I'm from R&D and will monitor your case to help you with the upgrade.

[JozkoMrkvicka]

Hi @Miri_Ofir ,

I was able to resolve my issue. I simply didn't follow upgrade steps 🙂

There is clearly stated that:

and my attempt was to upgrade ONLY Multi-Domain Log Server, which is not possible.

Anyway, it would be great to state it in the log files, because there is no info that MDS wasn't upgraded to R80.20.

[HeikoAnkenbrand]

Hi

We have the same/similar kind of the issue from point 3,5,6 & 9 and more.

Regards

Heiko

[Miri_Ofir]

@HeikoAnkenbrand 

Same for you, If you still have problems with the upgrade, please share with me the SR number and I will try to assist 

[Tal_Paz-Fridman]

Hi Oliver.

Regarding Customer 9, you wrote:

"After importing SmartConsole complains about objects with leading or trailing spaces. We are wondering why this is not checked with the pre-upgrade verifier."

Can you give specific examples or perhaps even a screenshot of the types of objects with the leading/trailing spaces?

Can you also state if it is a Warning or an Error? Do you have to fix it in order to Publish?

Kind regards,

Tal

[JozkoMrkvicka]

Is there some way how to run cma_migrate in some of debug mode ? We have exported single CMA from R77.30 based on R80.20 tools, but on R80.20 we cannot import this CMA due to some errors (suspect some files are corrupted). I have followed THIS procedure.

[IrinaK]

Hi Jozko,

can you send the errors you see in import procedure, please? you can contact me at irinamot@checkpoint.com.

regards,

Irina.

[JozkoMrkvicka]

Hi @IrinaK ,

Here is the end of the import log.

The interesting thing is that I was able to export this CMA from R77.30 without any errors (just 1 warning about global policy present). So I am wondering why the import is not possible...

Logs from file: migrate-<DATE>.log:

Click to Expand

[7 May 21:54:09] ...--> CanonicalizePath
[7 May 21:54:09] [CanonicalizePath] Canonicalizing path '/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/conf'
[7 May 21:54:09] [CanonicalizePath] Resulting path: '/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/conf/'
[7 May 21:54:09] ...<-- CanonicalizePath
[7 May 21:54:09] [DbUpgrader::ExpandDbPaths] Expanded destination directory: '/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/conf/'
[7 May 21:54:09] ..<-- DbUpgrader::ExpandDbPaths
[7 May 21:54:09] ..--> DbUpgrader::UpgradeMainDatabase
[7 May 21:54:09] ...--> DbUpgrader::HandleSourcePluginsTables
[7 May 21:54:09] [DbUpgrader::HandleSourcePluginsTables] File with merged plugins tables is present at '/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/tmp//export_db/main_db//upgrade_tools_tables.C'
[7 May 21:54:09] ...<-- DbUpgrader::HandleSourcePluginsTables
[7 May 21:54:09] ...--> DbUpgrader::BuildMainDbUpgradeCommand
[7 May 21:54:09] ....--> DbUpgrader::GetSourceMainDbVersion
[7 May 21:54:09] .....--> MigrateConfig::Instance
[7 May 21:54:09] .....<-- MigrateConfig::Instance
[7 May 21:54:09] .....--> MigrateConfig::GetConfigSet
[7 May 21:54:09] .....<-- MigrateConfig::GetConfigSet
[7 May 21:54:09] .....--> IsVersionSetValid
[7 May 21:54:09] .....<-- IsVersionSetValid
[7 May 21:54:09] .....--> GetVersionString
[7 May 21:54:09] ......--> IsVersionSetValid
[7 May 21:54:09] ......<-- IsVersionSetValid
[7 May 21:54:09] .....<-- GetVersionString
[7 May 21:54:09] ....<-- DbUpgrader::GetSourceMainDbVersion
[7 May 21:54:09] ....--> DbUpgrader::GetSourceHfaLevel
[7 May 21:54:09] .....--> MigrateConfig::Instance
[7 May 21:54:09] .....<-- MigrateConfig::Instance
[7 May 21:54:09] .....--> MigrateConfig::GetConfigSet
[7 May 21:54:09] .....<-- MigrateConfig::GetConfigSet
[7 May 21:54:09] .....--> IsVersionSetValid
[7 May 21:54:09] .....<-- IsVersionSetValid
[7 May 21:54:09] ....<-- DbUpgrader::GetSourceHfaLevel
[7 May 21:54:09] ....--> UpgradeMacroReplacer::Instance
[7 May 21:54:09] ....<-- UpgradeMacroReplacer::Instance
[7 May 21:54:09] ....--> CanonicalizePath
[7 May 21:54:09] [CanonicalizePath] Canonicalizing path '/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/bin/'
[7 May 21:54:09] [CanonicalizePath] Resulting path: '/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/bin/'
[7 May 21:54:09] ....<-- CanonicalizePath
[7 May 21:54:09] ....--> DbUpgrader::GetDestMainDbVersion
[7 May 21:54:09] .....--> MigrateConfig::Instance
[7 May 21:54:09] .....<-- MigrateConfig::Instance
[7 May 21:54:09] .....--> MigrateConfig::GetConfigSet
[7 May 21:54:09] .....<-- MigrateConfig::GetConfigSet
[7 May 21:54:09] .....--> GetVersionString
[7 May 21:54:09] ......--> IsVersionSetValid
[7 May 21:54:09] ......<-- IsVersionSetValid
[7 May 21:54:09] .....<-- GetVersionString
[7 May 21:54:09] ....<-- DbUpgrader::GetDestMainDbVersion
[7 May 21:54:09] ...<-- DbUpgrader::BuildMainDbUpgradeCommand
[7 May 21:54:09] ...--> DbUpgrader::ExecuteCpdb
[7 May 21:54:09] [DbUpgrader::ExecuteCpdb] Executing cpdb using the following arguments: "/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/bin/cpdb" up --no-ngm-db-drop --no-ngm-stop --src_version 6.0.4.0 --src_hfa_level 30 --src_path "/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/tmp//export_db/main_db/" --db_path "/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/conf/" --default_path "/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/conf/defaultDatabase" --db_type cma --db-migration
[7 May 21:54:09] [runShellCommand] Executing command: '"/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/bin/cpdb" up --no-ngm-db-drop --no-ngm-stop --src_version 6.0.4.0 --src_hfa_level 30 --src_path "/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/tmp//export_db/main_db/" --db_path "/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/conf/" --default_path "/opt/CPmds-R80.20/customers/cma_TEST_IMPORT/CPsuite-R80.20/fw1/conf/defaultDatabase" --db_type cma --db-migration'
[7 May 21:54:10] [runShellCommand] Execution result: 1, exit code: -1
[7 May 21:54:10] [DbUpgrader::ExecuteCpdb] ERR: cpdb completed with error code '-1'
[7 May 21:54:10] ...<-- DbUpgrader::ExecuteCpdb
[7 May 21:54:10] ..<-- DbUpgrader::UpgradeMainDatabase
[7 May 21:54:10] [DbUpgrader::exec] ERR: Failed to upgrade main database
[7 May 21:54:10] .<-- DbUpgrader::exec
[7 May 21:54:10] <-- ConditionalExecutor::exec
[7 May 21:54:10] [ActivitiesManager::exec] ERR: Activity 'ConditionalExecutor' failed
[7 May 21:54:10] [ProgressUpdater::UpdateProgressToGaia] Progress Updated to '70.2128
[7 May 21:54:10] [ActivitiesManager::exec] WRN: Activities execution finished with errors
[7 May 21:54:10] [ActivitiesManager::exec] WRN: Activities 'ConditionalExecutor' have failed
[7 May 21:54:10] [ActivitiesManager::exec] Designated exit code is 1
[7 May 21:54:10] --> CleanupManager::Instance
[7 May 21:54:10] <-- CleanupManager::Instance
[7 May 21:54:10] --> CleanupManager::DoCleanup
[7 May 21:54:10] [CleanupManager::DoCleanup] Starting to perform cleanup
[7 May 21:54:10] .--> CmaImportFailureMarker::exec
[7 May 21:54:10] [CmaImportFailureMarker::exec] Checking if cleaner is active
[7 May 21:54:10] [CmaImportFailureMarker::exec] Cleaner is active, starting cleanup
[7 May 21:54:10] [CmaImportFailureMarker::exec] Checking migrate's exit code
[7 May 21:54:10] [CmaImportFailureMarker::exec] Migration had failed, creating a marker file
[7 May 21:54:10] ..--> UpgradeMacroReplacer::Instance
[7 May 21:54:10] ..<-- UpgradeMacroReplacer::Instance
[7 May 21:54:10] [CmaImportFailureMarker::exec] Created a marker file
[7 May 21:54:10] .<-- CmaImportFailureMarker::exec
[7 May 21:54:10] [CleanupManager::DoCleanup] Completed the cleanup
[7 May 21:54:10] <-- CleanupManager::DoCleanup

Logs from file: upgrade_log-<DATE>.log:

Click to Expand

[112235 4067981120]@R8020_MDS[7 May 23:42:20] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: Fault returned from remote server: SOAP 1.1 fault: SOAP-ENV:Server[no subcode]
"An internal error has occurred."
Detail:

[112235 4067981120]@R8020_MDS[7 May 23:42:20] CCkpDbObjectsCWsRemoteImpl::LogWriteFailure: ERROR: Failed to update 1 objects:
[112235 4067981120]@R8020_MDS[7 May 23:42:20] =================LIST START=================
[112235 4067981120]@R8020_MDS[7 May 23:42:20] Uid: '{5570CCBF-8889-45CD-ADFF-E478F1B6438E}', Name: 'configurable_attributes_obj'
[112235 4067981120]@R8020_MDS[7 May 23:42:20] ==================LIST END==================
2019.05.07_23:42:20 - Failed to update remote server with object 'configurable_attributes_obj'
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - failed to write object to file
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveObject] ERROR: Failed to update the DB. Object 'configurable_attributes_obj' (Table 'uncovered_global_props' / Class 'configurable_attributes') will not be added to the DB.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveObject] ERROR: CPMI Error: 0x8003001D (Could not access file for write operation). Update error: An internal error has occurred..
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] Failed to add object 'configurable_attributes_obj' to table 'uncovered_global_props'.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] Ended saving table 'uncovered_global_props'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] ERROR: Failed to add 1 objects to table 'uncovered_global_props' in the DB. The resulting DB will be corrupted.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] Started saving table 'update_times_web_queries'.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] Ended saving table 'update_times_web_queries'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] Started saving table 'urlf_groups'.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] Ended saving table 'urlf_groups'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] [CUpgCPMIInterface::SaveTable] Started saving table 'user_check_interactions'.
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:20] CCkpDbObjectsCWsRemoteImpl::Write: session = c1uRhuWCu0a6zMfVr_Aa5B5_1H-VuTjovpEmuO8QaCQ
[112235 4067981120]@R8020_MDS[7 May 23:42:22] [CUpgCPMIInterface::SaveTable] Ended saving table 'user_check_interactions'. Saved '21' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:22] [CUpgCPMIInterface::SaveTable] Started saving table 'users'.
[112235 4067981120]@R8020_MDS[7 May 23:42:22] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:22] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:22] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:22] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:22] CSrvObj::Update - Diff found (or contains 'object_permissions'), continue to update...
[112235 4067981120]@R8020_MDS[7 May 23:42:22] CCkpDbObjectsCWsRemoteImpl::Write: session = c1uRhuWCu0a6zMfVr_Aa5B5_1H-VuTjovpEmuO8QaCQ
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'users'. Saved '5' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'voip_objects'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'voip_objects'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'vs_slot_objects'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'vs_slot_objects'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'vs_vsx_licenses'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'vs_vsx_licenses'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'web_authority_URLs'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'web_authority_URLs'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'web_authority_allow_rules'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'web_authority_allow_rules'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'web_authority_effect_rules'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'web_authority_effect_rules'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'web_authority_must_rules'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'web_authority_must_rules'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'web_sites'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'web_sites'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'wf_added_objects'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'wf_added_objects'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'wf_configuration'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'wf_configuration'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'wf_object_changes'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'wf_object_changes'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'wf_removed_objects'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'wf_removed_objects'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'wf_sessions'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'wf_sessions'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Started saving table 'wf_tagged_objects'.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::SaveTable] Ended saving table 'wf_tagged_objects'. Saved '0' objects.
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::RunUpgradeCommands] Starting to send upgrade command
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::RunUpgradeCommands] Sending upgradeCommand fwset size 113
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand start ====
[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand fwset header (
:AdminInfo (
:chkpf_uid ("{72BABBE4-E3EF-714C-8CA5-19D54EBFD516}")
:ClassName (replace_uids_command)
)
)

[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::RunUpgradeCommands] upgradeCommand full fwset (
:AdminInfo (
:chkpf_uid ("{72BABBE4-E3EF-714C-8CA5-19D54EBFD516}")
:ClassName (replace_uids_command)
)
)

[112235 4067981120]@R8020_MDS[7 May 23:42:23] [CUpgCPMIInterface::RunUpgradeCommands] ==== upgradeCommand end ====
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CUpgCPMIInterface::Save] ERROR: Failed to update some of the tables. The resulting DB will be corrupted.
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CUpgradeMgr::PerformCPMIUpdate] WARNING: Failed to save the DB.
[112235 4067981120]@R8020_MDS[7 May 23:42:26] CNgmProxyImpl::init: this = 0xa154f88
[112235 4067981120]@R8020_MDS[7 May 23:42:26] CNgmProxyImpl::init: this = 0xa0a5950
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CUpgradeMgr::PerformUpgrade] ERROR: Failed to perform cpmi update
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [writeUpgradeResult] The path to result file is: '/opt/CPmds-R80.20/customers/cma_TEST_GWC/CPsuite-R80.20/fw1/log/upgrade_result'
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [writeUpgradeResult] Wrote the following to the result file: '80004005'
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CCPDBMain::Run] cpdb ended with result '0x80004005' (Unspecified error).
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 03927bfb-9919-43ee-9548-7468c4e86320 to post_objects
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CCPDBMain::Run] Setting upgrade phase in CPM Server
[112235 4067981120]@R8020_MDS[7 May 23:42:26] [CCPDBMain::setNgmUpgradeStatus] Setting upgrade phase for domain 03927bfb-9919-43ee-9548-7468c4e86320 to finalize_domain
[112235 4067981120]@R8020_MDS[7 May 23:42:34] [main] FATAL ERROR: Operation failed.
[112235 4067981120]@R8020_MDS[7 May 23:42:34] [UpgradeToRenaissanceInfra::shutdownJavaUpgradeServer] Sending shutdown request to java upgrade server

---

@IrinaK wrote:

Hi Jozko,

can you send the errors you see in import procedure, please? you can contact me at irinamot@checkpoint.com.

regards,

Irina.

---

[JozkoMrkvicka]

So the whole issue was that Global Policy was not imported before importing CMA.

Once I imported Global Policy, the CMA can be imported.

Why such a condition is not part of pre-verifier tool ? Simply check if imported CMA has Global Policy inside. If Global Policy is already imported --> proceed with import. If Global Policy is not imported --> error and instruct the user how to import Global Policy first.

[Vladimir]

There is another use case that actually requires different behavior:

I have one client that wanted to upgrade and move different CMAs from multiple MDS to a new target R80.20 MDS.

Both, global objects and global policies were present in the source environments.

In this case, "deglobafication" of the source CMAs and objects would be in order.

Provided that there are no collisions on the target R80.20, this would've been possible.

As is, I have recommended to move CMAs around in R77.30 before cloning and upgrading the resultant composite MDS to R80.20.

I am still waiting on the news of DB split per CMA, before being comfortable recommending move to R80.++ MDS to my clients.

[Vladimir]

We've performed about 20 upgrades from 77.XX to R80.10 and R80.20. So far, knock on wood, there were no issues that we have encountered and were not able to solve.

Few worth mentioning were:

1. In the presence of the APPC/URLF layer, had to enable HTTP and SSH to the gateways in it in addition to the implied rules.

2. Issue resolved in JHFA 73 (ongoing), preventing smartview access from non-admin account.

3. Some tricky sequencing for a client who had the Management HA with SmartEvent on one of the members (not recommended by Check Point). It was their setup in R77.30 and we had to replicate same in R80.20.

4. Phantom expiring IPS contracts. For some reason, just this blade claims to be activated with evaluation license and even though valid license was applied, the self-popped-up evaluation license has to run a day past its expiration, before production license takes effect. There were no workarounds for that.

5. Few problems caused by fragmentation errors, such as described in "Traffic sent over a VPN tunnel does not reach its destination because SecureXL does not start fragm...

Besides these, I've turned down two very large clients requesting large scale upgrade for MDS managed global infrastructures. These cases were difficult to begin with and clients didn't want to go through the preliminaries of performing lab preps and staged migrations.

To the best of my knowledge, they were still on R77.30 a year later after choosing to engage CP ProServe directly.

I have to say that the current means of MDS HA + MLM upgrades could use some improvement. The "unplug all", replace with new install the policies, pray while it is happening and rollback to the old if it will not work, does not inspire confidence, especially if there were global policies and objects in the source environments.

Overall, the experience is a definite improvement and is getting progressively better.

Regards,

Vladimir

[JozkoMrkvicka]

I am simply testing all possible upgrade methods available out there for future use in production.

At this very moment, I am trying to export single CMA from R77.30 and import it to R80.20. It turned out, that my issue is with some relation to Global Policy. Exported CMA has Global Policy on it. During import, I get a warning about that, but as this is just warning, all should be fine.... WRONG !

Once I tried to remove Global Policy from this CMA, the import is possible. 

Of course, I tried to migrate Global Policy into R80.20 before importing CMA. Still the same - import failed.

UPDATE: Once I imported Global Policy on freshly installed MDS, CMA import was done successfully. 

I hope that the upgrade tools were improved within R80.30, as I found many issues with it.

Example 1:
The upgrade from R77.30 MDS to R80.20 MDS using "installer upgrade" is successful. But in case you will try to export/import R77.30 CMA and import it into R80.20 MDS, you will get an error that Global Use for Gateway is enabled and MUST be disabled before import to R80.20. This condition wasn't found by pre-upgrade verifier in case of "installer upgrade" method.

Example 2:
Once the CMA is imported from R77.30 to R80.20, you need to manually add "Trusted Clients" to the domain, as after import there is nothing and you will get an error that "Unauthorized client":