UNABLE TO UPGRADE MDM R80.20 TO R80.40

[n4gu41]

Hi all,

I would like to expose in checkmates my case to see if somebody can shed some light in an issue im encountering in the upgrade process from R80.20 to R80.40. 

Initial statements

1. The setup is the following: MDM Primary on R80.20 with JHFA T147 / JHFA T149 and R80.40 upgrade tools installed.
2. The machines im using are VMs running in VMware workstation with 600Gb os vHDD (LVM), 4 cores and 16Gb RAM.
3. Upgrade process starting R80.10 is using a Major upgrade approach making use of the un-partioned space to create Autosnahost
4. The upgrade process is ending miserably with the following cryptic error after the iniatil reboot after the first upgrade process phase ended:

data=<?xml version="1.0" ?>
<request>
<!-- User information -->
<requestVersion>1.0.0</requestVersion>
<ck>CK-PNP,6016F82428C1</ck>
<hmac>SRedumRLv6Yj/ic9QUdXsQ==</hmac>
<gateway_id>22af0673-be59-4847-ac56-8f41dd8db7db</gateway_id>
<HOTFIX></HOTFIX>
<!-- Current machine information -->
<serviceVersion>1889-megatron</serviceVersion>
<clientVersion>-1.-1.-1</clientVersion>
<clientHFA></clientHFA>
<osVersion>6.0.5.1</osVersion>
<osHFA></osHFA>
<os>Gaia</os>
<configuration>MDS</configuration>
<machineModel>VMware</machineModel>
<!-- Operation information -->
<operation name="Install" pkgName="Major" pkgKey="CheckPoint#Major#All#6.0#5#1#R80.40_T294" size="3937096045" status="Failed" duration="" last_error="Failed on importing MDS settings to destination." timestamp="Thu Apr 16 23:05:01 2020" installed_hfs="" last_error_code="MSG_MAJOR_ERROR_MGMT_IMPORT" last_error_category="UNKNOWN" />
</request>

5. Ive seen issues as well on the cpm file:

[Expert@BT--MDM01:0]# cat /opt/CPsuite-R80.20/fw1/log/cpm.elg | grep -i "ERROR internal.operation.OperationSvcImpl"
16/04/20 23:09:55,687 ERROR internal.operation.OperationSvcImpl [taskExecutor-1]: caught exception "Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.0.v20140809-296a69f): org.eclipse.persistence.exceptions.DatabaseException
16/04/20 23:09:55,688 ERROR internal.operation.OperationSvcImpl [taskExecutor-1]: caught exception "Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.0.v20140809-296a69f): org.eclipse.persistence.exceptions.DatabaseException
16/04/20 23:10:02,490 ERROR internal.operation.OperationSvcImpl [taskScheduler-5]: caught exception "Object not found - Failed to find object ID 'a4ec28e6-f193-4eba-8675-ea6a1f9cef2e' of class com.checkpoint.management.dlecommon.ngm_api.WorkSession" from class com.checkpoint.management.coresvc.ObjectNotFoundException
16/04/20 23:11:15,553 ERROR internal.operation.OperationSvcImpl [qtp2124734651-79]: caught exception "Tried to persist object 7e5c230a-6a18-4e3b-9f8e-dc92c59fe46d with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 20dac1e0-fcf5-9b4c-bba3-d9251ecfcb72" from class java.lang.IllegalArgumentException
16/04/20 23:11:15,559 ERROR internal.operation.OperationSvcImpl [qtp2124734651-79]: caught exception "Tried to persist object 7e5c230a-6a18-4e3b-9f8e-dc92c59fe46d with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 20dac1e0-fcf5-9b4c-bba3-d9251ecfcb72" from class java.lang.IllegalArgumentException
16/04/20 23:11:24,604 ERROR internal.operation.OperationSvcImpl [qtp2124734651-113]: caught exception "Tried to persist object c259cc28-a6a4-4dfa-913f-0f396f22d4c1 with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 16220252-1cee-3546-bd7d-727f3a5d37d8" from class java.lang.IllegalArgumentException
16/04/20 23:11:24,607 ERROR internal.operation.OperationSvcImpl [qtp2124734651-113]: caught exception "Tried to persist object c259cc28-a6a4-4dfa-913f-0f396f22d4c1 with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 16220252-1cee-3546-bd7d-727f3a5d37d8" from class java.lang.IllegalArgumentException
16/04/20 23:12:14,183 ERROR internal.operation.OperationSvcImpl [qtp2124734651-113]: caught exception "Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.0.v20140809-296a69f): org.eclipse.persistence.exceptions.DatabaseException
16/04/20 23:12:14,475 ERROR internal.operation.OperationSvcImpl [qtp2124734651-114]: caught exception "Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.0.v20140809-296a69f): org.eclipse.persistence.exceptions.DatabaseException
16/04/20 23:12:14,650 ERROR internal.operation.OperationSvcImpl [qtp2124734651-110]: caught exception "Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.0.v20140809-296a69f): org.eclipse.persistence.exceptions.DatabaseException
16/04/20 23:12:17,366 ERROR internal.operation.OperationSvcImpl [qtp2124734651-111]: caught exception "Tried to persist object 7e5c230a-6a18-4e3b-9f8e-dc92c59fe46d with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 20dac1e0-fcf5-9b4c-bba3-d9251ecfcb72" from class java.lang.IllegalArgumentException
16/04/20 23:12:17,367 ERROR internal.operation.OperationSvcImpl [qtp2124734651-111]: caught exception "Tried to persist object 7e5c230a-6a18-4e3b-9f8e-dc92c59fe46d with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 20dac1e0-fcf5-9b4c-bba3-d9251ecfcb72" from class java.lang.IllegalArgumentException
16/04/20 23:14:11,908 ERROR internal.operation.OperationSvcImpl [qtp2124734651-75]: caught exception "Tried to persist object 7e5c230a-6a18-4e3b-9f8e-dc92c59fe46d with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 26fdd56a-3338-dd4f-b39a-ccff249da537" from class java.lang.IllegalArgumentException
16/04/20 23:14:11,909 ERROR internal.operation.OperationSvcImpl [qtp2124734651-75]: caught exception "Tried to persist object 7e5c230a-6a18-4e3b-9f8e-dc92c59fe46d with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 26fdd56a-3338-dd4f-b39a-ccff249da537" from class java.lang.IllegalArgumentException
16/04/20 23:14:12,038 ERROR internal.operation.OperationSvcImpl [qtp2124734651-110]: caught exception "Tried to persist object c259cc28-a6a4-4dfa-913f-0f396f22d4c1 with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 26fdd56a-3338-dd4f-b39a-ccff249da537" from class java.lang.IllegalArgumentException
16/04/20 23:14:12,039 ERROR internal.operation.OperationSvcImpl [qtp2124734651-110]: caught exception "Tried to persist object c259cc28-a6a4-4dfa-913f-0f396f22d4c1 with domain a0eebc99-afed-4ef8-bb6d-fedfedfedfed while active domain is 26fdd56a-3338-dd4f-b39a-ccff249da537" from class java.lang.IllegalArgumentException

I have been working long last weeks to perform this upgrade but with no success. I wonder if somebody could help in here.

Thanks in advance

[_Val_]

Did you run pre-upgrade verifier before going for the upgrade? If you did, and no errors are reported, open a TAC case.

[n4gu41]

Hi Val,

Thanks for your input. Much appreciated. Below the answers to your questions:

1. The TAC case has been opened already non after researching and dealing with the issue:

2. The pre_upgrade_verifier tools its seems to be only useful in a major upgrades like from R77.30 to R80.10 according to the Install and Upgrade guide:

pre_upgrade_verifier

Analyzes compatibility of the currently installed configuration with the version, to which you upgrade.
It gives a report on the actions to take before and after the upgrade.
Note - This tool is required only when you upgrade from R77.30 (and lower) version to R80.40.

3. Seems that the upgrade process is already doing a pre-checks before even trying to upgrade to certify the intented upgrade can be peformed.

4. The issue is the following: HV000028 Unexpected exception during isValid call (see attached image taken from the upgrade report)

Thanks in advance

[n4gu41]

According to the upgrade report file the issue encountered is after trying to import User Domains to the PostgreSQL DB, system Data, IPS Data, APLC Data and Logging Data are imported successfully. After this stage is trying to import the data for the user domains moment where the import is falling. So my suspicion is that something is corrupted at this stage or there is something that cannot be imported due other factors which i dont know. 

[n4gu41]

[Expert@BT--MDM01:0]# /var/log/CPda/metadata/CheckPoint#Major#All#6.0#5#1#R80.40_T294/tmp/pre_upgrade_verifier

This is Check Point Pre-Upgrade Verifier for version R80.40.

Usage: pre_upgrade_verifier -p SecurityManagementPath -c CurrentVersion -t TargetVersion [-u | -a][-f FileName] [-w]
Or: pre_upgrade_verifier -p SecurityManagementPath -c CurrentVersion -i [-f FileName] [-w]
-p Path of the installed Security Management Server (FWDIR).
-c Currently installed version.
-t Target version.
-i Check originality of Inspect files only.
-u Perform plug-in related checks.
-a Skip main train version checks, perform plug-in related checks only.
-f Output in file.
-w Web format file.

where the Currently installed version is one of the following:
R75.40 (aliases: R75.40_R75.40, 6.0.2.5)
R75.40VS (aliases: R75.40VS_R75.40VS, 6.0.3.0)
R76 (aliases: R76_R76, 6.0.3.5)
R77 (aliases: R77_R77, 6.0.4.0)

where the Target version is one of the following:
R75.40 (aliases: R75.40_R75.40, 6.0.2.5)
R75.40VS (aliases: R75.40VS_R75.40VS, 6.0.3.0)
R76 (aliases: R76_R76, 6.0.3.5)
R77 (aliases: R77_R77, 6.0.4.0)
R80 (aliases: R80_R80, 6.0.4.8)
R80.20 (aliases: R80.20_R80.20, 6.0.4.9)
R80.30 (aliases: R80.30_R80.30, 6.0.5.0)
R80.40 (aliases: R80.40_R80.40, 6.0.5.1)

[Eran_Habad]

Hello,

My name is Eran and I'm a manager in the R&D. My team is responsible for the upgrade of the Management server. I'm sorry about the bad experience so far, our team from Check Point has already approached you offline to try and assist, and we'll do our best to promote the issue swiftly together. Let's continue the discussion by mail and via the SR, and I hope for a quick resolution for this issue to enable a fast and efficient upgrade to R80.40.

Regards, Eran.

[_Val_]

Thanks for your prompt action, @Eran_Habad 

[n4gu41]

Thanks Eran for your effective and swift response. Thanks Valerie as well for your inputs always are valuable. You have been doing so much for the community since long time ago.

[Itai_Minuhin]

Hi all,

My name is Itai, I'm a Team Leader in Check Point R&D, responsible for a the management server upgrade.

We looked into this upgrade issue tonight, we believe we've found the problem, and hope to update very soon on the solution. 

Thanks to our updatable upgrade code mechanism, which allows faster delivery of fixes and enhancements, we will upload an updated upgrade tools package once the fix is ready. For the benefit of all customers.  

Such upgrade packages are automatically downloaded from the Download Center to online environments, so it will be already installed when one wants to upgrade. It is also available for offline environments, and requires to download latest upgrade package from the Download Center.

Regards,

Itai

[Magnus-Holmberg]

Any updates on this? now when R80.40 is the recommended

[Itai_Minuhin]

Hi, 

This issue has been fix a while ago.

To enjoy this fix, and more fixes and enhancements,  please download and install the latest upgrade tools package from sk135172.

Thanks, 

Itai