Locked out after Gateway upgrade from 80.40 to 81.10

[George_Sas]

Hi guys.

I am facing again strange issue when upgrading the gateway to 81.10

I upgraded the management server to 81.10 and also the small appliances and all seems ok on this side.
Now I am trying to upgrade 2 x 6600 Gateways that are in a Cluster XL configuration.

I do an in place upgrade via CPUSE and the upgrade process itself of the passive member goes OK, but after the upgrade I loose access to the gateway via SSH or Gaia.

[2023-06-14 - 10:33:51][26105 32165]:BLINK::skipping Major_PostInstallScript
[2023-06-14 - 10:33:51][26105 32165]:BLINK::skipping Major_LVMRenameAndSnapshotDetails
[2023-06-14 - 10:33:51][26105 32165]:Running Command || Description: Success DA Post install : Writing response
[2023-06-14 - 10:33:51][26105 32165]:Running Command || Command: echo -n "0" > /var/log/blink/processOutput
[2023-06-14 - 10:33:51][26105 32165]:Finished executing Blink post actions sequence
[2023-06-14 - 10:33:51][26105 32165]:------ Post-Install Script:  ------
[2023-06-14 - 10:33:51][26105 32165]:Lock release successes
[2023-06-14 - 10:33:52][26105 32165]:Blink Install Info || Completed : false || State : run_post_script || Status Description : Run post.sh script
[2023-06-14 - 10:34:06][26105 29768]:Return code: 1
[2023-06-14 - 10:34:06][26105 32165]:Blink Install Info || Completed : true || State : finish_message || Status Description : The installation has finished successfully, reboot is suspended, perform it manually in order to finish the installation.
[2023-06-14 - 10:34:06][26105 32165]:------ Finishing:  ------
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for MGMT//6.0//HOTFIX_R80_40_JUMBO_HF_MAIN//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for FW1//6.0//HOTFIX_R80_40_JUMBO_HF_MAIN//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for PPACK//6.0//HOTFIX_R80_40_JUMBO_HF_MAIN//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CVPN//6.0//HOTFIX_R80_40_JUMBO_HF_MAIN//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_R80_40_JUMBO_HF_MAIN//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_R80_40_MAAS_TUNNEL_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_DEP_INSTALLER_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_INFRA_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_ESOD_SCANNER_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_ESOD_CSHELL_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_HCP_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_GOT_TPCONF_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_CORE_FILE_UPLOADER_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_CPSDC_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_GENERAL_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_CPOTELCOL_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_CPVIEWEXPORTER_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:07][26105 32165]:Found previous installed build for CPUpdates//6.0//BUNDLE_ENDER_V17_AUTOUPDATE//PreviousInstalled
[2023-06-14 - 10:34:13][26105 32165]:crsXml file: /opt/CPda/repository/CheckPoint#Major#All#6.0#5#3#BLINK_R81_10_T335_JHF_T95_GW/crs.xml doesn't exist.
[2023-06-14 - 10:34:13][26105 32165]:------ Going to reboot:  ------
[2023-06-14 - 10:34:28][26105 32165]:------  ------

I only have access to this gateway via Lights Out Management.

If I log in via LOM and try to ssh to localhost I can fine ssh , telnet to localhost on https port also ok.

If i try to fetch the policy I get an error (even if I modified the cluster version in the management console ) :

Fetching FW1 Security Policy From: xx.xx.xxx.xx

 Management rejected fetch for this module - version matching problem.
 Policy Fetch Failed
 Failed to fetch policy from masters in masters file

Fetching Threat Prevention Security Policy From: xx.xx.xxx.xx

Connectivity out of the upgraded gateway is no problem , I can ping , ssh , ftp etc from the upgraded gateway but nothing towards it.

So I end up reverting to the snapshot as I have been struggling with this for the past 2 days.

Any ideas ?

Thank you in advance.

[_Val_]

It seems that on the management side your GW version is still the old one, hence the policy fetch fails. Go to SmartConsole, update your GW version, and push policy, that should resolve the issue.

[George_Sas]

Hi Val... I did change the version in the SmartConsole ... and the cluster will "see" one gateway as 80.40 and the other as 81.10 .. but I will try again now and do also a policy push even if I remember I tried that.

[Ruan_Kotze]

Hi George,

It wouldn't hurt to enable MVC and try pushing policy once: 'set cluster member mvc on'.  Once you've had a successful policy push you can disable it again.

Thanks,
Ruan

[George_Sas]

Would that be on the Active node or on both ?

My environment is in production and would not wanna risk a gateway without policy comming up..

[George_Sas]

Also when "fw unloadlocal " I still can't Ping or SSH the upgraded gateway ...

[George_Sas]

I tried again now ... changed version in SmartConsole and tried to push policy , which will of course fail as one node is 80.40 and one 81.10 (but down).

When I try to fetch , I get same error as before :

Fetching FW1 Security Policy From: xx.xx.xx.145

Management rejected fetch for this module - version matching problem.
Policy Fetch Failed

[George_Sas]

Small update .... I am able to SSH on the Gateway on the External interface... so something goes wrong on interface configuration after upgrade or some anti spoofing.
Digging ...

[Ruan_Kotze]

That's possible, anti-spoofing is not unloaded as part of 'fw unloadlocal'.

You can test by disabling AS on the fly, replace 0 with 1 to enable again.

• fw ctl set int fw_antispoofing_enabled 0
• fw ctl set int sim_anti_spoofing_enabled 0 -a

[George_Sas]

Seems to be a routing problem even if everything works fine on 80.40 after upgrade to 81.10 some route changes or the way Checkpoint routes the traffic ...

I changed the management IP address of the upgraded member in and I am now able to push policy on it. Also ssh and Gaia now are available on that IP...

I'm having our core networking guys take a look at the core switches and will get back with an update.

[AkosBakos]

Hi Geroge_Sas

After you upgraded the SMartCenter to R81.10 did you push "install database" on the SmartCenter? Sometimes it is needed to do it manually

A

[AkosBakos]

And one more, as I see you didn't set the cluster version to R81.10.

1. Switch it to R81.10
2. unload the policy on the newly upgraded gw
3. Push the policy to the gateway. IMPORTANT: remove the tik from here:
   
4. The policy will be installed on the  R81.10 gateway
5. After you will be able to continue the upgrade procedure

Don't forget: the full connectivity upgrade is not supported on cluster with two members. #mcv has limitations (around what to sync)