Security Gateway upgrade when Gaia Management server is behind the same gateway

[Roshan_Sinha]

Hi Team,

Can you please suggest the best approach when we have to upgrade Gaia OS for security gateway and we have single Gaia Mgmt server connected to the same gateway. as we if upgrade it we will not be able to push the policy due to Version mismatch.

The another option , if i change the version on Gaia Mgmt server before upgrade, and if the upgrade fails , i will not be able to restore the gateway and push policy as Gaia mgmt server will not be available if Gateway is down.

Kindly suggest the best approach for upgrade. Let me know if any details require on the same.

[_Val_]

Do I understand correctly, you tried upgrading GW first and only then MGMT server?

Please elaborate

[Roshan_Sinha]

No, Mgmt server is already upgraded, we want to upgrade now Gateway where Mgmt server is connected directly with the same gateway, Means if Gateway is down, we wont be able to access Mgmt server. 

Note: Here Gaia Mgmt server is on VM and Gateway is Hardware Appliance.

[_Val_]

Bad architecture. You should have at least one GUI client machine which does not cross GWs managed by the same MGMT.

Put a windows VM on the same network where MGMT is, and push policy on the upgraded GW

[RamGuy239]

There are multiple ways to handle this.

Make sure that you have a policy on the gateway that ensures that you can reach the gateway using SSH throughout the entire process. Also, make sure that you can reach the management server using SSH from the gateway. Then you simply connect to the gateway using SSH. Do the in-place upgrade. The gateway should be booting with the initial policy after the upgrade so you are still able to reach it using SSH.

Then you utilise ssh from the gateway to reach the management and you can utilise mgmt_cli to push policy from the management to the upgraded gateway. If you already changed the gateway object so it has the correct version using SmartConsole beforehand all you have to do is to push policy using mgmt_cli which is quite simple:

mgmt_cli install-policy policy-package "NAME-OF-THE-POLICY-PACKAGE" access true threat-prevention true

And you're done.

If you are upgrading to R81 or R81.10 you can utilise the version upgrade feature directly in Smart Console. Just right-click the gateway --> actions --> version upgrade and do it through the wizard in Smart Console. You will obviously lose track of what is going on you will lose access to Smart Console when the gateway is upgrading. But this is simply using central deployment tool so as soon as you have initiated the process it doesn't matter if you lose access to Smart Console or not. Everything is running directly between the management server and the gateway using cdt so unless something happens during the process causing it to fail it will transfer the cpuse package or blink package of your choosing to the gateway, it will do the upgrade, it will automatically switch the gateway object in the management database to the new version and it will automatically push policy. So you should be back up and running. This is risky of course as you lose access so you don't really know what is happening and if something fails it might be difficult for you to understand what is going on besides looking at various .elg files through SSH on the gateway and management server.

This whole thing can be done even if you are not going to R81 or R81.10 by using the central deployment tool the old way (not through Smart Console). It's a more manual process but it's quite easy. Just follow sk111158.

You even have a third option. As long as you have SSH to the gateway you can temporarily enable AllowTcpForwarding within /etc/ssh/sshd_config on the gateway. Then you can utilise tcp forwarding within Putty to simply forward TCP-18190 (CPMI) and TCP-19009 (CPM) through your gateway and towards your management server. As long your gateway is allowed to run SSH to the management server you can then run Smart Console towards localhost / 127.0.0.1, Putty is then forwarding the traffic to your management server via the gateway. Just remember to disable AllowTcpForwarding afterwards as it's a major security risk to have it enabled. This has saved me so many times during upgrades, especially when going from R77 to R80 as many customers didn't have control connections activated under global properties / implied rules and they didn't allow for CPM-19009 that was added in R80 so we ended up with no Smart Console after the upgrade.

[Roshan_Sinha]

hi Ram,

Thanks for sharing multiple option, If i go with Option 1 where i can allow SSH between Gaia Mgmt server and Gateway..

is it possible to change the gateway object correct version  by login into Mgmt server via SSH ?

[RamGuy239]

Hi, @Roshan_Sinha 

Just make sure that you have a rule in your security policy:

src: gateway, dst: management, service: ssh, action: allow
src: management, dst: gateway, service: ssh, action: allow

This will most likely be allowed by default through "Accept control connections" under Global Properties in Smart Console. This makes it so that implied rules (rule #0) makes sure that traffic between the gateway and management server over the required ports and services are allowd in rule #0.

I just like to be sure before doing upgrades by having a manual rule making sure that I will have the correct access during the upgrade.

After doing the upgrade of the gateway it will boot up with initial policy. Initial policy will allow for SSH.

You should be able to change the version using mgmt_cli as well. Haven't done this before myself but I just tested it out in my LAB and it worked:

For a single gateway:
mgmt_cli set simple-gateway name "NAME-OF-THE-GATEWAY-OBJECT-IN-SMARTCONSOLE" version R81.10

For a cluster:
mgmt_cli set simple-cluster name "NAME-OF-THE-CLUSTER-OBJECT-IN-SMARTCONSOLE" version R81.10

So if you make sure that you have SSH access. You would simply start the upgrade on the gateway through SSH:

CLISH

installer download Check_Point_R81.10_T335_Fresh_Install_and_Upgrade.tar
installer verify Check_Point_R81.10_T335_Fresh_Install_and_Upgrade.tar
installer upgrade Check_Point_R81.10_T335_Fresh_Install_and_Upgrade.tar

Once it has upgraded and rebooted. You reconnect to the gateway using SSH:

Expert mode

ssh username@IP-OF-YOUR-MANAGEMENT
mgmt_cli set simple-gateway name "NAME-OF-THE-GATEWAY-OBJECT-IN-SMARTCONSOLE" version R81.10
mgmt_cli install-policy policy-package "NAME-OF-THE-POLICY-PACKAGE" access true
mgmt_cli install-policy policy-package "NAME-OF-THE-POLICY-PACKAGE" threat-prevention true

And you should be done.

[RamGuy239]

All my examples are using R81.10. You will obviously have to change it so it corresponds with whatever version you are going with.

[Roshan_Sinha]

Thanks Ram for detailed steps... 

just one query, when you say "After doing the upgrade of the gateway it will boot up with initial policy" , will it be same policy which was there earlier before upgrade ?.. please confirm. 

Sorry for this query, as i'm little new to the checkpoint and this type of scenario.

[_Val_]

No, GW will not assume the same policy after upgrade. The policy package has to be compiled and pushed to the GW with a correct version. Hence the suggestion to change GW version and push policy via mgmt_cli above

[RamGuy239]

Hi, @Roshan_Sinha.

Initial Policy is not the same as the current policy installed on the gateway before the upgrade. Policy packages are compiled based on whatever version of Check Point GAiA that is installed on the gateway. So after a upgrade it will boot up with initial policy and it's waiting for a policy push.

This is why we have to change the gateway version and do the policy push using mgmt_cli.

This is what Check Point says about the initial policy:

The Initial Policy
Until the Security Gateway administrator installs the Security Policy on the Security Gateway for the first time, security is enforced by an Initial Policy.

The Initial Policy operates by adding the predefined implied rules to the Default Filter policy.

These implied rules forbid most communication, yet allow the communication needed for the installation of the Security Policy.

The Initial Policy also protects the Security Gateway during Check Point product upgrades, when a SIC certificate is reset on the Security Gateway, or in the case of a Check Point product license expiration.


As the initial policy makes sure that implied rules are in place you will be able to access the gateway using SSH etc after the upgrade has succeeded. Making you capable of running ssh from the gateway to the management server so you can run the mgmt_cli commands to change the gateway version within the database and for you to install policy using mgmt_cli. Then you should be back to normal.

[_Val_]

@RamGuy239 nice trick here 🙂

Still I insist, having MGMT access through a managed GW only without any out of band option is a bad practice. Any mistake with a policy, or GW problem, you lose control over the whole system. 

[RamGuy239]

@_Val_ 

I fully agree. This would be considered best-practice and ideal. But as I've mentioned there are several ways to work around such limitations if having access directly to the management and Smart Console during the upgrade is not an option.

Most of my upgrades are done remotely and I always tell my customers that they need to be prepared as something unexpected might always occur. Thus requiring them to have someone on-site make it possible for us to reach the gateway using console and reaching the management server using ssh and smart console directly if needed. This is normally achieved by having someone on-site with local access accessible using remote tools such as Zoom, TeamViewer etc via mobile hotspots so I'm able to gain remote access while the gateway is not working.

But in most scenarios, many customers tend to not take this too seriously. I've ended up in various scenarios where something goes wrong and I have no access and my suggestions tends to help me work around most of these kinds of issues. Most of the time upgrades happen without any issues, to begin with.

[_Val_]

Been there done that, still trying to fix any architectural issue I see 🙂