Central license open platform cluster hardware upgrade primary

[Adam276]

When upgrading hardware with central licensing on open platform, How do you deal with the upgrading the final primary system hardware during the upgrade in a way that you can fully test the secondary and still be able to have the management access the primary to remove the license on the primary after the secondary is verified working as expected?

I have seen steps where you issue a 'cpstop' command on the primary to make the upgraded secondary go active.  Wouldn't that break communications between management and the primary in a way that the management wouldn't be able to remove the license from the primary when it is time to upgrade the primary hardware?  I wouldn't want to remove the license from the primary until after the secondary is verified working so that a fail-over back to working state would be as quick as possible if secondary doesn't work.

Would clusterXL_admin down be better to use when switching to the secondary so that the primary would still be able to communicate with the management?  Or is it ok/safe to issue a cpstart on the primary after the secondary has taken over?  A note that the primary will still be on R80.10 and the secondary will be on R81.20 (in case that matters with the cpstart).

[G_W_Albrecht]

When installing on a new hardware, you automatically have the PnP license (14 days afaik) on it.  If you install config from backup this usually includes the license. Also, if you have the module/management portions of the license, you could install the module (GW) part manually (e.g. from CLI or WebGUI).

sk107042: ClusterXL upgrade methods and paths

R81.20 Installation and Upgrade Guide - Upgrade Methods

[Chris_Atkinson]

Is there some part of the license process that you're trying to avoid for some reason i.e. reactivation via UC?

1. Depends on the topology of the management interface is it clustered or private

2. Leveraging eval licenses temporarily is an option 

3. Licenses can be removed locally

[Adam276]

Chris,
The problem that I see is that the normal processes that are explained in the forums for hardware changes for as little downtime as possible involves running cpstop on the primary at some point where you make the secondary active.  If I do that, I assume management will not be able to detach (check back in) the central license from the old primary hardware when it is time to upgrade the primary hardware (after an hour or two of testing).  So management at that point will think it is still attached and maybe not allow me to reattach it again to the same object when I put the new hardware in place without detaching the old central primary license through SmartUpdate.

I am not trying to bypass anything.  I am just trying to get the license on the new primary hardware during the hardware upgrade process after the secondary has been upgraded.  The secondary is easy as I can detach that before I start the upgrade while the old secondary is still in cpstart state.  I will have the primary to failover to if the secondary fails so I don't want to detach any licenses or modify the primary in any way before the secondary has proven in good working state.

The primary will be in cpstop state though so the management will still think the central license is still attached (checked out) to the old primary when I remove it and put the new primary in place.  The management needs to know that the gateway central license it is no longer attached(checked out) to the old primary so that I can reattach it to the same object when new firewall is in place and SIC reset.  Is this figured out automatically somehow and will allow me to reattach it again to the same object that management thinks it has already been attached to?

Is there some mechanism for central licenses (attached to management IP) where the management will see that it has been attached in management but the gateway doesn't have it and allow it to attach it again or is there some other way to force detach the license in management so that it can then be reattached to the same object?


Albrecht,
Thanks for info but I didn't see any information about changing to new hardware and dealing with central licenses during the hardware upgrade process in those documents.

Everything I have read including on this forum has mentioned that you must use management to install central licenses on gateways (restoring from backups are an exception for like hardware). I use SmartUpdate for that.  This is going to new hardware (much newer) so the normal webui backup wouldn't work from what I have read and I am also transitioning to a new version at the same time so it the new hardware was installed from scratch with the new version.

The 14 day trial license has expired already, but even if I reinstall and it wasn't expired, that would only be temporary if the management has checked out the license already to the old hardware and thinks it is still attached when it really isn't (hardware was replaced)..

Thanks for the feedback on this.  Maybe there is some automatic stuff that happens in this scenario where management thinks a central gateway license is attached to a gateway but it isn't actually attached on the hardware itself (hardware replaced) so that it marks it as not checked out to that gateway anymore and allows you to reattach it or reattaches it automatically if it is checked out but not actually on the hardware.  I am being cautious as I haven't found anything that hints at that.

For regular licenses that are not central and don't require management to attach them, it is much easier I would think but for central licenses, management (SmartUpdate) has to check out and check in the licenses to know what they are attached to so that you can't attach a license twice.

[Adam276]

Is the answer to this to detach (check back in) the license using SmartUpdate when new hardware on the primary is put in place after establishing SIC?  Did Checkpoint plan for this and it would be corrected (checked back in) after that and then allow reattaching?  It would fail I would assume though because the license doesn't actually exist on the new hardware.  Maybe that is a problem.

[Chris_Atkinson]

CLI commands e.g. "cplic put" can be used locally on the gateway itself where the scenario requires it. 

Also 30-day eval licenses are available via self-service subject to acceptable use conditions:

https://usercenter.checkpoint.com/ucapps/prodeval

[Adam276]

Thanks but everything I have read indicates that cplic put for central licenses only works from the management server and can't be done from the local gateway itself.  I assume this is because the IP address for the license is the management server so maybe something special happens on the gateway itself to allow the management IP in the license to be valid instead of it's external IP when the license is attached from management.  An example mentioning that central licensing can only be attached from the management.
https://community.checkpoint.com/t5/Security-Gateways/Central-License-CLI-Security-Gateway/td-p/2981...

Are you saying though that it will actually work?  The IP in the license would be the management and not the external IP of the firewall.

What would be the procedure for central licensing (license contains IP of management) if you had a gateway failure and the backups were not working for example?  You would have to setup a new firewall from scratch.  The management would think the license is already attached (checked out) to the firewall, but you have put in a new firewall so it doesn't have the new license yet.  Would cplic put still work or would it give an error because the management thinks it is already attached? It actually isn't though because it is new hardware.  Maybe once SIC is established, pushing the license again manually form command line on the management to the remote gateway  would fix the mismatch and put the license on the remote gateway?  There must be a procedure for this scenario I would think.

[Bob_Zimmerman]

You can just use 'cphastop'. That stops all clustering, but leaves CPD, the firewall, and so on all running normally.

'cpstop && cpstart' would also work. When a cluster member comes online, it checks for CCP from an existing cluster member before deciding what state it should assume. If it detects a cluster with the same ID (and if you use the same objects, it will have the same ID) but running a different version, it will go to Ready rather than Active/Standby/Active Attention/Down/whatever else. A member in Ready state will become Active if it stops hearing the CCP traffic.

[Adam276]

Thanks for all the help and info on this.

Yea,  I am upgrading gateway hardware and software major versions at the same time for these central licensed firewalls.

That is good to know that the cphastop on the primary would not cause any issues with switching to the secondary.  I would have e expected it to work but was not sure since everything I read for hardware upgrades was to use cpstop.  I was worried during a major version upgrade and hardware upgrade there was some other thing that wouldn't allow it to switch or cause problems causing it to fail back.

Ideally what I would like to have is both the new and old hardware with the real license on it so that there are no other needed steps to put the license back on the old firewall in case of problems.  That is where I was hoping to find a working process to push the license to a new firewall even though management thinks it is already assigned to the same firewall object (but there is new hardware in place setup from scratch).  I would then have simpler steps as I could just switch back to the old hardware and it would have the license still on it.  This would be similar scenario with the management like I listed above where you have to replace a central licensed gateway where the backups are not working, etc.  Management thinks it has already attached and checked out the license to the gateway object but the new hardware doesn't have it yet.

[Adam276]

The clarification on the cphaprob stop was very helpful.  Thanks.  I don't seem to have an answer for how to deal with one of the scenarios though.

With central license (licensed for IP of management and not gateway), There is still the possible scenario where the new Dell hardware fails and I need to go back to the old hardware.  If the failed hardware/OS is completely non-functional and can't communicate with the management, I wouldn't be able to pull the central license from the current gateway using the management SmartUpdate.  In that scenario, How would I get management to allow the central license to be able to be attached back to the replacement firewall?  Does checkpoint management somehow detect this and mark the license in the repository as detached if the new firewall doesn't have a license but management thinks it does?  Does management go out and check this occasionally and update it's repository with what is really on the gateway or some action can be done to do that?  This scenario seems like it would happen rarely but enough that there would be a procedure for it.  How do you get the existing central license back on the new hardware and make sure management license database is in sync with reality of what is on the gateway when dealing with central licenses that are generated with the management IP that require a check in / check out?  Is it automatic?

[Bob_Zimmerman]

That seems like an extremely unlikely edge case.

You should take snapshots of the firewalls you're replacing and of the management server before you do anything. If something goes wrong and you roll back rather than troubleshooting forward, revert to those snapshots.

[Adam276]

I am just surprised there isn't a process to recover in case of a hardware failure so that you can get the central license back into management as detached so that you can push it to the replacement firewall quickly.  I guess that would take a TAC case or something if the rare chance it happens.  I know of 1 appliance with another company that failed about a year ago so I know this is rare but can happen.  That was one with regular license generated for the IP of the firewall though and not a central license.

My thought process on this is to make sure that I have a valid license on all gateways at all times so that I can flip back to the old hardware very quickly in case of problems.  I also of course need to make sure the check out / check in of the central license is done correctly.  I also don't want to pull the license from the old gateway hardware and it go to a single core and need to deal with the core count changes, etc.  Making use of a trial license sounds like that is the process to help with this.  I had an issue a few years ago where the core count (fw workers) on R80.40 got locked to 1 even with a valid 8 core license on it.  After a TAC case it was discovered that a file wasn't being changed correctly that restricted it.  I am sure that is fixed by now, but I would rather not have the old hardware go to 1 core when I detach the central license from it if I can.

OldP = Old Primary
NewP = New Primary
OldS = Old Secondary
NewS = New Secondary

Goals with this procedure...
Minimize downtime and steps in case of rollback to old hardware.

Make sure license changes don't cause core count change (having a condition where removing the central license resulting in no license on old hardware)... which seems to require a reboot when that changes.

A valid license on both old and new gateways at all times in case of rollback needed.

Deal with the management and it's view of the central license (gateway licenses generated for management IP) needs to be in sync with the check in and check out to the gateways.

1. Verify that OldP is active in HA, MVC is enabled on NewS, and that same Gaia/OS settings are on Old* vs New*.

2.. Add the 2 trial licenses to management so that management is aware of them for the future.

3. Manually attach the 2 trial licenses to NewP and NewS on the console of the new gateways (which are off the network at this point) and reboot to make sure corexl fw worker core counts are correct.  These are not central licenses so can manually be attached on the gateway itself.

4. Use SmartUpdate to attach the trial licenses to existing OldP and OldS.  At this point OldP and OldS now have 2 licenses (existing central and new trial) so that one can be removed later on and there will still be a license at all times on the gateways in case of rollback needed.

5. Detach central license from OldS through Smart Update.  (Trial license is still left on the OldS).

6. Physically disconnect network cables from OldS and Connecting to NewS.

7. Establish SIC to NewS

8. Attach central license to NewS using Smart Update from management that was previously removed from OldS.  There are now 2 licenses on NewS (trial and central).  Verify that gateway is seeing a valid license.

9. Change cluster version in management for gateway cluster to new version (R81.20).

10. Push policy (with checkbox unchecked during policy that allows only 1 firewall to succeed because of major Checkpoint version change)

11. Verifying that HA state is READY on NewP.

12. cphaprob stop on OldP to force secondary to go active

13. Test thoroughly NewS and verify logging, etc.

14. Detach central license from OldP. (The trial license is still left on OldP).

15. Physically disconnect network cables from OldP and connect to NewP.

16. Establish SIC to NewP

17. Attach central license to NewP using Smart Update from management that was previously removed from OldP.  There are now 2 licenses on NewP (trial and central).

18. Push policy (checkbox back to being checked to require both firewalls to get policy)

19. Verify that state is active on NewS and backup on NewP

20. clusterxl_admin down on NewS to force NewP to go active, verify state and then clusterxl_admin up to allow secondary to be able to take over in case of problem.

21. test thoroughly NewP and verify logging.

22. Make sure to disable MVC on gateways.

Any critiques of that?  I have always attached new licenses before removing old ones and haven't experienced any issues so I assume 2 licenses (if both are valid) will be ok for the short time during this to make sure there is at least 1 valid license at all times on the firewalls (even the old hardware).

[Adam276]

So short of it is that Checkpoint planned for the scenario with central licensing where the license might be stuck on the old hardware that is similar to a completely failed hardware situation... which is what I was hoping.

I setup a lab and tested it.  There is no need to use a trial license.  You can recover from a license being left on the old firewall.  I just established SIC with the new firewall.  I then selected the member in the gateways list in SmartConsole and at the bottom went to the license section to see what license was there and it showed no license.  Based on it taking a few seconds, It seems like it went out and checked the new hardware and detected that the license is not attached to it and updated the license attachment status.  I checked SmartUpdate and it now knew that the license was not on the gateway and showed detached.

I am not 100% sure if viewing the license tab of the firewall object in SmartConsole is necessary.  It is possible just opening SmartUpdate made the management and viewing the license page there caused it to go out and check/update the license status of the new firewall (using same object in management) or maybe something that happens automatically after establishing SIC on the new hardware.

New steps...

OldP = Old Primary
NewP = New Primary
OldS = Old Secondary
NewS = New Secondary

1. Verify that OldP is active in HA, MVC is enabled on NewS, and that same Gaia/OS settings are on Old* vs New*.

2. Physically disconnect network cables from OldS and Connecting to NewS.

3. Establish SIC to NewS

3a (NEW STEP) Open SmartConsole and go to the firewall object and view the license tab at the bottom to verify that it shows no license.  Not 100% sure this is required as maybe launching SmartUpdate does something when launched to detect that the gateway license is no longer attached (new hardware put in place).  I didn't have time to see if viewing it in SmartConsole was required or not.NEW STEP 16a: Open SmartConsole and go to the firewall object and view the license tab at the bottom to verify that it shows no license.  Not 100% sure this is required as maybe launching SmartUpdate does something when launched to detect that the gateway license is no longer attached (new hardware put in place).

4. Attach the central license to NewS using Smart Update.

5. Change cluster version in management for gateway cluster to new version (R81.20).

6. Push policy (with checkbox unchecked during policy that allows only 1 firewall to succeed because of major Checkpoint version change)

7. Verifying that HA state is 'backup' on NewS and 'active attention' on OldP.  (MVC allowed NewS to go to backup state instead of READY)

8. clusterXL_admin down on OldP to force secondary to go active

9. Test thoroughly NewS and verify logging, etc.

10. Physically disconnect network cables from OldP and connect to NewP.

11. Establish SIC to NewP

12. (NEW STEP) Open SmartConsole and go to the firewall object and view the license tab at the bottom to verify that it shows no license.  Not 100% sure this is required as maybe launching SmartUpdate does something when launched to detect that the gateway license is no longer attached (new hardware put in place).

13. Attach central license to NewP using SmartUpdate.

14. Push policy (checkbox back to being checked to require both firewalls to get policy)

15. Verify that state is 'active' on NewS and 'backup' on NewP

16. clusterxl_admin down on NewS to force NewP to go active, verify state and then clusterxl_admin up to allow secondary to be able to take over in case of problem.

17. test thoroughly NewP and verify logging.

18. Make sure to disable MVC on gateways and test failover again to make sure states are kept.