replacing appliances in a SecurityGroup with newer hardware

[Wolfgang]

Following sk162373 - Quantum Maestro supported combinations of mix-and-match appliances it's possible to mix different appliances for migration.

"Quantum Maestro supports all other combinations for migration purposes. For example, suppose you would like to replace a 6500 Security Appliance model with a 16200 Security Appliance model: In this scenario, you can include the two Security Appliance models in the same Security Group, allow it to clone the configuration to the 16200 Security Appliance model, remove the 6500 Security Appliance model, and continue to work with the 16200 Security Appliance model only. This migration procedure should include CXL / SND cores configuration and adjustment on new Security Appliances models, which requires downtime because of the reboot. As a result, you must do the migration during a maintenance window."

But now we are on the way doing such migration and have some problems. Often we got an advice from TAC not
adding the appliances to the old SG, creating a new SG with the new appliances should be the way. But this will
result in a needed longer maintenance schedule and much more work to do.
And this will be contrary to the scalability of Maestro.

As an example we want to replace 6600 with 9300 and 16600 with 9700.

Any experience with that or any suggestions ?

[AkosBakos · Accepted solution]

Hi @Wolfgang 

We do have experience with the change (CP6500 -> CP9300):

A short summary of the chnage: 

• removed the 2 old SGMs from site A
• put the new SGM to site A
• Came up in down state (it took ~10 minutes)
• #cphaprob synscat showed policy installation error
  • realized that the original lic expired - we knew that therefore created EVAL earlier-  and the EVAL license has just expired - (that1s the fun fact) 
    • we assume that because of the lack of valid lic (in this situation only InitialPolicy was availabe on the SGM) the SGM couldn't pull from its own lic from the usercenter (the initialPolicy didn't allower the traffic to usercenter)
• We created a new EVAL for the new SGM
• install the lic with cplic put
• rebooted the SGM, then it came up in Active state
• install the necessary jumbo take 
• than manual failover to the new SGM

The arrange of the other three SGMs into Security Group were easy after this experience.

Cleaunup: we remoed all EVAL licenses from the SGM-s g_cplic del <signature>

If you have question, and if you want, contact me in private, or drop an update here.

Here is the tread: https://community.checkpoint.com/t5/General-Topics/MAESTRO-SGM-change/m-p/233169#M38960

Akos

[alexgnunez2]

Definitavemente tuve un problema para levantar el cluster de Maestro , el comportamiento de los miembros del cluster (show cluster status) quedaba en LOST, despues de instalar la licencia de evaluacion el cluster Maestro levanto (SGMs) .

[AkosBakos]

@alexgnunez2 

Me alegra poder ayudarte, si tienes alguna pregunta no dudes en escribirme.
(traducido por el traductor de google)

Akos

[emmap]

I've done it recently, I found it easier to add a new one, patch it up to required JHF level, remove all the old ones, then add the remaining new ones with auto-clone enabled, then check/enable dynamic balancing as required. 

[Wolfgang]

@emmap that's what we expected. We can add the 9700 to the SGM with 16600 but nothing happened and the appliance does not reboot to join the SG. SGM is shown as member of the SG, not seen in Cluster state, no error in messages. If we remove the 9700 from the SGM the mini reset is run, the appliance reboots and is back online again as unattached.

[emmap]

I've heard of that happening, you can try rebooting the SMO SGM while it's waiting to add, I've heard of that helping, or even I've heard it just starts after 2 hours. I had it happen to me but it was the second new SGM, so I went to what I said above and removed all the old ones before auto-cloning it into the group, which worked fine. The slient_install log file showed nothing useful unfortunately, I don't know what causes the issue.

[CPengine]

I agree with @emmap's method, especially if you are moving from 6000 series to the 9000 series. We have seen success with the steps below:

Maestro Replacement Procedure: 

• Unbox new firewalls and use a USB to reinstall the base image or reset to factory default 

• Remove the SMO from the security group. Wait a few minutes, unplug, and remove. 

• Connect the new firewall, and power on. Give about 8-10 minutes to show up in MHO 

• Ensure that "auto-clone" is off. 

• Add the new firewall to the security group 

• Wait 8-10 minutes for the firewall to join the security group and return to SMO status 

• Once firewall is joined to the security group, upgrade the firewall to the latest jumbo hotfix 

• (NOTE: Based on traffic utilization, you can choose to remove the other members one at a time or all at once) 

• Remove remaining SGMs from the security group. Wait a few minutes, unplug, and remove. 

• Turn "auto-clone" on the SMO 

• Connect the new SGMs, and power on. Give about 8-10 minutes to show up in MHO 

• Add the new SGMs to the security group 

• Wait 20-25 minutes for the SGMs to join the security group and auto-clone from the SMO  
• (run# asg monitor)  

• (NOTE: Console into the SMG to watch progress during auto-cloning) 

• Once SGMs are joined to the security group, check to make sure they were auto-cloned and upgraded to the latest jumbo hotfix (cpinfo –y all) 

• Turn “auto-clone” off

• Push policy on the security group and monitor traffic. 

• Replacement complete 

[RickLin]

I have a question , how to let new 9000 appliance get the lowest ID be the SMO role when the Security Group still have old 6000 appliances?

Could CheckPoint teach us what are the right steps?

[AkosBakos]

Hi,

if you remove one SGM from the SG, the newly attached will get the lowest ID first.

so if you remove 1_1, the new will be 1_1.

Last time we unattached all SGMs from the site, then attached the Quantum Force appliances. 
works fine

akos

[RickLin]

Unattach old appliance then attach new appliance then publish (once) or Unattach then publish, wait a moment, then attach new appliance and publish again(twice)?

[AkosBakos]

Hi,

I would do one-by-one. I would start the next change when the newly added SGM is in active state.

"Last time we unattached all SGMs from the site, then attached the Quantum Force appliances. 
works fine" -> standby site -> removed all SGMs -> attached one SGM -> waited until came up to Active -> install the required jumbo hotfix -> failover to the new Quantum Force 9300 -> short smoke test -> then remove all old SGMs -> only the 9300 is in duty -> attach the remaining 9300s to the SG.

If you have more questions you can contact me directly (private message)

Akos

[CPengine]

@AkosBakos  is correct. The MHO knows who the SMO is by where you have the DAC cables plugged in. So removing the 1_1 and replacing it, the new firewall will come up as 1_1.

Whatever you do, please do not try to auto clone a 9000 series from a 6000 series. It will not work, as the OS version is slightly different. I would imagine that you can't auto clone a 9000 series with any other model. 9000 series have there own SK page in which you can download the OS version.

Just let the 9000 series join the security group, upgrade it to the latest JHF, then you can auto clone the rest of the new replacement firewalls from the new 9000 series SMO.

Hope this helps.

[AkosBakos]

Yes, good summary. Only the configuration "clone" works between 6000 series and 9000 series, so you have to install jumbo manually after attaching the SG ne SGM.

Akos

[moritz_r]

Hey guy, we are in a similar situation; DUAL Site MHO140 with 6x SG6600 running R81.10 JHFT170. We need to Upgrade the Security Group to 4x SG9700 for better bandwidth.

I will build the hole SG running R81.20 from scratch in parallel but with different Uplink Ports. Later we will plug out / in the SPF+ Cables from the old SecurityGroup Uplink Ports to the newly SecurityGroup Uplink Ports.
Since there is no Admin Guide how we should do this if mix and match does not work for 6600 and 9700 we decided to change the SG with a short outage in a planned maintenance window.

In addition, if something fails, and we run into issues, then time is running out, we still have the old SecurityGroup as our FallBack by switching back the SPF+ Cables.

Are there any concerns regarding this procedure?
Best regards

[AkosBakos]

Hi @moritz_r 

Almost any kind of appliaces can work for the configuration sync. If you put the new SGM into an existing SG, the config should sync to itself.

This helped for you?

Check this thread: https://community.checkpoint.com/t5/General-Topics/MAESTRO-SGM-change/m-p/233169#M38960

[moritz_r]

Hello @AkosBakos,

so you mean, the SG9700 (running R81.20) will sync all configurations even with SG6600 which are running with R81.10? I'm not sure about it, but I'm not able to test it in advance.
As well, no unchanged fallback will be available in a very short time if we need a stable network again (worst case).

Best regards

[emmap]

They won't sync between different versions. If you upgrade the existing security group to R81.20 though you can add the 9700s to the SG with the 6600s temporarily for migration purposes. 

[Wolfgang]

I would prefer doing this with the same SG and not creating a new one. New SG needs a new firewall-object, configure all settings, adding them to the rules (install targets, source, destination etc.)

You can add the new appliances one by one to the existing SG. Add one appliance and see what happens, if something goes wrong you can remove the appliance or set cluster_down for the new one.

You can add the appliance to the SG if all running the same version. R81.20 is mandatory for the 9000 series. @emmap mentioned that, you have to upgrade your environment. Additional your appliances must run the same speed on the downlink ports, meaning if your 6000 running 10G, you have to use 10G with the 9000 for the downlink ports. You can't mix ports with different speed.

[moritz_r]

Hello,
thanks for your Replies.
Yes, I understand all your points, but I decided to create a fully new SG either.
I can't Upgrade SG6600 to R81.20 and as well I can't add the 9700 to the 6600
> Need a running fallback, swapp uplinks to the old SG is faster than revert snapshots etc.
> already tried an Upgrade to R81.20 twice and running into BGP Flapping issues etc.
> already multiple In-Place Upgrades are done with the SG6600 from R80.30
> SG6600 Downlink are 10G and SG9700 Downlink are 100G, so sync is not possible as you said. thats one more reason not to add the SG9700 to the SG6600

So we want to Build a fully new SG with 9700, Policy Package are the same, as well the SG-Object in the SMS
Yes, it is quite inconvenient because we must cut off the SIC from the SG6600-SG, but don't need to change anything else. The second SG with 9700 are already configured manually and ready, now I'm looking forward to the MW and Cut-Over ‌‌

Many thanks for your thoughts
best regards