Re: Maestro R81.10 Jumbo Hotfix install

Trevor_Bruss · ‎2022-01-20

I believe I have successfully migrated one of our Maestro environments from R80.20SP to R81.10. Now I've run into a problem when I attempted to install the latest Jumbo (at the time Take 22). The upgrade of the Orchestrators was not an issue, but it was during the upgrade of our security group that I ran into problems.

First, I downloaded the latest update and transferred it over to the security group and imported it. I then removed one member from the cluster. It was on that member that I installed Take 22. Upon a reboot, it never came Active again. It stayed in the Down state for hours. I even did a reboot of the member and it still would come back online in the Down state. I uninstalled the hotfix and returned the security group back to what it was before trying to install Take 22 on that member. The member came back up after the reboot and showed all members Active again.

Just curious if anyone else had attempted to update to a newer Take on their upgraded R81.10 Maestro environment and seen any issues or for that matter no issues. I see that Take 30 is now out, but I didn't see anything in it that lead me to believe it would make a difference. I should have captured what cphaprob -state showed after the upgrade on the one member, but I forgot to. If it makes a difference, I chose the member of the security group that was not the SMO. Again, shouldn't make a difference as in the past with R80.20SP I would always start with the non-SMO member when updating takes without any issue.

Danny · ‎2022-01-20

cphaprob list would have been helpful to check why it was in down state.
Did you try to perform cpstop on the other SGMs to check if the upgraded SGM turns active?
What do you mean by "removed one member from the cluster"? You performed a cpstop?

Trevor_Bruss · ‎2022-01-20

My Maestro security group contains two members. I cannot risk taking the Active (non-patched) server down while the second member (patched) is in a Down state to see if it just happens to go active.

So basically, I'm following the normal Maestro instructions on how to install a jumbo hotfix on a security group. Since their are only two members in the group I begin patching with member 2, and when it is done and both members show Active I move on to doing member 1. This is the first time I'm putting a jumbo hotfix on a newly updated R81.10 Maestro environment. So the hotfix was finished installing and it rebooted member 2. When it came back up, member 2 stayed in a Down state. I'll have to run an update again to see if I can capture the cphaprob state if it fails again. It had to do with something about it not being able to communicate or sync with the other member.

Danny · ‎2022-01-20

My guess is that R81.10 cannot sync with R80.20SP.

So after finishing your upgrade on one SGM to R81.10 JHF 30, when is stays in down state:

log into SmartConsole and change the SGO's version to R81.10, install policy
verify the running security policy on all SGMs via fw stat
run cphaprob stat; cphaprob list on both SGMs
in case your R81.10 SGM is still down, within a maintenance window run cpstop && sleep 300 && cpstart on your R80.20SP SGM
- if your R81.10 SGM turns active, kill the sleep routine on the R80.20SP SGM
- if your R81.10 SGM stays in down state, try to install the security policy again
- if your R80.10 SGM still stays in down state, if possible run cphaprob stat; cphaprob list. No worries, your R80.20SP SGM will turn active again in a couple seconds

Glossary:

SGO - Single Gateway Object > Maestro Security Group object within SmartConsole that talks to the SMO
SGM - Single Gateway Module/Security Group Member > Check Point Appliance that's part of a Security Group
SMO - Single Management Object - Active Check Point Appliance with the lowest SGM ID#

Check Point Maestro - FAQ

Trevor_Bruss · ‎2022-01-20

My Maestro environment is already upgrade to R81.10. My problem occurred when I began the process of installing R81.10 Jumbo Take 22 on these newly upgraded machines, specifically the security gateways as I had no problem installing the jumbo on the MHO devices. Apologies if I didn't make that clear.

K_montalvo · ‎2022-01-20

Hello @Trevor_Bruss

I had not worked with Maestro but did something similar recently on regular Quantum Security gateways running r81.10 on a HA Cluster XL. What i did is checked that clusters showed active>standby with cphaprob stat then did a clusterXL_admin down on Active member. After that downloaded the JHT via Web (Gaia) with CPUSU thing > run verifier after downloaded and then installed. After successfully installation and reboot i waited 15 minutes and then upgraded the active member the same way via WEB and after waiting 15 minutes ran clusterXL_admin up and check sync again. I did not experienced any downtime.

Tal_Ben_Avraham · ‎2022-01-22

Hi Trevor.

We have installed R81.10 JHF take 22 successfully in our labs (and also 30). So yes, it is expected to work.

If issue persists go ahead and open a support ticket.

kainneb · ‎2022-09-05

Hi all

I have exactly same problem as Trevor.

2 mho-140

2 SGM 16500HS (R80.30SP - take 97)

1 vsx gateway - 3 Virtual system

Problem: unable to upgrade with the last hotfix take 101.

following the procedure, I first turn SGM2 in down state and start upgrade on it. when this SGM2 finsih installation of the hotfix and reboot, it stay in Down state.

--------------------------------------------------------------------------------------------------------------------------------

[Expert@FW-MAESTRO-IA-ch01-01:0]# cphaprob state

Cluster Mode: HA Over LS

ID Unique Address Assigned Load State Name

1 (local) 192.0.2.1 100% ACTIVE FW-MAESTRO-IA-ch01-01
2 192.0.2.2 0% DOWN FW-MAESTRO-IA-ch01-02

Active PNOTEs: None

Last member state change event:
Event Code: CLUS-112004
State change: DOWN -> ACTIVE
Reason for state change: USER DEFINED PNOTE
Event time: Fri Sep 2 15:37:03 2022

------------------------------------------------------------------------------------------------------------------------------

some differences on cphaprob list command

[Expert@FW-MAESTRO-IA-ch01-01:0]# cphaprob list

There are no pnotes in problem state

[Expert@FW-MAESTRO-IA-ch01-02:0]# cphaprob list

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Current state: problem
Time since last report: 430 sec

-----------------------------------------------------------------------------------------------------------------------------

On smart console, only vsx gateway is green, but the 3 Virtual systems seems disconnected (red). Yet SIC seems to be good on both gateways:

[Expert@FW-MAESTRO-IA-ch01-01:0]# cp_conf sic state

Trust State: Trust established

------------------------------------------------------------------------------------------------------------------------------------

2 SGM show differents policy install. but unable to push any policies

[Expert@FW-MAESTRO-IA-ch01-01:0]# fw stat
HOST POLICY DATE
localhost FW-SNET-IA_VSX 2Sep2022 15:36:52 : [>Sync] [<Sync] [>magg3] [<magg3]

[Expert@FW-MAESTRO-IA-ch01-02:0]# fw stat
HOST POLICY DATE
localhost InitialPolicy 5Sep2022 9:01:07 : [>Sync] [<Sync] [>magg3] [<magg3]