cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post

Connections dropped during Connectivity Upgrade

Jump to solution

Hello,

I am upgrading our firewalls from R77.30 to R80.10. We have 17 ClusterXL clusters in high availability mode. Each cluster has two members running the GAIA OS on VMWare. I’m attempting to use the connectivity upgrade method to minimise disruption.  To create enough space to copy  the installation file “Check_Point_R80.10_T421_Fresh_Install_and_Upgrade_from_R7X.tgz” onto the firewall and import it I have had to:

  • expand the hard drive size (in VMWare vSphere)
  • expand the /dev/sda3 partition,
  • resize the physical volume,
  • then use lvm_manager to expand the lv_log volume

During these steps you need to reboot the firewall a number of times. This is fine when upgrading the standby cluster member. Then after that I use the cphacu commands to failover to the newly upgraded cluster member. So far so good and no disruption to the network. Then when I expand the hard drive size etc on the former active member and reboot it, it becomes the active member again. This is a hard failover and any open connections are dropped. You need to reboot 3 times and each time we failover over to the upgraded member and back causing disruption on the network.

Running cphaprob stat on the upgraded member shows that it is in the ready state because “another member was detected with a lower fw version.” Running cphaprob stat on the non-upgraded member only shows the local firewall and it is in the active state.

How can I perform the upgrades without dropping any connections? I.e. once we have failed over to the upgraded member, I want it to stay the active member until the other firewall is upgraded, surviving reboots of the other firewall.

0 Kudos
1 Solution

Accepted Solutions
Admin
Admin

Re: Connections dropped during Connectivity Upgrade

Jump to solution

I got a couple of suggestions from R&D on how to resolve this:

1. Do all the necessary disk resizing prior to upgrading the cluster to R80.10 (while cluster is still using R77.30) and before you use the cphacu command. This does cost an extra failover and is the recommended solution. 

2. If you can't have the extra failover, you can "trick" the older version into thinking it has a newer version as follows:

  • Add in fwkern.conf on the non-upgraded member: fwha_version=3500 (FYI the "official" number for R80.10 is 3120)
  • Run cphacu
  • After rebooting the upgraded member, R77.30 will think it has a higher version and will be in Ready state (thus won't fail back)
  • Before upgrading the other R77.30 member, remove the fwha_version line from fwkern.conf.
11 Replies
Admin
Admin

Re: Connections dropped during Connectivity Upgrade

Jump to solution

There's a setting in the Gateway Cluster Properties that controls this.

Make sure "Maintain current active Cluster Member" is enabled prior to doing an upgrade.

0 Kudos

Re: Connections dropped during Connectivity Upgrade

Jump to solution

Thanks Dameon, I saw that setting there, however I thought that a cluster member having a lower software version would override this setting?

See https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

There are cluster members with a lower software version on this subnet / VLAN
[member with higher software version will go into state 'Ready'].

0 Kudos
Admin
Admin

Re: Connections dropped during Connectivity Upgrade

Jump to solution

I don't believe that is necessarily the case here.

That said, this point is not something mentioned in the Best Practices guide for a Connectivity Upgrade: Best Practices - Connectivity Upgrade (CU)

Speaking of which, there's a step listed there that talks about changing the version of the cluster before proceeding to upgrade the primary, specifically pushing the policy only to the secondary member.

Did you do that step?

I believe that's relevant to solving the problem as the "policy ID" is encoded in the CCP packets (see the ATRG): ATRG: ClusterXL 

When you do this step only on the secondary member, the policy ID will be different, which should cause the original primary to back off until it gets back in sync.

0 Kudos

Re: Connections dropped during Connectivity Upgrade

Jump to solution

OK thanks I'll give it a shot with the 'Maintain current active Cluster Member' setting enabled and report back on how it goes.

I did change the cluster version and push it out to the cluster. As per the instructions the installation succeeded on the upgraded member and failed on the non-upgraded member.

0 Kudos

Re: Connections dropped during Connectivity Upgrade

Jump to solution

So I tried out using the "Maintain current active Cluster Member" setting , however the same problem as described in my original post occurs. After upgrading fw2 and failing over to it using the cphacu commands, then rebooting the non-upgraded firewall(fw1) the status of the firewalls is:

 

On the upgraded firewall

fw2> cphaprob stat

 

Cluster Mode:   High Availability (Active Up) with IGMP Membership

 

Number     Unique Address  Assigned Load   State

 

1          n.n.n.17     0%              ClusterXL Inactive or Machine is Down 

2 (local)  n.n.n.18     0%              Ready

 

Ready state reason: another member was detected with a lower FW version.

On the non-upgraded firewall

fw1> cphaprob stat

 

Cluster Mode:   High Availability (Active Up) with IGMP Membership

 

Number     Unique Address  Assigned Load   State

 

1 (local)  n.n.n.17     100%            Active

0 Kudos
Admin
Admin

Re: Connections dropped during Connectivity Upgrade

Jump to solution

I got a couple of suggestions from R&D on how to resolve this:

1. Do all the necessary disk resizing prior to upgrading the cluster to R80.10 (while cluster is still using R77.30) and before you use the cphacu command. This does cost an extra failover and is the recommended solution. 

2. If you can't have the extra failover, you can "trick" the older version into thinking it has a newer version as follows:

  • Add in fwkern.conf on the non-upgraded member: fwha_version=3500 (FYI the "official" number for R80.10 is 3120)
  • Run cphacu
  • After rebooting the upgraded member, R77.30 will think it has a higher version and will be in Ready state (thus won't fail back)
  • Before upgrading the other R77.30 member, remove the fwha_version line from fwkern.conf.
Admin
Admin

Re: Connections dropped during Connectivity Upgrade

Jump to solution

Cool trick , thx 

0 Kudos

Re: Connections dropped during Connectivity Upgrade

Jump to solution

Great Thanks Dameon, wish I had thought of option 1 myself.

0 Kudos
Employee+
Employee+

Re: Connections dropped during Connectivity Upgrade

Jump to solution

CDT Smiley Happy 

0 Kudos
Admin
Admin

Re: Connections dropped during Connectivity Upgrade

Jump to solution

Please provide a link that describes it

0 Kudos
Employee+
Employee+

Re: Connections dropped during Connectivity Upgrade

Jump to solution

sk111158: 

Introduction

The Central Deployment Tool (CDT) is a utility that runs on an R77 / R77.X / R80 / R80.10 Security Management Server / Multi-Domain Security Management Server (running Gaia OS).
It allows the administrator to automatically install CPUSE Offline packages (Hotfixes, Jumbo Hotfix Accumulators (Bundles), Upgrade to a Minor Version, Upgrade to a Major Version) on multiple managed Security Gateways and Cluster Members at the same time.

Cluster upgrades are performed automatically according to the sk107042 - ClusterXL upgrade methods and paths.

The CDT uses CPUSE Agents on the remote managed Security Gateways and Cluster Members to perform package installation. The CDT monitors and manages the entire process.

The CU use fwha_version kernel parameter part of the process.

Check Point Software Technologies: Download Center  page 14.

0 Kudos