CheckPoint Migration - Issues with sync interfaces

[k_b]

Hi,

We are currently in the progress of migrating our old firewall to newer devices. This migration includes a change of both, the firmware and the hardware. All required / recommended steps have been conducted and most of our new clusters are looking good and are ready for the migration but with one cluster are we facing an issue: The cluster is not showing any sync interface. This can be validated via cphaprob -a if. Both in the SmartConsole using the managment server and the WebUI of the gateways we can see that the sync interfaces are configured correctly. Unfortunately are they not used.

After further investigation we found the following after starting the cluster in the debug mode (cphastart -d).

/opt/CPsuite-R81/fw1/bin/cphaconf -D -i 1 -n 1337 -c 2 -m 4 -l 1 -t bond2.10 bond2.11 ... start

but it should be

/opt/CPsuite-R81/fw1/bin/cphaconf -D -i 1 -n 1337 -c 2 -m 4 -l 1 -t bond2.20 bond2.21 ... start

That is interessting because bond2.10 and bond2.11 have been the sync interface on the old firewalls from which we migrate. The interfaces have been changed in this context to use different on the new cluster. These two VLANs (10 and 11) are also not configured anymore on the gateway (we can neither ping the ip addresses in these networks, nor can we see them in the configuration (cli, GUI)). Additional we found a hint to "file_diff_compare_data_to_file: returning 1 for file /opt/CPsuite-R81/fw1/database/cluster_policy.C" in the log files. In this policy, both VLANs (10 and 11) are defined / documented, but the new ones (20 and 21) are not. (Note: Also the following can be observed "file_diff_write_data_if_changed: not writing data. Identical to /opt/CPsuite-R81/fw1/database/cluster_policy.C"). The VLANs are not defined, that´s true, but the old VLANs are configured somehow with the IP address of the new VLANs. Acctually I don´t understand that. Policy installation via SmartConsole is also not possible, we can only fatch it (fw fetch...).

I wonder: Have you every experianced a similar case? Is there any way to adjust the cluster configuration (locally and) manually on the gateway itself? Do you have any idea why everything got fixed up so much?

I am happy about every comment.

With kind regards

Note: Version R81, Take 87

Note: cphaconf [-t <sync IF 1>...] [-d <non-monitored IF 1>...] add had no effect at all.

[Vladimir]

Few things are not clear from your post, specifically, how the migration to new firewalls/clusters is being performed. If you are importing Gaia configs from the old members, the bond and the VLAN data will be replicated exactly.If you are moving SYNC interface to a different bond/subinterface, those must be configured on individual devices, as the VLAN added to corresponding switch interfaces.

Only then the topology of the cluster must be altered to reflect these new interfaces (bond/vlan IPs and SYNC defined) and the policy installed.

A reminder- the bond itself must NOT have IP assigned to it.

[k_b]

Hi,

Thank you so far for the response. Please see the "runbook" for the migration, I hope, I remembered everything correctly.

Managemetn system (has been delivered with R81)

- Installed R80.40 via USB for the migration
- Installed DA Client with a newer version (DeploymentAgent_000002267_1.tgz) via gaia web ui
- Installed Package ngm_upgrade_wrapper_995000611_1.tgz was installed successfully. (Update Tools) via gaia web ui
- Installed JHF for R80.40 (Take 192) via gaia web ui
- Added licenses via clish
- Executed migration script on old system (R77.30)
./migrate export --exclude-licenses --exclude-uepm-postgres-db /var/log/db-export/mgmt_system-dump
- Faced some issues so we had to correct the policy. Used the following command to find entries which have to be adjusted.
# grep --color='auto' -P -n "[\x80-\xFF]" in objects_5_0_0.c and rulebases_5_0.fws ($FWDIR/conf)
- Executed migration script again, now with success.
- Import on R80.40 (new hardware):
/opt/CPsuite-R80.40/fw1/bin/upgrade_tools/migrate import /home/transfer/mgmt_system-dump.tgz
- Upgrade to R81 via Check_Point_ICE_T392_600SM_main_T470_R81_Gaia_3_10_Install_and_Upgrade
- Installed JHF for R81
- Exported GAIA Config from old system and imported it to new system (line-by-line)
show configuration / save configuration /tmp/config.xx

Gateways

- Already delivered with R81 (no update required)
- Used clish to import the GAIA configuration of the old system
- Adjusted interfaces, which have been changed, via GAUI and topology in the dashboard
- Installed policies
- Installed JHF / new DAagent

[Hugo_vd_Kooij]

Sounds about right. It pretty much looks like my project plan for similar upgrades.

[_Val_]

Do you have bon interfaces properly set on GWs? Also, when establishing SIC with the new GWs, fetch interfaces and define sync interfaces on the cluster object. Finally, two sync interfaces are no longer supported, use just one bond for sync, not two.

[Vladimir]

Some points of interest here, two that @_Val_  has mentioned already and a few more, but in this sequence:

Gaia configuration migration performed not line-by line, but using "save configuration <filename>" and "load configuration <filename>". If moving from R77.30, you may also have to use this sequence:

1. On old gateway:

save configuration <filename>

SCP config to the PC you are working from

2. On new gateway:

Gaia config load

The above is the process described in my book "Check Point Firewall Administration R81.10+" and has been mentioned in this forum in the past.

3. Establish SIC with new cluster members

4. Change the cluster hardware to the new appliance series

5. Change the cluster version to the upgrade target (r81)

6. Change the cluster objects network settings to reflect new single Sync interface (you can do "get interfaces WITHOUT topology" to simplify the process, then go over new interfaces and verify that each is configured as per your requirements.

7. Load the policy.

Regards,

Vladimir

[the_rock]

I agree with all the points @Vladimir made. We definitely need to verify all those details.

Andy

[the_rock]

Also, to help us further, can you send below from expert mode from both members

cphaprob roles

cphaprob state

cphaprob list

cphaprob -a if

cphaprob syncstat

[k_b]

Hi,

Thank you. Please see below the output of the requested commands.

[Expert@gatewayA:0]# cphaprob roles

ID Role

1 (local) Non-Master
2 Master

[Expert@gatewayB:0]# cphaprob roles

ID Role

1 Non-Master
2 (local) Master

[Expert@gatewayA:0]# cphaprob state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 10.1.20.21 0% DOWN gatewayA
2 10.0.20.149 100% ACTIVE(!) gatewayB

Active PNOTEs: IAC

Last member state change event:
Event Code: CLUS-110600
State change: STANDBY -> DOWN
Reason for state change: Incorrect configuration - Sync interface has not been detected
Event time: Mon Aug 14 13:05:20 2023

Last cluster failover event:
Transition to new ACTIVE: Member 1 -> Member 2
Reason: Member state has been changed due to restart of the Cluster module
Event time: Mon Aug 14 12:55:02 2023

Cluster failover count:
Failover counter: 2
Time of counter reset: Fri Aug 11 11:06:38 2023 (reboot)

[Expert@gatewayB:0]# cphaprob state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 10.0.20.21 0% DOWN gatewayA # Note: This is an IP address in VLAN 20
2 (local) 10.1.20.149 100% ACTIVE(!) gatewayB # Note: This is an IP address in VLAN 3020

Active PNOTEs: IAC

Last member state change event:
Event Code: CLUS-116505
State change: DOWN -> ACTIVE(!)
Reason for state change: All other machines are dead (timeout), Incorrect configuration - Sync interface has not been detected
Event time: Mon Aug 14 12:55:02 2023

Last cluster failover event:
Transition to new ACTIVE: Member 1 -> Member 2
Reason: Available on member 1
Event time: Mon Aug 14 12:55:02 2023

Cluster failover count:
Failover counter: 2
Time of counter reset: Fri Aug 11 11:06:38 2023 (reboot)

[Expert@gatewayA:0]# cphaprob list

Built-in Devices:

Device Name: Interface Active Check
Current state: problem

[Expert@gatewayB:0]# cphaprob list

Built-in Devices:

Device Name: Interface Active Check
Current state: problem (non-blocking)

[Expert@gatewayA:0]# cphaprob -a if

CCP mode: Manual (Unicast)
Required interfaces: 2
Required secured interfaces: 0

Interface Name: Status:

bond2.20 (LS) UP
bond2.3020 (LS) UP

Warning: Sync will not function since there aren't any sync(secured) interfaces

S - sync, HA/LS - bond type, LM - link monitor, P - probing

Virtual cluster interfaces: 1

bond2.20 10.0.20.20 VMAC address: 00:1C:7F:00:1A:59

[Expert@gatewayB:0]# cphaprob -a if

CCP mode: Manual (Unicast)
Required interfaces: 2
Required secured interfaces: 0

Interface Name: Status:

bond2.20 (LS) UP
bond2.3020 (LS) UP

Warning: Sync will not function since there aren't any sync(secured) interfaces

S - sync, HA/LS - bond type, LM - link monitor, P - probing

Virtual cluster interfaces: 1

bond2.20 10.0.20.20 VMAC address: 00:1C:7F:00:1A:59

[Expert@gatewayA:0]# cphaprob syncstat

Delta Sync Statistics

Sync status: Off - Sync interface down.

Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0

Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0

Sent messages:
Total generated sync messages................ 24352
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1

Received messages:
Total received updates....................... 0
Received retransmission requests............. 0

Sync Interface:
Name.........................................
Link speed...................................
Rate......................................... 0 [Bps]
Peak rate.................................... 0 [Bps]
Link usage................................... 0%
Total........................................ 0 [B]

Queue sizes (num of updates):
Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50

Timers:
Delta Sync interval (ms)..................... 100

Reset on Mon Aug 14 14:50:59 2023 (triggered by fullsync).

[Expert@gatewayB:0]# cphaprob syncstat

Delta Sync Statistics

Sync status: Off - Sync interface down.

Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0

Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0

Sent messages:
Total generated sync messages................ 71328
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1

Received messages:
Total received updates....................... 0
Received retransmission requests............. 0

Sync Interface:
Name.........................................
Link speed...................................
Rate......................................... 0 [Bps]
Peak rate.................................... 0 [Bps]
Link usage................................... 0%
Total........................................ 0 [B]

Queue sizes (num of updates):
Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50

Timers:
Delta Sync interval (ms)..................... 100

Reset on Fri Aug 11 11:26:14 2023 (triggered by fullsync).

[_Val_]

From the output above:

Warning: Sync will not function since there aren't any sync(secured) interfaces

You need to define a sync interface on the cluster object.

[the_rock]

@_Val_ Makes an excellent point...shows you dont have sync even configured. Ok, maybe you have it set on the OS level, BUT, if its not defined in topology, this will never work. Can you verify topology in the smart console?

Andy

[k_b]

Hi,

First of all, thank you all for your comments. A sync interface was and is configured in the topology, this can be validated via SmartConsole. At the begining, we had two, now, after your comments, I have changed it to one sync interface. But it is still true, that the cphastart script is somehow still referring to the old interfaces.

/opt/CPsuite-R81/fw1/bin/cphaconf -D -i 1 -n 1337 -c 2 -m 4 -l 1 -t bond2.10 bond2.11 ... start

Where are these parameter (-t bond2.10 bond2.11) coming from?

And does anyone know what the reference is for this comparsion?

file_diff_compare_data_to_file: returning 1 for file /opt/CPsuite-R81/fw1/database/cluster_policy.C

The policy has been fetched from the management system but there is still no change in the behavior of the gateway(s). Do I have to load something else / additional to the policy from the management system so that the sync interfaces are recognized.

[the_rock]

K, lets start at the beginning (no pun intended). So, can you ensure that all the required interfaces are indeed configured properly on both clustered members? If yes, head over to smart console, edit the cluster object, go to topology and MAKE SURE that every cluster interface is configured as such and VIP is set as well. Click "get interfaces WITHOUT topology", if good, install policy and run cphaprob state command on both again. If it gives an error, please send it here, so we examine further.

Dont worry, I feel confident we will be able to help you fix this.

Andy

[k_b · Accepted solution]

Hi,

We were able to fix the issue. Somehow the policy installation always considered an old version / status so that changes to the topology / interfaces were no adopted (fw fetch debug showed information to interfaces which have already been deleted / changed). To get a clean basis, we used "fw unloadlocal", and reinstalled the policy via SmartConsole. After these steps, the new sync interfaces were correctly initiated and set, the cluster is healthy and the services are running as required.

Once again, thank you all for the comments and recommendations how to handle and to fix this issue.

[the_rock]

Good job!!