Problems with vsx_util change_private_net command to increase allowed interfaces on Virtual Systems

[israelsc]

Hello everyone!
I hope you can help me with a problem that I am having on a VSX when trying to run the procedure to increase allowed interfaces on a VS.

I am following the sk99121 (How to add more than 64 interfaces for a Virtual System in VSX cluster R77.x and higher)
https://support.checkpoint.com/results/sk/sk99121

I have a VSX HA Cluster with two firewalls and a MDS Server, all with R81.10 Take 45.

We have the default configuration of the VSX cluster interfaces, I mean:
“VSX cluster internal communication network is 192.168.196.0 / 22 which allows up to 64 interfaces in each Virtual System” as the sk says.

We tried changing the private addressing with the command “vsx_util change_private_net” and with the network 192.168.160.0/21 to be able to configure 128 interfaces (and 30 Virtual Systems).

However, we get an error stating:
**** Push configuration to module failed
**** change cluster private network operation finished with errors.

I attach a screenshot with the issue:

The information shown in the log does not give much indication of what could be the error in the operation.
I share the log changing the name of the MDS and VSX Cluster for confidentiality:

[Expert@MDS:0]# more /opt/CPmds-R81.10/customers/mdscma/CPsuite-R81.10/fw1/log/vsx_util_20240529_14_33.elg

******************************************************************************************
* Note: the operation you are about to perform changes the information in the management *
* database. Back up the database before continuing. *
******************************************************************************************

SessionEstablishedCB started:
BindHandlerCB started:
OpenDbHandlerCB started:
OpenDbHandlerCB: opened a connection in read/write mode
DiscardHandlerCB started:
GetVsxObjectsName started:
GetVsxObjectsNameCB started:
This is the first time we are trying to lock vsx object
DisplayVSXObjectsAndSelect started:
The choosen command is relevant for VSX clusters only
DisplayVsxObjectsOptionAndSelect started:
GetVsxMembers started:
GetLockVsxObjectCommand started:
SendCommand started:
Starting operation...
Command body is:
(
:vsx_name (CLUSTER)
:vsx_activation_key ("****")
)

MyCommandCB started
***Reply is : (
:vsx_status (0)
:vsx_stat_str ("vsx object was successfully locked")
:AdminInfo (
:cpmi_cmd_status_code (0)
:subject (vsx-lock-vsx)
)
)

lockVsxObjectCB started
Lock status: vsx object was successfully locked
HandleClientRequest started:
HandleChangePrivateNetwork started:

*** Warning: downtime is expected during this process ***

The current selected vsx object is 'CLUSTER'
HandleChangePrivateNetworkCB started:
Current IPv4 cluster private network is 192.168.196.0/22

GetChangePrivateNetworkCmd started:
SendCommand started:
Starting operation...
Command body is:
(
:vsx_name (CLUSTER)
:cluster_private_network (192.168.160.0)
:cluster_private_network_mask (255.255.248.0)
:resume (false)
:vsx_activation_key ("****")
)

MyCommandCB started
***Reply is : (
:note ("
Push configuration to module CLUSTER started...")
:format (line)
:vsx_status_code (0)
:vsx_operation_result (0)
:message_type (1)
:AdminInfo (
:cpmi_cmd_status_code (0)
:subject (operation-note)
:operation (change-cluster-private-network-vsx)
)
)

Push configuration to module CLUSTER started...
MyCommandCB started
***Reply is : (
:note ("**** Push configuration to module failed
")
:format (line)
:vsx_status_code (0)
:vsx_operation_result (-2147467259)
:message_type (1)
:AdminInfo (
:cpmi_cmd_status_code (0)
:subject (operation-note)
:operation (change-cluster-private-network-vsx)
)
)

**** Push configuration to module failed

MyCommandCB started
***Reply is : (
:note ("

**** change cluster private network operation finished with errors.
**** Please resolve errors above and then run it again to complete the operation.
")
:format (line)
:vsx_status_code (0)
:vsx_operation_result (0)
:message_type (1)
:AdminInfo (
:cpmi_cmd_status_code (0)
:subject (operation-note)
:operation (change-cluster-private-network-vsx)
)
)

**** change cluster private network operation finished with errors.
**** Please resolve errors above and then run it again to complete the operation.

MyCommandCB started
***Reply is : (
:save-db (true)
:vsx_status (0)
:AdminInfo (
:cpmi_cmd_status_code (0)
:subject (change-cluster-private-network-vsx)
)
)

Saving db...
SaveDbCB started:
succeeded to close db
Database saved successfully.
[Expert@MDS:0]#

We are testing this in a lab before perform this procedure to production environment with our customer.

It is important to note that the MDS Server is correctly replicated.

The VSX HA Cluster was replicated with two physical Security Gateways model 6200.
The cluster has only the “Mgmt” and “Sync” interfaces physically connected.
All bondings, VLANs and other physical interfaces of the real cluster were simulated with dummy interfaces.
This using the commands in expert mode:
modprobe dummy numdummies=40
ip link set name eth1-01 dev dummy0
ip link set name eth1-02 dev dummy1
ip link set name eth1-03 dev dummy2
ip link set name eth1-04 dev dummy3
ip link set name eth1-05 dev dummy4
ip link set name eth1-06 dev dummy5
ip link set name eth1-07 dev dummy6
ip link set name eth1-08 dev dummy7
ip link set name eth2-01 dev dummy8
ip link set name eth2-02 dev dummy9
ip link set name eth2-03 dev dummy10...
...and so on until you get to eth5-08 with dev dummy40

With the command “cphaprob state” I see that:
FW1 status is ACTIVE (!)
The status of FW2 is DOWN

Although I can ping or reach network level to the Mgmt or Sync interfaces, I get an interface problem.
We know this when we run the command “cphaprob -l list”.
Everything is OK, except “Interface Active Check”.

Expert@GW2CLUSTER:0]# cphaprob -l list

Integrated devices:

Device name: Interface Active Check
Current status: problem

Device name: Recovery Delay
Current status: OK

Device name: CoreXL Configuration
Current status: OK

Registered devices:

Device name: Fullsync
Registration number: 0
Timeout: None
Current status: OK
Elapsed time since last report: 377.1 sec

Hopefully this information can help determine what the root cause is or give us an idea of what I can check to fix the error “change cluster private network operation finished with errors”.

Greetings!!

[emmap]

It may have failed because one of the nodes is Down. If you just open up the cluster object in SmartConsole and click OK, does it complete successfully (do you get a green tick with no alert icon)?

You have a sync pnote as well as the interface down pnote, that might not be helping. 

[israelsc · Accepted solution]

Hello @emmap 
Thank you very much for the input.

I was testing in a parallel lab, where I created a “clean” MDS and VSX HA Cluster without any client configuration.
Here, I created only one VS and created 64 dummy interfaces to replicate the error.
I performed the sk99121 procedure and I see that in this “clean” environment where:
-I have no errors in my cluster formation or any PNOTE.
-where my cluster is in “ACTIVE-STANDBY” correctly and without errors,

in this environment the command “vsx_util change_private_net” completes correctly.
I did a reboot on that cluster and I was able to add more than 64 interfaces on the VS.

Then, I went back to my other lab where the operation is completed with errors and I see that it does make the change in the “Cluster members internal communication network” section.
Here I notice that the network changes from 192.168.196.0/ 22 (default for 64 interfaces per VS) to 192.168.160.0/21 (segment for 128 interfaces per VS).

 
Although the operation completes with errors, I restart those two cluster firewalls and I see that after this, I can add more than 64 interfaces per VS.

So I concluded that the error that was showing me was definitely some error related to the PNOTES of my cluster (which were probably related to not finding all the interfaces of the cluster and these were replaced by dummy interfaces “modprobe”).

Thanks for the help, I have solved the issue in the end.
Thank you!!! 🙂