cluster fail

[ChinChang]

Dears,
I'm learning cluster, but I met some trouble.
Please refer my topology & lab steps, and my SMS and 2 gateways version are R82.
I want to add cp101 & cp102 into my cluster, when I finish cluster operate and press publish, it failed.
Expect get help, tks.

[PhoneBoy]

What was the precise error received when you attempted to publish?

[ChinChang]

after publish by smartconsole. it show alert about clusterXL as screenshot, i think my lab should install it, and then i did also.

[PhoneBoy]

I'd start with disabling and re-enabling ClusterXL in cpconfig: https://support.checkpoint.com/results/sk/sk88360 

[ChinChang]

thanks your support, i tried it, disable and then re-enable, still failed.

[the_rock]

If you send us the error you get, it would help, for sure.

Andy

[ChinChang]

after publish by smartconsole. it show alert about clusterXL as screenshot, i think my lab should install it, and then i did also.

[Lesley]

Looks like you have not defined eth2 as sync interface is this correct? Clusters needs to have a sync interface between 2 firewalls. 

[ChinChang]

I try change eth2 to sync interface, still same result.

[the_rock]

Can you please send a screenshot of publish failure you get?

Andy

[ChinChang]

it had not publish fail,
in fact, publish success, but after publish like screenshot, status are red alert.

[the_rock]

Can you send below commands from expert mode from BOTH members?

Andy

cphaprob roles

cphaprob state

cphaprob -a if

cphaprob -i list

cphaprob -l list

cphaprob syncstat

I would say if you cant figure it out, try cphastop; cphastart on both members, if still nothing, run cpconfig, disable cluster membership, reboot, re-enable, reboot (do on both). If still no luck, I think remote session might be needed to verify everything.

Andy

[ChinChang]

i try it, cphastop and then cphastart, i got same result.
and below is follow your guide.

cp1> expert
Enter expert password:

Warning! All configurations should be done through clish
You are in expert mode now.

[Expert@cp1:0]# cphaprob roles

ID Role

[Expert@cp1:0]#
[Expert@cp1:0]# cphaprob state

HA module not started.

Cluster policy should be installed - please run cphastart
[Expert@cp1:0]# cphaprob -a if

HA module not started.

Warning: Sync will not function since there aren't any sync(secured) interfaces

[Expert@cp1:0]# cphaprob -i list

Built-in Devices:

Device Name: Interface Active Check

Device Name: HA Initialization

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: Boot

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

[Expert@cp1:0]# cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: Boot

Device Name: Policy
Registration number: 1
Timeout: none

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

Device Name: cxld
Registration number: 3
Timeout: 30 sec

Device Name: HD
Registration number: 4
Timeout: none

Device Name: fwd
Registration number: 5
Timeout: 30 sec

[Expert@cp1:0]# cphaprob syncstat

Delta Sync Statistics

Sync status: Off - Sync interface down.

Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0

Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0

Sent messages:
Total generated sync messages................ 0
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1

Received messages:
Total received updates....................... 0
Received retransmission requests............. 0

Sync Interface:
Name.........................................
Link speed...................................
Rate......................................... 0 [Bps]
Peak rate.................................... 0 [Bps]
Link usage................................... 0%
Total........................................ 0 [B]

Queue sizes (num of updates):
Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50

Timers:
Delta Sync interval (ms)..................... 100

Reset on Mon Dec 16 10:12:39 2024 (triggered by fullsync).

[Expert@cp1:0]#
====================================================
cp2> expert
Enter expert password:

Warning! All configurations should be done through clish
You are in expert mode now.

[Expert@cp2:0]# cphaprob roles

ID Role

[Expert@cp2:0]#
[Expert@cp2:0]# cphaprob state

HA module not started.

Cluster policy should be installed - please run cphastart
[Expert@cp2:0]#
[Expert@cp2:0]# cphaprob -a if

HA module not started.

Warning: Sync will not function since there aren't any sync(secured) interfaces

[Expert@cp2:0]#
[Expert@cp2:0]# cphaprob -i list

Built-in Devices:

Device Name: Interface Active Check

Device Name: HA Initialization

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: Boot

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

[Expert@cp2:0]#
[Expert@cp2:0]# cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: Boot

Device Name: Policy
Registration number: 1
Timeout: none

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

Device Name: cxld
Registration number: 3
Timeout: 30 sec

Device Name: HD
Registration number: 4
Timeout: none

Device Name: fwd
Registration number: 5
Timeout: 30 sec

[Expert@cp2:0]#
[Expert@cp2:0]# cphaprob syncstat

Delta Sync Statistics

Sync status: Off - Sync interface down.

Drops:
Lost updates................................. 0
Lost bulk update events...................... 0
Oversized updates not sent................... 0

Sync at risk:
Sent reject notifications.................... 0
Received reject notifications................ 0

Sent messages:
Total generated sync messages................ 0
Sent retransmission requests................. 0
Sent retransmission updates.................. 0
Peak fragments per update.................... 1

Received messages:
Total received updates....................... 0
Received retransmission requests............. 0

Sync Interface:
Name.........................................
Link speed...................................
Rate......................................... 0 [Bps]
Peak rate.................................... 0 [Bps]
Link usage................................... 0%
Total........................................ 0 [B]

Queue sizes (num of updates):
Sending queue size........................... 512
Receiving queue size......................... 256
Fragments queue size......................... 50

Timers:
Delta Sync interval (ms)..................... 100

Reset on Mon Dec 16 10:12:54 2024 (triggered by fullsync).

[Expert@cp2:0]#

[the_rock]

This is WHY it is NOT working. Can we do remote? Im free to assist if you are allowed to do zoom remote.

Andy

[Expert@cp1:0]# cphaprob -a if

HA module not started.

Warning: Sync will not function since there aren't any sync(secured) interfaces

[Lesley]

I also spotted this couple days ago;

Looks like you have not defined eth2 as sync interface is this correct? Clusters needs to have a sync interface between 2 firewalls. 

This indeed really needs to be checked and fixed. 

[the_rock]

I believe based on the last screenshot, it shows eth2 defined as sync, but something is clearly wrong, since the firewall still does not see it as such.

Andy

[Lesley]

Some quick tips.

check if interface is up ifconfig / ethtool

tcpdump on relevant interface

check if topology matches interfaces configured on firewalls. Sometimes different subnet is set

check if you can ping the other via on sync ip

check if you see 8116 and 256

[the_rock]

Agree with those.

[ChinChang]

i will check your instruction. TKS!
and then, what is the 8116, 256?

[PhoneBoy]

TCP ports in use, I believe. netstat -an

[Martijn]

Hi,

When following the First Time Wizard on the appliances, did you select 'part of a cluster'?
As Dameon says, is cluster membership enabled when you check 'cpconfig'.

Please send us the output of 'cpconfig' from the gateways.
Can you also share (in expert mode) the output of 'cphaprob stat' and 'cphaprob -l list'?

Looks like clustering is not enabled on the members.

Regards,
Martijn

[ChinChang]

did you select 'part of a cluster'?
>>yes, in cp101 & cp102.

cp101> cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Disable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products

(11) Exit

Enter your choice (1-11) :11

Thank You...
cp101> expert
Enter expert password:

Warning! All configurations should be done through clish
You are in expert mode now.

[Expert@cp101:0]# cphaprob stat

HA module not started.

Cluster policy installation failed on gateway (Error code: 204).
(see sk125152).[Expert@cp101:0]#
[Expert@cp101:0]# cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: cpstop

Device Name: Policy
Registration number: 1
Timeout: none

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

Device Name: cxld
Registration number: 3
Timeout: 30 sec

Device Name: HD
Registration number: 4
Timeout: none

Device Name: fwd
Registration number: 5
Timeout: 30 sec

[Expert@cp101:0]#

===============================================================================

cp102> cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Disable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products

(11) Exit

Enter your choice (1-11) :11

Thank You...
cp102> expert
Enter expert password:

Warning! All configurations should be done through clish
You are in expert mode now.

[Expert@cp102:0]# cphaprob stat

HA module not started.

Cluster policy should be installed - please run cphastart
[Expert@cp102:0]# cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: cpstop

Device Name: Policy
Registration number: 1
Timeout: none

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

Device Name: cxld
Registration number: 3
Timeout: 30 sec

Device Name: HD
Registration number: 4
Timeout: none

Device Name: fwd
Registration number: 5
Timeout: 30 sec

[Expert@cp102:0]#

[ChinChang]

did you select 'part of a cluster'?
>>yes, i did. in cp1 & cp2.

cp1> cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products

(11) Exit

Enter your choice (1-11) :11

Thank You...
cp1> cphaprob stat

HA module not started.

Cluster policy should be installed - please run cphastart
cp1>
cp1> cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: Boot

Device Name: Policy
Registration number: 1
Timeout: none

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

Device Name: cxld
Registration number: 3
Timeout: 30 sec

Device Name: HD
Registration number: 4
Timeout: none

Device Name: fwd
Registration number: 5
Timeout: 30 sec

cp1>
=====================================
cp2> cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products

(11) Exit

Enter your choice (1-11) :11

Thank You...
cp2> clear
cp2> cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Check Point CoreXL
(10) Automatic start of Check Point Products

(11) Exit

Enter your choice (1-11) :11

Thank You...
cp2> cphaprob stat

HA module not started.

Cluster policy should be installed - please run cphastart
cp2> cphaprob -l list

Built-in Devices:

Device Name: Interface Active Check

Device Name: Recovery Delay

Device Name: CoreXL Configuration

Registered Devices:

Device Name: Fullsync
Registration number: 0
Timeout: none
Additional description: Boot

Device Name: Policy
Registration number: 1
Timeout: none

Device Name: routed
Registration number: 2
Timeout: none
Additional description: Cluster Start

Device Name: cxld
Registration number: 3
Timeout: 30 sec

Device Name: HD
Registration number: 4
Timeout: none

Device Name: fwd
Registration number: 5
Timeout: 30 sec

cp2>

[the_rock]

Did you run cphastart?

Andy

[ChinChang]

yeah, run the cphastart.

[the_rock]

I still believe considering cphaprob -a if shows there is no sync configured clearly tells us this is why this does not work. I would clearly examine how you have the topology configured, as clearly something is incorrect. I will send you screenshot from my lab cluster topology tomorrow .

Best,

Andy

[ChinChang]

i got it, i will learn to sync interface this step.

[the_rock]

I will run exact same commands I asked you to run and send you the output in word doc. I will also attach screenshot of the topology, so you get an idea.

I hope it will be helpful.

Andy

[the_rock]

@ChinChang Please review what I attached (its my lab working cluster) and let me know if any questions.

Andy

[the_rock]

@ChinChang I also took short video as well for you.

Andy