Lower Number of CoreXL Instances in ClusterXL HA

[ED]

Hi,

ClusterXL setup with HA.

The active member shows this.

The other member shows this:

It says that a lower number of CoreXL instances was detected on the other member. Which is correct when I check.

The active member:

The member that is in Ready state:

What could have caused this? How come SmartConsole doesn't show any red signs in "Gateway and Servers" overview?

What is the correct way to fix this? Increase to 4 CoreXL FW instances on the active member? Will it be an outage then since it will require a reboot and will it failover to the member that is in Ready state? 

[_Val_]

This is by design. if CoreXl settings are not identical, one of cluster members will be in ready state. adjust FW instances to be the same. If you will be doing it by adding additional core on active member, it will have to be rebooted. Since there is no sync, there will be a short connectivity cut whne active goes down and Ready one will take over

[Dmitry_Krupnik]

Exactly. I just want to share the sk98737:

"CoreXL and ClusterXL:

Number of CoreXL FW instances must be identical on all members of the cluster because the state synchronization between members is performed per CoreXL FW instance (e.g., Instance #2 on Member_A can synchronize only with Instance #2 on Member_B).

Note: Member with higher number of CoreXL FW instances will enter the 'Ready' state. Refer to sk42096 (Cluster member is stuck in 'Ready' state)"

Regards,

Dmitry

[ED]

Thanks Valeri, I will test it tomorrow. Do you know what could have caused different CoreXL settings?

In cpconfig -> CoreXL -> I will have to select the number of Firewall Worker cores. Should I choose the number 3?

I am a bit confused how many SND/IRQ cores and Firewall worker cores are set. Could you please tell me. Confused by the standby member that shows fw_3 under CPU 0. 

The active member:

The standby member:

[Dmitry_Krupnik]

Enis,

Default number of the instances in the R77.X and in the R80.X is different. Therefore if one member was upgraded from R77.30 and another R80.X GW was after clean install we can see different number of instances on the members.

You can see the number of instances in the output of the "fw ctl multik stat" command

Or take a look the CPView output

[ED]

Dmitry,

When we upgraded the gateway cluster, we used same method on both and it's probably over a year since we did that. I have never seen this error in the past and the HA was working with no problems. I know that I have not changed it through cpconfig so therefore I find it strange. What I did in the past was just installing the latest JHF and deleting old unneccessary packages in CPUSE.

[Dmitry_Krupnik]

Enis,

For us important to understand, when and after which actions the Cluster's state (number of CoreXL instances) was changed. Could you take a look the all /var/log/messages (messages, messages.1, messages.2, messages.3 etc) logs and find the report from ClusterXL HA about state changing, probably it was recently? Did you have any actions with licenses?

"How come SmartConsole doesn't show any red signs in "Gateway and Servers" overview?" - good question, I will check it in my lab. Tell me please, which version of GAIA and Smart Console are you using?

Regards,

Dmitry.

[ED]

Since I only have license for 4 cores it seems that the active member is correct setup, with 1 SND/IRQ core and 3 Firewall Worker cores which is standard on 4 cores. I should instead reduce the number of Worker cores on the standby member to 3, right?

[Dmitry_Krupnik · Accepted solution]

Enis,

Yes. Please, share the output of the "fw ctl affinity -l -r" and "cphaprob stat" commands after change.

Regards,

Dmitry.

[ED]

Dmitry,

When I tried to change down to 3 Firewall Worker cores, it was already set to 3, like this [3] in the CoreXL via cpconfig. But cpview was showing 4 CoreXL instances. So I set it down to 2, rebooted and changed back to 3 again and rebooted. Now everything seems fine. 

After changes:

Before changes (even if it number [3] was shown on the CoreXL setting via cpconfig):

Makes sense? At least HA is now in Active and Standby mode on the two FW's.

[Dmitry_Krupnik]

Hi Enis,

I am glad to hear, that the issue was resolved. If it reproduce again, I would recommend to open a ticket in the TAC and inform me by email, which available in my profile.

For us important to understand, when and after which actions the Cluster's state (number of CoreXL instances) was changed. Could you take a look the all /var/log/messages (messages, messages.1, messages.2, messages.3 etc) logs and find the report from ClusterXL HA about state changing, probably it was recently? Did you have any actions with licenses?

[ED]

Hi Timothy Hall‌

Do you think this could be some case of what you call, the trial license "Core Crunch"?

Since it's an open hardware firewall with 6 physical cores but only license for 4 cores. With 6 cores, default is 4 firewall worker cores and as we can see on the pic above that it is from fw_0 to fw_3. The strange thing is that this should occur now and that it did not impact on HA cluster before. I suspect something happened with the latest installations of HF's and rebooting of the gateways. Just because HA was working fine before. 

[Timothy_Hall]

It could have been a core crunch if you have exactly six physical cores, as the default number of workers/instances that will be allocated on a trial/eval license will be 4 in that case.  If you have 8 or more physical cores there would have been the number of physical cores minus 2 number of instances.  In the case of 8 physical cores, you would have had six instances battling each other for only three physical cores once the permanent 4-core license is applied, not a pretty sight and this was covered in my book.

--
"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com