Details and risks about Dynamic Balancing on Maestro

[kamilazat]

Hello everyone.

- Maestro Dual Site (HA), R81.20, JHF Take 146
- 4x 9200 appliances on each site.

We've noticed that Dynamic Balancing (Dynamic Split) is disabled on the gateways, so we want to enable it. As far as I remember, the gateways come with DS enabled by default, and we don't know why it's disabled. So we need to try to understand why it's disabled, and how to re-enable it on a production setup with preferably no traffic loss.

As per CoreXL Dynamic Balancing we checked the requirements:

- CoreXL enabled

- 9200 (part of supported 9000 appliances)

- Firewall is User mode, SecureXL KPPAK

- No VSX, MDPS, ElasticXL etc.

- GNAT enabled: 
[Expert@Hostname-ch02-01:0]# g_fw ctl get int fwx_gnat_enabled
-*- 8 blades: 1_01 1_02 1_03 1_04 2_01 2_02 2_03 2_04 -*-
fwx_gnat_enabled = 1

- fwkern.conf has these values on all SGMs:
fwha_ssm_type=16
fwha_ch_arp_forwarding=1
fwha_mbs_mixed_appliance=0

- DS config file:
[Expert@Hostname-ch02-01:0]# cat $FWDIR/conf/dynamic_split.conf
ALPHA=10
EMERGENCY_CPU_HANDLING_THRESHOLD=40
LOG_LEVEL=0
STD_DIV_MULTIPLAYER=1
DEFAULT_BOOT_STATUS=1
MACHINE_CONFIG_OK=0
VERBOSE_DEBUG=0
STATE_VRIFICATION_FAILURE_DETECTED=0
MONITOR_MODE=0
CLUSTER_MODE=1
AUTOMATION_MODE=0
INITIALIZATION=0

- fwaffinity.conf set to auto and contains "i default auto", and Multi-Queue untouched.

- No custom weight distributions:
[Global] Hostname-ch02-01> show smo security-group sgm-weight current
Chassis: Blade: State: Distribution:
1 1 Active 25.0% 128/512
1 2 Active 25.0% 128/512
1 3 Active 25.0% 128/512
1 4 Active 25.0% 128/512
2 1 Active 25.0% 128/512
2 2 Active 25.0% 128/512
2 3 Active 25.0% 128/512
2 4 Active 25.0% 128/512

And there are some weird indications that we don't quite understand:

- SMT status unknown. 
Expert@Hostname-ch02-01:0]# cat /proc/smt_status
cat: /proc/smt_status: No such file or directory

- cpconfig shows us that FW cores are supposed to be 6:

This machine has 8 CPUs.

Note: All cluster members must have the same number of firewall instances enabled.

How many IPv4 firewall instances would you like to enable (2 to 8) [6] ?

- CPVIEW shows a weird distribution:
CPVIEW.CPU
Overview SysInfo Network CPU I/O software-blades Hardware-Health Advanced
Overview Top-Protocols Top-Connections Spikes Processes
Host

Overview:

CPU type CPUs Avg utilization
CoreXL_SND 4 24%
BOTH 3 71%
CoreXL_FW 1 62%

CPU:

CPU Type User System Idle I/O wait Interrupts
0 CoreXL_SND 0% 28% 72% 0% 690,262
1 CoreXL_SND 0% 27% 73% 0% 690,298
2 CoreXL_SND 0% 20% 80% 0% 690,298
3 CoreXL_SND 0% 20% 80% 0% 690,302
4 BOTH 16% 49% 35% 0% 345,160
5 BOTH 22% 50% 28% 0% 345,159
6 CoreXL_FW 34% 28% 38% 0% 345,159
7 BOTH 21% 55% 24% 0% 345,164

- And again in CPVIEW we see "CoreXL instances" value as 4.
CPVIEW. SysInfo
Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced
Configuration Information:

Platform Gaia 64Bit
Configuration Check Point Security Gateway
CoreXL Status On
CoreXL instances 4
Dynamic Balancing Status off
SecureXL Status On
USFW Status On
UPPAK Status off

Now, we want to clean this situation up. But first I need to understand why we see such discrepancies. And then I want to use your experience to avoid unwanted scenarios and risks. 

After settling with the question marks above, I'm thinking about enabling DS in this order:

1. Enable DS globally
g_dynamic_balancing -o enable

2. Reboot all SGMs on the Standby chassis (say, chassis 2).

3. Failover to chassis 2.

4. Reboot all SGMs on the new Standby chassis (now chassis 1).

5. Failover back to chassis 1.

What do you think about the procedure and why do you think we have DS disabled in the first place? Any issues that you've encountered in a similar situation?

I'd heavily appreciate all opinions 🙂

Cheers!

[Timothy_Hall]

If you have ever configured a custom static CoreXL split on your firewall (including before upgrading to R81.20, when it was first supported on Maestro), Dynamic Split (DS) will be off.  

Also, any custom changes to Multi-Queue can inhibit DS from starting.

To get DS going:

1) Run mq_mng --show, make sure all interfaces are showing "Auto". This changes to "Dynamic" once DS is successfully started.

2) Undo the custom split and revert to the default number of instances (6 for an 8-core firewall), reboot, and Dynamic Split should be enabled.  The dynamic_balancing -o enable command can also accomplish setting the split back to default for you.

If it is still not working, check the following two files for error messages; the issue will probably be something like sk184372: Dynamic Balancing stays in the "Initializing" stage, or this sk163815: "Dynamic Balancing was disabled due to state verification failure" log error.

$FWDIR/log/dynamic_split.elg

$FWDIR/log/dsd.elg

[kamilazat]

Hi @Timothy_Hall 

Yes, I forgot to check the MQ configuration, since it's almost always in Auto. And we see that it's in Manual mode right now:

------------------------
Affinity of Multi-Queue IRQs
------------------------
Note: 'cpmq' is deprecated and no longer supported. For multiqueue management, please use 'mq_mng'
Current multiqueue status:
Total 8 cores. Available for MQ 4 cores
i/f driver driver mode state mode (queues) cores
actual/avail
------------------------------------------------------------------------------------------------
ethsBP1 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP1-01 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP1-02 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP1-03 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP1-04 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP2 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP3 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP4 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP5 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP6 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP7 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP8 igb Kernel Up Manual (4/4) 0,1,2,3

So next steps will be to;

1. Enable Multi-Queue
mq_mng --set-mode auto

2. Enable Dynamic Balancing
dynamic_balancing -o enable

Now onto the second phase, implementation. Since this is a Maestro environment, and I don't have a Maestro lab at my disposal to test things, I believe we need to tread carefully to avoid traffic issues.

- Do you think we can enable both guys and finish the job in one reboot?

- What do you think about the procedure for minimum (preferably zero) downtime?

1. Enable DS globally
g_dynamic_balancing -o enable
2. Reboot all SGMs on the Standby chassis (say, chassis 2).
3. Failover to chassis 2.
4. Reboot all SGMs on the new Standby chassis (now chassis 1).
5. Failover back to chassis 1.

[Timothy_Hall]

I'd suggest making the MQ changes from clish (followed by a save config, reboot, and recheck of MQ state for Auto mode) to ensure that the changes stick.  Once that's complete, you can enable Dynamic Split globally and start rebooting. I wouldn't try to make both the MQ and DS changes for a single reboot, although it may work.

Also, because these changes are similar to doing a code upgrade (or static CoreXL split adjustment) to one SGM at a time, I'd recommend reviewing the PMTR-74532 section of Known Limitations in sk164155: CoreXL Dynamic Balancing, to avoid any possible traffic handling issues as you may see some alarms in the Security Group until all members have been modified.

[kamilazat]

@Timothy_Hall Is there a way to do mq_mng --set-mode auto for all interfaces using clish? I have failed to find such a command in the documentation. Is there a difference in how mg_mng and clish commands work?

[Timothy_Hall]

Not that I can think of.  I suppose mq_mng --set-mode auto might actually update the clish startup config, although things done like that in expert mode normally don't survive a boot.  I'd try the mq_mng command, then check MQ config of all interfaces in clish.  If it looks good do a save config just before rebooting to be sure.

[Lesley]

Reboot one stand-by SGM before you make the changes to mq. After reboot make sure it is auto. 

If you do all changes at once you cannot know if it goes wrong, what the cause was. 

If it all looks good you can change the dynamic balancing and after that make it active. 

I also checked the latest bugs and I dont see any at the moment for dynamic balancing. 

[kamilazat]

@Lesley Fair point. At this point what concerns me more is that even during a MW we cannot shut all traffic down to reduce the load on the SMO. 

Do you think making changes and reboots on the chassis 2 (standby), and then doing a failover from ch1 -> ch2 can at least allow us to keep processing whatever traffic is there during the MW? 

These gateways are located at a very critical position, where stopping all traffic is simply not possible. So we're afraid that we'll lose access to the SMO when it stays the only gateway if we follow PMTR-74532.

[kamilazat]

Hi everyone.

In case anyone else is/will go through a similar procedure, I'll briefly write down the steps we took. It worked for us, hope it works others as well.

Assume Chassis 1 is active and Chassis 2 is standby.

1. Checked overall health with usual diagnostic commands, like asg diag verify, hcp -r all etc.

2. We took Chassis 2 DOWN.

asg_chassis_admin -c 2 down

3. We set the default number of FW instances not on cpview, but using the dynamic split command in gclish:

set dynamic-balancing state enable set_default_fw_instances

4. We enabled the parameter fwha_allow_different_corexl_instances so that ClusterXL allows us to do failovers even when there are different FW/SND distributions among sites.

g_fw ctl set -f int fwha_allow_different_corexl_instances 1

5. Rebooted all SGMs in Chassis 2, and waited until all of them got back together with ACTIVE (!) state, while Chassis 2 still was DOWN.

6. Brought Chassis 2 up, did a failover from Chassis 1 to Chassis 2. Chassis 2 became ACTIVE.

7. Took Chassis 1 DOWN.

asg_chassis_admin -c 1 down

8. Rebooted all SGMs in Chassis 1, and waited until all of them got back together with ACTIVE (!) state, while Chassis 1 still was DOWN.

9. Brought Chassis 1 back up, did a failover from Chassis 2 to Chassis 1. Chassis 1 became ACTIVE.

Now at this stage you might expect everything to get back to normal, but it didn't in our case, probably because we had auto-clone disabled (we had a Mix & Match on this SG).

10. As a last step we restarted Dynamic Balancing a couple of times until everything showed desired outputs.

g_dynamic_balancing -r

11. We reverted the fwha_allow_different_corexl_instances parameter back to 0.

sed -i '/^fwha_allow_different_corexl_instances=1$/d' $FWDIR/boot/modules/fwkern.conf

12. Ran system diagnostics again, just in case.

As you may or may not have noticed, we didn't touch MQ settings at all. In the beginning it was globally in Manual mode, and in the end it was globally Dynamic. So turns out that you don't need to manually set it to Auto if you're making these changes both on MQ and DB.

What increased our adrenaline during the procedure:

- There was still a bit of traffic going through the gateways during the whole operation, and we received occasional connection loss complaints. Could be some long living TCP sessions, but I've got no detailed information from customer's infra.

- We monitored the system constantly using commands below.

asg stat -v
g_allc "fw ctl affinity -l -v -a | grep fw_"
g_allc "mq_mng -o | grep -v igb"

And sometimes the outputs didn't make sense. But we.. ahem.. rawdogged it, because we were told both by TAC and PS that those weird outputs will be expected. In the end, after resetting Dynamic Balancing a couple of times on both sites, everything automatically got resolved. Again, it may be because we didn't have auto-clone enabled.

Thank you all for providing ideas, that helped us help TAC to come up with a proper plan of actions.

Hope it'll be useful for someone in the future.

Cheers!