Details and risks about Dynamic Balancing on Maestro

[kamilazat]

Hello everyone.

- Maestro Dual Site (HA), R81.20, JHF Take 146
- 4x 9200 appliances on each site.

We've noticed that Dynamic Balancing (Dynamic Split) is disabled on the gateways, so we want to enable it. As far as I remember, the gateways come with DS enabled by default, and we don't know why it's disabled. So we need to try to understand why it's disabled, and how to re-enable it on a production setup with preferably no traffic loss.

As per CoreXL Dynamic Balancing we checked the requirements:

- CoreXL enabled

- 9200 (part of supported 9000 appliances)

- Firewall is User mode, SecureXL KPPAK

- No VSX, MDPS, ElasticXL etc.

- GNAT enabled: 
[Expert@Hostname-ch02-01:0]# g_fw ctl get int fwx_gnat_enabled
-*- 8 blades: 1_01 1_02 1_03 1_04 2_01 2_02 2_03 2_04 -*-
fwx_gnat_enabled = 1

- fwkern.conf has these values on all SGMs:
fwha_ssm_type=16
fwha_ch_arp_forwarding=1
fwha_mbs_mixed_appliance=0

- DS config file:
[Expert@Hostname-ch02-01:0]# cat $FWDIR/conf/dynamic_split.conf
ALPHA=10
EMERGENCY_CPU_HANDLING_THRESHOLD=40
LOG_LEVEL=0
STD_DIV_MULTIPLAYER=1
DEFAULT_BOOT_STATUS=1
MACHINE_CONFIG_OK=0
VERBOSE_DEBUG=0
STATE_VRIFICATION_FAILURE_DETECTED=0
MONITOR_MODE=0
CLUSTER_MODE=1
AUTOMATION_MODE=0
INITIALIZATION=0

- fwaffinity.conf set to auto and contains "i default auto", and Multi-Queue untouched.

- No custom weight distributions:
[Global] Hostname-ch02-01> show smo security-group sgm-weight current
Chassis: Blade: State: Distribution:
1 1 Active 25.0% 128/512
1 2 Active 25.0% 128/512
1 3 Active 25.0% 128/512
1 4 Active 25.0% 128/512
2 1 Active 25.0% 128/512
2 2 Active 25.0% 128/512
2 3 Active 25.0% 128/512
2 4 Active 25.0% 128/512

And there are some weird indications that we don't quite understand:

- SMT status unknown. 
Expert@Hostname-ch02-01:0]# cat /proc/smt_status
cat: /proc/smt_status: No such file or directory

- cpconfig shows us that FW cores are supposed to be 6:

This machine has 8 CPUs.

Note: All cluster members must have the same number of firewall instances enabled.

How many IPv4 firewall instances would you like to enable (2 to 8) [6] ?

- CPVIEW shows a weird distribution:
CPVIEW.CPU
Overview SysInfo Network CPU I/O software-blades Hardware-Health Advanced
Overview Top-Protocols Top-Connections Spikes Processes
Host

Overview:

CPU type CPUs Avg utilization
CoreXL_SND 4 24%
BOTH 3 71%
CoreXL_FW 1 62%

CPU:

CPU Type User System Idle I/O wait Interrupts
0 CoreXL_SND 0% 28% 72% 0% 690,262
1 CoreXL_SND 0% 27% 73% 0% 690,298
2 CoreXL_SND 0% 20% 80% 0% 690,298
3 CoreXL_SND 0% 20% 80% 0% 690,302
4 BOTH 16% 49% 35% 0% 345,160
5 BOTH 22% 50% 28% 0% 345,159
6 CoreXL_FW 34% 28% 38% 0% 345,159
7 BOTH 21% 55% 24% 0% 345,164

- And again in CPVIEW we see "CoreXL instances" value as 4.
CPVIEW. SysInfo
Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced
Configuration Information:

Platform Gaia 64Bit
Configuration Check Point Security Gateway
CoreXL Status On
CoreXL instances 4
Dynamic Balancing Status off
SecureXL Status On
USFW Status On
UPPAK Status off

Now, we want to clean this situation up. But first I need to understand why we see such discrepancies. And then I want to use your experience to avoid unwanted scenarios and risks. 

After settling with the question marks above, I'm thinking about enabling DS in this order:

1. Enable DS globally
g_dynamic_balancing -o enable

2. Reboot all SGMs on the Standby chassis (say, chassis 2).

3. Failover to chassis 2.

4. Reboot all SGMs on the new Standby chassis (now chassis 1).

5. Failover back to chassis 1.

What do you think about the procedure and why do you think we have DS disabled in the first place? Any issues that you've encountered in a similar situation?

I'd heavily appreciate all opinions 🙂

Cheers!

[Timothy_Hall]

If you have ever configured a custom static CoreXL split on your firewall (including before upgrading to R81.20, when it was first supported on Maestro), Dynamic Split (DS) will be off.  

Also, any custom changes to Multi-Queue can inhibit DS from starting.

To get DS going:

1) Run mq_mng --show, make sure all interfaces are showing "Auto". This changes to "Dynamic" once DS is successfully started.

2) Undo the custom split and revert to the default number of instances (6 for an 8-core firewall), reboot, and Dynamic Split should be enabled.  The dynamic_balancing -o enable command can also accomplish setting the split back to default for you.

If it is still not working, check the following two files for error messages; the issue will probably be something like sk184372: Dynamic Balancing stays in the "Initializing" stage, or this sk163815: "Dynamic Balancing was disabled due to state verification failure" log error.

$FWDIR/log/dynamic_split.elg

$FWDIR/log/dsd.elg

[kamilazat]

Hi @Timothy_Hall 

Yes, I forgot to check the MQ configuration, since it's almost always in Auto. And we see that it's in Manual mode right now:

------------------------
Affinity of Multi-Queue IRQs
------------------------
Note: 'cpmq' is deprecated and no longer supported. For multiqueue management, please use 'mq_mng'
Current multiqueue status:
Total 8 cores. Available for MQ 4 cores
i/f driver driver mode state mode (queues) cores
actual/avail
------------------------------------------------------------------------------------------------
ethsBP1 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP1-01 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP1-02 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP1-03 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP1-04 ice Kernel Up Manual (7/7) 0,1,2,3,4,5,7
ethsBP2 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP3 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP4 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP5 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP6 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP7 igb Kernel Up Manual (4/4) 0,1,2,3
ethsBP8 igb Kernel Up Manual (4/4) 0,1,2,3

So next steps will be to;

1. Enable Multi-Queue
mq_mng --set-mode auto

2. Enable Dynamic Balancing
dynamic_balancing -o enable

Now onto the second phase, implementation. Since this is a Maestro environment, and I don't have a Maestro lab at my disposal to test things, I believe we need to tread carefully to avoid traffic issues.

- Do you think we can enable both guys and finish the job in one reboot?

- What do you think about the procedure for minimum (preferably zero) downtime?

1. Enable DS globally
g_dynamic_balancing -o enable
2. Reboot all SGMs on the Standby chassis (say, chassis 2).
3. Failover to chassis 2.
4. Reboot all SGMs on the new Standby chassis (now chassis 1).
5. Failover back to chassis 1.

[Timothy_Hall]

I'd suggest making the MQ changes from clish (followed by a save config, reboot, and recheck of MQ state for Auto mode) to ensure that the changes stick.  Once that's complete, you can enable Dynamic Split globally and start rebooting. I wouldn't try to make both the MQ and DS changes for a single reboot, although it may work.

Also, because these changes are similar to doing a code upgrade (or static CoreXL split adjustment) to one SGM at a time, I'd recommend reviewing the PMTR-74532 section of Known Limitations in sk164155: CoreXL Dynamic Balancing, to avoid any possible traffic handling issues as you may see some alarms in the Security Group until all members have been modified.

[kamilazat]

@Timothy_Hall Is there a way to do mq_mng --set-mode auto for all interfaces using clish? I have failed to find such a command in the documentation. Is there a difference in how mg_mng and clish commands work?

[Timothy_Hall]

Not that I can think of.  I suppose mq_mng --set-mode auto might actually update the clish startup config, although things done like that in expert mode normally don't survive a boot.  I'd try the mq_mng command, then check MQ config of all interfaces in clish.  If it looks good do a save config just before rebooting to be sure.

[Lesley]

Reboot one stand-by SGM before you make the changes to mq. After reboot make sure it is auto. 

If you do all changes at once you cannot know if it goes wrong, what the cause was. 

If it all looks good you can change the dynamic balancing and after that make it active. 

I also checked the latest bugs and I dont see any at the moment for dynamic balancing. 

[kamilazat]

@Lesley Fair point. At this point what concerns me more is that even during a MW we cannot shut all traffic down to reduce the load on the SMO. 

Do you think making changes and reboots on the chassis 2 (standby), and then doing a failover from ch1 -> ch2 can at least allow us to keep processing whatever traffic is there during the MW? 

These gateways are located at a very critical position, where stopping all traffic is simply not possible. So we're afraid that we'll lose access to the SMO when it stays the only gateway if we follow PMTR-74532.