Validation Request: Maestro Mixed Appliances CoreX...

RickLin

Hi CheckMates and fellow MVPs,

I have a scheduled maintenance window this coming Sunday early morning and I’m looking to validate my execution plan regarding a Maestro Mixed Appliances environment (based on sk162373).

The Current Environment (R82): Currently, I have a single Security Group consisting of:

3x 9100 appliances (8 cores)
6x 6200 appliances (4 cores)

As seen in the attached screenshot generated from the R82 insight command, Maestro is functioning as expected to maintain symmetric load balancing. To match the physical 4-core limit of the 6200s (running 3 FW / 1 SND), the Maestro has restricted the 9100s to 3 FW instances, pushing the remaining 5 cores to SND (3 FW / 5 SND).

The Maintenance Plan: My primary goal is to maximize the performance of the 9100s by restoring their optimal CoreXL split (e.g., 6 FW / 2 SND).

My planned procedure is:

Gracefully remove all six 6200 appliances from this Security Group.
Leave only the three 9100 appliances in the SG.
Reboot all three 9100 appliances simultaneously.

Questions for the Experts / R&D:

Validation of Step 3: Once the 6200s are removed and the 9100s are rebooted, will Auto-CoreXL automatically recalculate and reallocate the 9100s back to a standard 8-core split (e.g., 6 FW / 2 SND) without manual cpconfig intervention?
Future Scenario: Assuming the 9100s successfully return to 6 FW / 2 SND, if we ever attempt to add a 6200 back into this active SG in the future, I assume it will fail to join (stay detached) because the 4-core appliance physically cannot spawn 6 FW instances to match the SG. Can you confirm if this is the exact behavior?

I want to make sure there are no hidden caveats before I execute this on Sunday. Any insights or confirmation would be highly appreciated!

Thanks in advance, Rick

emmap

Hi Rick

Do you have Dynamic Balancing enabled on this security group? If yes, then you shouldn't have to reboot the 9100s. Also from your screenshot it looks like the 9100s are the first gateways in the group, which would normally mean that they have the 'base' CXL config, which should mean they are allowed to go to more than 3 CXL instances already, if Dynamic Balancing is going. I'm not sure it would be though with the 6200s included.

RickLin

Hi @emmap,

Thanks for the quick reply and the sharp observation!

Regarding Dynamic Balancing, I just checked the status and here is the exact output:

1_01 to 1_03 (the 9100s): Dynamic Balancing is currently On.
1_04 to 1_09 (the 6200s): Dynamic Balancing is currently Stopped with the message: (Dynamic Balancing cannot run on Maestro Security Group members with fewer than 4 cores).

As for your point about the 9100s being the first gateways, you hit the nail on the head. The reason for this is historical: slots 1_01, 1_02, and 1_03 were originally 6200 appliances. We gradually hardware-replaced them with 9100s over time.

Because of this, it seems the SG's "base" CoreXL config is still inherited from its original 6200 days, forcing the 9100s to stay at 3 FW instances to maintain symmetric balance across the mixed group.

Given this context—where the base config is rooted in the old 6200s and Dynamic Balancing is only actively running on the 9100s—what is your expectation when I detach the remaining 6200s? Will Dynamic Balancing automatically override that old base config and scale up the FW instances on the 9100s on the fly, or is a reboot (or running cpconfig) still strictly necessary to flush out the old base configuration?

Thanks again for your insights!

emmap

Hi Rick

With dynamic balancing being On for the 9100s I don't expect that you should have to reboot them, they should just adjust automatically after the 6200s are gone. Definitely don't change anything in cpconfig as that would disable dynamic balancing.

RickLin

Hi @emmap,

Thanks for the clear confirmation!

Fingers crossed that the Dynamic Balancing mechanism kicks in and handles the reallocation automatically as expected. I will definitely stay away from cpconfig to ensure I don't break the native dynamic balancing feature. I guess the real moment of truth will be during my maintenance window this Sunday when I actually detach the 6200s from the SG.

I do have one remaining question from my original post regarding the behavior after the 9100s are optimized:

Let's assume the 9100s successfully auto-adjust to their optimal state (e.g., 6 FW / 2 SND). If, for some reason, we need to add a 6200 back into this active Security Group later, what exactly happens?

Scenario A: Does the 6200 successfully join as 3 FW / 1 SND while the 9100s continue running happily at 6 FW / 2 SND? (I assume Maestro requires symmetric FW instances, so this might not be possible?)
Scenario B: Does the SMO dynamically force the 9100s to downgrade back to 3 FW / 5 SND on the fly to accommodate the 6200's hardware limitations?
Scenario C: Will the 6200 simply fail to join the SG (e.g., stay in a detached/down state) because it physically cannot match the 6 FW instances of the existing 9100 members?

Knowing this behavior in advance would be incredibly helpful for our future capacity planning. Thanks again for guiding me through this!

emmap

I believe Scenario B would happen, due to Dynamic Balancing not being supported on the 6200s.

RickLin

Hi @emmap,

Just a quick update following my maintenance window. I went ahead and completely detached all the 6200 appliances from the Security Group, leaving only the three 9100s active.

Unfortunately, the Dynamic Balancing mechanism did not automatically adjust the core allocation. As you can see from the newly attached screenshot, the three 9100s are still stuck at a 3/5 FW/SND split.

It appears that the old "base config" inherited from the 6200 days is still firmly enforced by the SMO, and Dynamic Balancing alone isn't triggering a recalculation to expand the FW instances to 6.

RickLin

Hi @emmap and everyone,

I want to share the final resolution from my Sunday maintenance window. It was quite a journey, but I finally got the 9100s to scale up to the expected 6 FW / 2 SND split!

Here is the complete troubleshooting process and the ultimate fix for anyone who might face this "mixed-appliance base config" issue in the future:

The Investigation: After detaching all the 6200s, I realized the 9100s were still stuck at 3 FW instances. At around 02:00 AM, I ran g_dynamic_balancing -r to force a re-evaluation. Checking $FWDIR/log/dsd.elg, it still showed Orig num. instances: 3 (as seen in my first screenshot; note that we also have IPv6 enabled, hence Orig num. IPv6 instances: 2).

I initially tried rebooting the three 9100s one by one, but after all of them came back up, they remained at the 3/5 split.

The Breakthrough: I then checked the exact state and discovered that Dynamic Balancing was actually Off. When I attempted to turn it back on using g_dynamic_balancing -o enable in Expert mode, the system gave me a very clear and helpful prompt:

Dynamic Balancing requires that: The configured number of IPv4 CoreXL Firewall instances must be equal to the default number of IPv4 CoreXL Firewall instances. Run this command: set dynamic-balancing state enable set_default_fw_instances

The Fix:

I went into gclish and executed the exact command provided by the system: set dynamic-balancing state enable set_default_fw_instances
After applying that, I performed a simultaneous reboot of all three 9100s using: reboot -b 1_01-1_03

The Result: Success! After the simultaneous reboot, the 9100s finally booted up with the optimal 6/2 FW/SND split. Checking the dsd.elg log again confirmed that it now correctly registers Orig num. instances: 6.

Conclusion: It seems that when scaling up from a legacy base config inherited from lower-spec appliances, the SMO won't just dynamically grow the FW instances if it conflicts with the configured baseline. Forcing the default FW instances via gclish and performing a simultaneous reboot of the SG members is the key to breaking that old configuration.

Thanks again for pointing me in the right direction regarding the base config behavior. I hope this detailed breakdown helps others planning similar Maestro hardware upgrades!

Best, Rick

Validation Request: Maestro Mixed Appliances CoreXL Reallocation Procedure (R82)