Hardware upgrade CoreXL issue

[nemezis_rock]

Hi mates,

Planning to upgrade from 6000-series to 9000-series without downtime. Two appliances of 6000-series in ClusterXL (HA-cluster).

Facing one prerequisite problem:

There is a prerequisite regarding coreXL count - they must be same number on both old and new hardware. But 9000 series doesnt allow me to change corexl instances count less than 10:

Illegal number of instances: 3 (should be between 10 and 20)

Did anyone faced such problem? Any solution?)

[emmap]

Your 9000s will have Dynamic Balancing enabled, and I think will just work it out for you. Worst case you may experience some connections having to re-establish - you'll know before there's any impact though, if the 9000 doesn't enter the Standby state and remains Ready or Down.

[nemezis_rock]

Hi emmap,

Thanks. Two quick things:

1. Is this based on a configuration you've actually tested/run in production — specifically a member with a lower CoreXL instance count (and no Dynamic Balancing) syncing to a bigger member with more cores and Dynamic Balancing ON? Or is it expected behavior in general?
2. Is there any SK or official source that states Dynamic Balancing allows state sync between members with different CoreXL instance counts? I couldn't find anything close in the SKs — the docs only say the instance count must be identical.

TAC is still delaying, so any first-hand confirmation (even "I did exactly this on R8x and it came up Standby") would really help.

[Bob_Zimmerman]

My team recently upgraded a bunch of QLS250s from R81.20 to R82.

R81.20 on that box reserves one core for fwd by default. Most of them were running 8 CoreXL_SNDs, 39 CoreXL_FW, and 1 FWD.

R82 no longer does this, and it uses all cores for SNDs or worker threads. When we upgraded, the first upgraded member went to 8 CoreXL_SNDs, 40 CoreXL_FW. They synchronized fine from fewer to greater, and the 40th worker thread simply didn't have any connections (as shown by 'fw ctl multik stat'). We didn't try failing back from R82 to R81.20, but we expect that would have involved an outage.

[Timothy_Hall]

As emmap said, you may have some connections get lost during the transition due to the inability to sync between models with a different number of Firewall Worker Instances defined.  If you would prefer to avoid this, one old trick I would use is to unset the "Drop out of state TCP packets" checkbox under Global Properties before starting the upgrade, and then reinstall policy to the cluster before starting the upgrade.  This will blunt the effect of non-synced connections and allow them to be resurrected into the state table without restarting them.  Be sure to "exercise" as many of these connections as possible during your test plan after the first member has been upgraded and you have failed over onto it.  Once the upgrade is complete, don't forget to re-check this box and re-install the policy!

Another possibility is setting fwha_allow_different_corexl_instances to 1 on all involved members before starting the upgrade, which will allow state sync to occur between cluster members with different numbers of Firewall Worker Instances.  However, I believe this variable is intended for use with Maestro's "mix and match" feature and may not achieve the desired effect in an upgrade scenario.  Good luck!

[Bob_Zimmerman]

I know on recent versions (at least R82, but maybe earlier), fewer cores can sync to more cores. The additional cores on the bigger member just don't get any connections until it's active. This means failing from a smaller member to a bigger member should maintain all connections, but failing back will probably involve an outage.

I haven't yet had a chance to test the limits of this to see if, for example, 10 cores can sync to 40 cores.