Posting a general procedure for a ClusterXL hardware refresh (member-by-member) where the new appliances have more CPU cores / more CoreXL FW instances than the old ones and the counts cannot be matched.
Context: ClusterXL HA, same software version + JHF on both old and new members. Old members run a low number of CoreXL FW instances; new members have many more cores, and the minimum selectable CoreXL instance count is higher than the old members' — so matching them is impossible. Dynamic Balancing is ON on the new members.
Procedure (member-by-member, start with the Standby):
- Pre-stage the new member: same software version + same JHF Take as the cluster, interfaces/bonds/VLANs configured with the same member IPs as the member it replaces, sync interface, licenses installed, SIC activation key set. Leave Dynamic Balancing as-is.
- Down the old Standby (clusterXL_admin down), power it off, cable in the new member (data, sync, mgmt). Verify links are up, sync is L2-connected, and mgmt reaches the SMS.
- SmartConsole → Reset SIC → Initialize. In the cluster topology, set the per-member interface names in the interface Advanced section (the cluster keys on IP/network, so different names per member are fine during the transition). Install Policy.
- Verify before any cutover: cphaprob state, cphaprob -a if, cphaprob syncstat, fw ctl multik stat.
Expected behavior during the mixed phase (different instance counts):
- The new (larger) member may not reach a clean STANDBY — it can be forced toward DOWN by a CoreXL instance mismatch (CLUS-113900: STANDBY -> DOWN | Reason: Mismatch in the number of CoreXL FW instances). On some builds this is non-blocking by design (sk182805); on others the parameter fwha_allow_different_corexl_instances (sk42096) is needed — confirm with TAC whether it exists/applies in your build.
- Even with the mismatch, fullsync completes and connections on the matching instance IDs (0,1,2,…) sync correctly — confirm with fw ctl multik stat (matching IDs loaded, the extra instances idle until the member is active).
- Failover direction matters: fewer → more instances preserves connections; the reverse interrupts the connections that were on the extra instances.
Cutover for the first member (if the new one won't hold STANDBY due to mismatch):
- A controlled hard cutover works: cpstop on the active old member + clusterXL_admin up on the new member. Because matching-ID connections are already synced, traffic moves over without a noticeable interruption (in our case a live video call didn't drop).
- Treat the mixed phase as a short degraded-HA window, not a stable state.
Second member:
- Replace it the same way (down → swap → Reset SIC → topology → Install Policy). No hard cutover needed now — with both members on the same (higher) instance count, the mismatch disappears: the new member comes up as a clean STANDBY, the active drops ACTIVE(!) → ACTIVE, and the COREXL/LPRB PNOTEs clear. Normal failover is restored.
Watch-outs from the field:
- Transceiver (SFP) compatibility on the new ports — links can show No Link with unsupported optics.
- CCP (UDP 8116) continuity for monitored VLANs across the switch path between members — Local Probing monitors the highest and lowest VLAN IDs on a trunk; if CCP/ARP doesn't traverse member-to-member on those VLANs you'll get an LPRB PNOTE even though the gateway/upstream pings fine. Verify with tcpdump -nni <iface> port 8116; fix on the switch side.
- Anti-spoofing / topology on the migrated interfaces — re-check after changing interface names/segments.
Takeaway: a member-by-member ClusterXL hardware refresh to larger appliances is doable with minimal/near-zero downtime even when CoreXL instance counts can't be matched. The mixed phase is a brief degraded window (sync works for matching instance IDs, fewer→more failover preserves connections), and the mismatch self-resolves once both members are the new, larger model.
Thanks @emmap, @Bob_Zimmerman and @Timothy_Hall — your input matched exactly what we observed.
