topic Re: New Maestro Cluster SIC errors in Hyperscale Firewall (Maestro)

New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 10:07:24 GMT

Hello Checkmates,

Recently we started the process of migrating to Maestro in our DC's.

For that we decided to go with 2xMHO140's and either 3 x SG9300 or SG9400 - depending on the size of the DC.

Racked everything in February, and since then we're battling with some weird things.

Starting from the connections and set-up, we're having a single site with 2 MHO's and each SG's connected in both.
Then we have the first 1 and 2 ports from MHO's connected to our ACI for SG Management - Vlan168 - and then the Uplinks on ports 5, 7, 8, 11 and 12) . For SG's - downlinks - we use port 25 (changed to downlink!) and 27 and 29 .

Now, for the installation process, we re-imaged the appliances with R82 T777 (back in February) and configured the Management, LOM and other standard settings.

We've built an SG (VSNext) with only one appliance, did the JHF60 (again it was back in February) and added it to Management - brand new VM with R82.

Added a 2nd node on the SG, and after everything synchronized we created an VS. Added that new VS to Management and all was fine.

When we wanted to push an updated policy to the newly created SG - VS0 - we got several failures due to different SIC or communication issues (as per below examples). And the FUN begins 🙂 .

From the investigations we started with Support Engineer and our Professional Services guys, we noticed that for whatever reason, when we are doing SIC verifications from Management, either 2 times out of 5 or 3 times out of 5 we get different SIC errors like:

But when we check on the SG directly, we can see that SIC is OK.

That error shows only when we have 2 or more nodes in the Security Group, either with VSNext or without.

We also applied the Certificate Hotfix for the JHF60, still no change and we also did the JHF73 without success - same SIC errors or policy push errors were seen.

We've verified the SIC certificate to be valid and Management knows it, we re-did SIC at least 10 times, no change. As long as we have a 2nd member in Security Group, it's starting to show the SIC error while validating or pushing policies.

Last Saturday, we have re-imaged all the appliances from Singapore DC to R82 T779 and applied JHF91, same behavior.

While waiting for the Support engineer to see with BU what can be wrong, I want to ask if any of you that work with Maestro, have seen this beavior, and

Thank you,

PS: in order to exclude the dedicated Management interface as a possible cause of the problem, we have 3 ports shut-down from MHO side, so right now we are working with a single Interface for the management of SG's. Same SIC errors are seen.....

Re: New Maestro Cluster SIC errors

_Val_ — Wed, 08 Apr 2026 10:23:57 GMT

@Lari_Luoma , @Anatoly Any comments?

Re: New Maestro Cluster SIC errors

emmap — Wed, 08 Apr 2026 11:10:31 GMT

How is your magg bond configured, and does it match what's on the switches? Are the switches doing any sort of MAC learning from outbound packets? If so this would need to be disabled.

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 11:24:39 GMT

Either way, If we go with normal Security Group or with VSNext, the management interface is configured active/back-up .
On VSNext, underneath it's an MAGG while on the V0 it's an WRP interface.

On ACI side we have all ports set as access, no MAC filtering.

[Global] ALVS-SGFW022-s01-01:0> show interface magg1
1_01:
state on
mac-addr 00:1c:7f:4c:18:dd
type magg
link-state not available
instance 500
mtu 1500
auto-negotiation off
speed N/A
ipv6-autoconfig Not configured
monitor-mode Not configured
duplex N/A
link-speed Not configured
comments
ipv4-address Not Configured
ipv6-address Not Configured
ipv6-local-link-address Not Configured

Statistics:
TX bytes:237474217 packets:716004 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:1307319726 packets:2281057 errors:0 dropped:149771 overruns:0 frame:0

SD-WAN: Not Configured

1_02:
state on
mac-addr 00:1c:7f:4c:07:c9
type magg
link-state not available
instance 500
mtu 1500
auto-negotiation off
speed N/A
ipv6-autoconfig Not configured
monitor-mode Not configured
duplex N/A
link-speed Not configured
comments
ipv4-address Not Configured
ipv6-address Not Configured
ipv6-local-link-address Not Configured

Statistics:
TX bytes:182919557 packets:593347 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:1173288484 packets:2164249 errors:0 dropped:148454 overruns:0 frame:0

SD-WAN: Not Configured

[Global] ALVS-SGFW022-s01-01:0>

[Global] ALVS-SGFW022-s01-01:0> show interfaces all
Interface wrp0
state on
mac-addr 00:12:c1:10:00:29
type wrp
link-state not available
instance 0
mtu 1500
auto-negotiation off
speed N/A
ipv6-autoconfig Not configured
monitor-mode Not configured
duplex N/A
link-speed Not configured
comments
ipv4-address 10.18.169.41/21
ipv6-address Not Configured
ipv6-local-link-address Not Configured

Statistics:
TX bytes:237437595 packets:715738 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:1145052615 packets:1333705 errors:321620 dropped:0 overruns:0 frame:0

SD-WAN: Not Configured

Leading to Virtual Switch: mgmt-switch (ID 500)
[Global] ALVS-SGFW022-s01-01:0>

Thank you,

Re: New Maestro Cluster SIC errors

Gennady — Wed, 08 Apr 2026 11:45:41 GMT

Good day!

Do you see any regular packet drop when you ping from the Management Server to the SG Management IP? May it be a general connectivity issue? I see 6.5% of RX errors on magg interfaces. This doesn't look good.

If we see significant packet loss from the Management Server to the SG, then it is expected that SIC verification and policy push fails. We need to narrow down the problem area starting from bottom up.

Re: New Maestro Cluster SIC errors

Gennady — Wed, 08 Apr 2026 11:52:00 GMT

In addition,

Did you have a chance to capture traffic on the SG (all SGMs at the same time) at the moment when you verify SIC from the Management Server?

The fact that the problem appears only if you add more than 1 SGM to the SG points to some distribution problem. It should not affect magg interface until packets from the Management Server in fact arrives on some Data interface or SMO role is flapping (this is very unlikely).

Re: New Maestro Cluster SIC errors

emmap — Wed, 08 Apr 2026 11:54:05 GMT

Not MAC filtering, MAC learning. If the ACI is learning MAC addresses from outbound packets, it's going to be constantly changing the MAC table for the magg interfaces, and packets are going to get lost, leading to SIC failures. On the SG management port, each SGM uses its own MAC address, so MAC learning can break things.

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 11:54:49 GMT

hello @Gennady ,

No we do not have packet loss between Management and SG's.

And even if we would have a packet loss.

If that would be the case, then how can we explain that in an VS created on top of the SecurityGroup, we get 5 out of 5 SIC validations and no errors when pushing the Policy?

Thank you,

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 12:00:02 GMT

Yes we did packet captures on Management and both SG members and shared them with the Support .

Indeed it points to a distribution problem, but the distribution set-up is for the data-path and not for management per my understanding.

SMO role was not flapping as we checked and it's always the first member that we used to build the SG.
still while doing tcpdumps on both members and we were checking SIC, we've seen that almost every time when SIC failed, the other member was showing traffic at one point. So why is shifting, I can't say.

Thank you,

Thank you,

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 12:06:23 GMT

As I know, we don't do any MAC learning/filtering on ACI side.

But, if we were to do any of that, we should have alerts on ACI side for MAC flapping and the Leaf ports that we have the Management connected - those first 2 ports from each MHO.
But since we are with only one port active right now, there is no MAC flapping, unless the Management IP - 10.18.169.41 in Singapore case - jumps between the 2 members for whatever reason....

I'll doublecheck and come back.

Thank you,

Re: New Maestro Cluster SIC errors

emmap — Wed, 08 Apr 2026 12:08:13 GMT

MAC learning would explain the shifting. If you do the tcpdumps with the MACs shown (-e) do you see the inbound MACs change when SIC stops working and you see traffic on the other SGM?

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 12:23:20 GMT

We'll run again the captures and watch the MAC and come back.
Still I repeat, if we are using a single port between MHO's and ACI for management, then we have a single Inbound MAC so ?

Thank you,

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 12:37:50 GMT

@emmap and @Gennady ,

I assume you have Maestro Clusters that you have set.
My questions to you is, from MHO SG management ports to your network, do you connect to an ACI or standard network clusters/stack ?
If it's an ACI like we have, did you configure it as Access Port only or ?

Thank you,

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 12:39:12 GMT

As promised, we checked the ACI side for MAC flapping and we don't have any alerts for the ports that we use for SG Management.

Re: New Maestro Cluster SIC errors

emmap — Wed, 08 Apr 2026 12:39:53 GMT

No it's MAC per SGM, not per MHO/port. If it works stable with one SGM then it's not MAC flapping. Not sure about VSNext if each VS also has its own MAC address on there but that might need checking too.

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 12:50:26 GMT

I have the same understanding, and in order to exclude VSNext, we have wiped and created an standard Security Group as well.
And with that we have the same SIC behavior.

Ty,

Re: New Maestro Cluster SIC errors

Gennady — Wed, 08 Apr 2026 13:13:24 GMT

I do have Maestro configured. Unfortunately, it is on R81.20 and it is connected to regular Cisco Nexus. This is why I cannot be helpful enough for your problem investigation.

However, please, take a look at this SK. It may give you a lead.
sk168181 - Communication problems with ClusterXL clusters connected to Cisco ACI

From general Maestro standpoint. Magg interfaces are excluded from distribution, and it should not be a problem until somehow management traffic is sent/received via a data interface. Usually, it happens because of some routing mistake. I am sure that it was already checked.

If you would like to look into SMO state in more details, then you can try this zdebug command:
fw ctl zdebug -T -d 'SMO,smo' -m cluster + conf

It is very lightweight and shows changes in SMO role.

"-d" puts a filter in kernel to match string "SMO" or string "smo", otherwise module cluster with flag conf returns too many data.

Re: New Maestro Cluster SIC errors

Martijn — Wed, 08 Apr 2026 13:15:31 GMT

Hi,

I am not a ACI expert and not sure how much of ACI features you are using, but I know ACI is some what different than a traditional switched network. Is there a possibility to connect the MAGG to a traditional (non-ACI) switch to see what happens?
Maybe you can rule out ACI.

Don't think it is relevant here, but did you configure a Primary Interface in the MAGG bond? I have seen strange things when creating an Active/Backup bond without a Primary Interface.

Martijn

Re: New Maestro Cluster SIC errors

Sorin_Gogean — Wed, 08 Apr 2026 13:43:59 GMT

ty @Gennady , I will check the SMO state and come back.

As for the ACI and SK you provided, that makes sense, but we should have alerts on ACI side if an IP changes MAC .

As you see, we have the same MAC for the SG IP:

ty,
PS: I've run the zdebug on both nodes while checking SIC and getting failures, and I did not get any packets, so during the SIC check SMO was not changing, I guess....

Re: New Maestro Cluster SIC errors

Gennady — Wed, 08 Apr 2026 13:54:40 GMT

If there are no messages in zdebug during the problem replication. then we can rule out SMO flap.

An example of SMO role change is below: