Maestro SG Duplicate MAC Addresses

[Alex_Shpilman]

Hi there,

Recently migrated from ~30 open servers to Maestro and on our last migration discovered something quite concerning.

In general, we have single-site deployments in different various sites.

On our last migration, we had some weird issues with VLANs that stretch between DCs, then we realised that the SG interfaces in both DCs have the same MAC. Started to check other SG's and not surprisingly the duplication was consistent. 

After playing in a non-prod SG, came to the theory below about how MAC addresses generated:

Looking on MAC address as 00:1c:7f:A1:B1:C1, where A1, B1 and C1 are variables.

00:1c:7f = Check Point reserved

A1:B1 = interface designator, where A1 is the MHO designator 81 for MHO1, 82 for MHO2, B1 is the port ID in hex

C1 = security group  designator, 64 for SG id 1, 65 for SG id 2, etc

So for example:

00:1c:7f:81:05:64 = MHO1 interface 5 (i.e. eth1-05) in SG id 1

00:1c:7f:82:05:64 = MHO2 interface 5 (i.e. eth2-05) in SG id 1 

Now Bond interfaces inherit the MAC address from the first added slave interface, so this can be influenced by the order slaves are added.

We managed to change the MAC address of the migrated site this way (before actually coming to this theory).

Firstly, would like to share this, if it helps anyone. 

Secondly, did anyone experience similar behaviour? Any way to change/tweak this rather than changing the SG id or physical port?

Cheers.

[Alex_Shpilman]

@Anatoly may be you can help here please?

[Anatoly]

I think, my colleague already answered on this by mail:

The macs in SP environments are calculated every boot the same way on all SP setups (hardcoded), therefore, the behavior is expected.

Bond is inherit the mac of the first slave. Once you delete eth1-05, the bond mac change to eth2-05 mac hence the 82 (81 for eth1-X, 82 for eth2-X) – also expected.

Just make sure it happen on all members (they must have the same MAC).

The MACs are not determine by the cluster but in the bfm level.

[Alex_Shpilman]

Thanks @Anatoly 

We have 2 single-site deployments connected with layer2 for some VLANs, any way to overcome this behaviour rather than manually change the MAC of one of the sites? For now, we "forced" one site to the 82 (by adding the eth2-X first to the bond).

[Chris_Atkinson]

Have you investigated VMAC at all, please see sk165674 ?

[eranas · Accepted solution]

Hi

Bond MAC Address:

As all bond slaves have to share the same MAC, one of the slaves has to be chosen as the MAC address for the bond.

in Scalable Platforms we choose the first added slave (as you already stated).

This decision is persistent (survives reboot) and the same for all members (to assure symmetry).

The only time it can change is when the first slave is removed.

The first index can be found in /config/active (1 is the bond id here):

[Expert@Maestro-ch01-01:0]# egrep "bonding:group:1:.*index" /config/active
bonding:group:1:port:eth1-11:index 1
bonding:group:1:port:eth1-12:index 2
bonding:group:1:port:eth1-10:index 3

As you can see in my case, eth1-11 is the first index, hence it's MAC address was chosen (0x0b = 11 decimal)

[Global] Maestro-ch01-01:0 > show interface bond1 mac-addr
1_01:
mac-addr 00:1c:7f:81:0b:df

2_01:

mac-addr 00:1c:7f:81:0b:df

MAC Addresses conflicts

The last 2 digits are based on the magic mac value:

[Global] DC_MXL_VSX-ch01-01:0 > fw ctl get int fwha_mac_magic
-*- 2 blades: 1_01 2_01 -*-
fwha_mac_magic = 223

Which by default, is derived from the last Mgmt port IP address:

[Global] DC_MXL_VSX-ch01-01:0 > show interface eth1-Mgmt1 ipv4-address
1_01:
ipv4-address 172.23.103.223/24

Which eventually results with the MAC address last 2 digits as df (0xDF = 223):

mac-addr 00:1c:7f:81:0b:df

Conclusions

Probably in your case the same broadcast domain was used for multiple subnets, so the last octet was overlapped (e.g a setup with IP address of 100.1.1.100/24, another one with 200.1.1.100/24, etc.).

You can modify the magic mac in order to overcome the collisions and have more MAC address uniqueness.

In Scalable Platforms, it has to be done by modifying the /etc/smodb.json file. Please refer to sk25977 - "Connecting multiple clusters to the same network segment (same VLAN, same switch)", refer to section "Change Source MAC Addresses - Gateway & VSX Mode - Gaia Scalable Platform R80.20SP, R80.30SP - Procedure".

Cheers

[Alex_Shpilman]

Thanks @eranas 

We came to the same conclusion ourselves and just noticed the updates here (very late)...

The issue we had is 2 SGs from different deployments connected with layer2 and have the same last 4th octet in their mgmt IP, therefore mac magic was the same as you suggested, changing one end fixed the problem.

Also, adding slaves into a bond with a different order would affect the master mac, this could be used as a temp workaround until a full outage is possible.

[Mauricio_Braca1]

Now you can use asg_unique_mac_utility to change the MAC address.

https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Maestro_AdminGuide/Content/T...