Create a Post
Alex_Shpilman
Collaborator

Maestro SG Duplicate MAC Addresses

Jump to solution

Hi there,

Recently migrated from ~30 open servers to Maestro and on our last migration discovered something quite concerning.

In general, we have single-site deployments in different various sites.

On our last migration, we had some weird issues with VLANs that stretch between DCs, then we realised that the SG interfaces in both DCs have the same MAC. Started to check other SG's and not surprisingly the duplication was consistent. 

 

After playing in a non-prod SG, came to the theory below about how MAC addresses generated:

Looking on MAC address as 00:1c:7f:A1:B1:C1, where A1, B1 and C1 are variables.

00:1c:7f = Check Point reserved

A1:B1 = interface designator, where A1 is the MHO designator 81 for MHO1, 82 for MHO2, B1 is the port ID in hex

C1 = security group  designator, 64 for SG id 1, 65 for SG id 2, etc

So for example:

00:1c:7f:81:05:64 = MHO1 interface 5 (i.e. eth1-05) in SG id 1

00:1c:7f:82:05:64 = MHO2 interface 5 (i.e. eth2-05) in SG id 1 

 

Now Bond interfaces inherit the MAC address from the first added slave interface, so this can be influenced by the order slaves are added.

We managed to change the MAC address of the migrated site this way (before actually coming to this theory).

 

Firstly, would like to share this, if it helps anyone. 

Secondly, did anyone experience similar behaviour? Any way to change/tweak this rather than changing the SG id or physical port?

Cheers.

0 Kudos
1 Solution

Accepted Solutions
eranas
Employee
Employee

Hi

Bond MAC Address:

As all bond slaves have to share the same MAC, one of the slaves has to be chosen as the MAC address for the bond.

in Scalable Platforms we choose the first added slave (as you already stated).

This decision is persistent (survives reboot) and the same for all members (to assure symmetry).

The only time it can change is when the first slave is removed.

 

The first index can be found in /config/active (1 is the bond id here):

[Expert@Maestro-ch01-01:0]# egrep "bonding:group:1:.*index" /config/active
bonding:group:1:port:eth1-11:index 1
bonding:group:1:port:eth1-12:index 2
bonding:group:1:port:eth1-10:index 3

 

As you can see in my case, eth1-11 is the first index, hence it's MAC address was chosen (0x0b = 11 decimal)

[Global] Maestro-ch01-01:0 > show interface bond1 mac-addr
1_01:
mac-addr 00:1c:7f:81:0b:df

2_01:

mac-addr 00:1c:7f:81:0b:df

 

MAC Addresses conflicts

The last 2 digits are based on the magic mac value:

[Global] DC_MXL_VSX-ch01-01:0 > fw ctl get int fwha_mac_magic
-*- 2 blades: 1_01 2_01 -*-
fwha_mac_magic = 223

 

Which by default, is derived from the last Mgmt port IP address:

[Global] DC_MXL_VSX-ch01-01:0 > show interface eth1-Mgmt1 ipv4-address
1_01:
ipv4-address 172.23.103.223/24

 

Which eventually results with the MAC address last 2 digits as df (0xDF = 223):

mac-addr 00:1c:7f:81:0b:df

 

Conclusions

Probably in your case the same broadcast domain was used for multiple subnets, so the last octet was overlapped (e.g a setup with IP address of 100.1.1.100/24, another one with 200.1.1.100/24, etc.).

You can modify the magic mac in order to overcome the collisions and have more MAC address uniqueness.

In Scalable Platforms, it has to be done by modifying the /etc/smodb.json file. Please refer to sk25977 - "Connecting multiple clusters to the same network segment (same VLAN, same switch)", refer to section "Change Source MAC Addresses - Gateway & VSX Mode - Gaia Scalable Platform R80.20SP, R80.30SP - Procedure".

 

Cheers

 

View solution in original post

6 Replies
Alex_Shpilman
Collaborator

@Anatoly may be you can help here please?

0 Kudos
Anatoly
Employee
Employee

I think, my colleague already answered on this by mail:

 

The macs in SP environments are calculated every boot the same way on all SP setups (hardcoded), therefore, the behavior is expected.

Bond is inherit the mac of the first slave. Once you delete eth1-05, the bond mac change to eth2-05 mac hence the 82 (81 for eth1-X, 82 for eth2-X) – also expected.

Just make sure it happen on all members (they must have the same MAC).

 

The MACs are not determine by the cluster but in the bfm level.

 

0 Kudos
Alex_Shpilman
Collaborator

Thanks @Anatoly 

We have 2 single-site deployments connected with layer2 for some VLANs, any way to overcome this behaviour rather than manually change the MAC of one of the sites? For now, we "forced" one site to the 82 (by adding the eth2-X first to the bond).

0 Kudos

Have you investigated VMAC at all, please see sk165674 ?

0 Kudos
eranas
Employee
Employee

Hi

Bond MAC Address:

As all bond slaves have to share the same MAC, one of the slaves has to be chosen as the MAC address for the bond.

in Scalable Platforms we choose the first added slave (as you already stated).

This decision is persistent (survives reboot) and the same for all members (to assure symmetry).

The only time it can change is when the first slave is removed.

 

The first index can be found in /config/active (1 is the bond id here):

[Expert@Maestro-ch01-01:0]# egrep "bonding:group:1:.*index" /config/active
bonding:group:1:port:eth1-11:index 1
bonding:group:1:port:eth1-12:index 2
bonding:group:1:port:eth1-10:index 3

 

As you can see in my case, eth1-11 is the first index, hence it's MAC address was chosen (0x0b = 11 decimal)

[Global] Maestro-ch01-01:0 > show interface bond1 mac-addr
1_01:
mac-addr 00:1c:7f:81:0b:df

2_01:

mac-addr 00:1c:7f:81:0b:df

 

MAC Addresses conflicts

The last 2 digits are based on the magic mac value:

[Global] DC_MXL_VSX-ch01-01:0 > fw ctl get int fwha_mac_magic
-*- 2 blades: 1_01 2_01 -*-
fwha_mac_magic = 223

 

Which by default, is derived from the last Mgmt port IP address:

[Global] DC_MXL_VSX-ch01-01:0 > show interface eth1-Mgmt1 ipv4-address
1_01:
ipv4-address 172.23.103.223/24

 

Which eventually results with the MAC address last 2 digits as df (0xDF = 223):

mac-addr 00:1c:7f:81:0b:df

 

Conclusions

Probably in your case the same broadcast domain was used for multiple subnets, so the last octet was overlapped (e.g a setup with IP address of 100.1.1.100/24, another one with 200.1.1.100/24, etc.).

You can modify the magic mac in order to overcome the collisions and have more MAC address uniqueness.

In Scalable Platforms, it has to be done by modifying the /etc/smodb.json file. Please refer to sk25977 - "Connecting multiple clusters to the same network segment (same VLAN, same switch)", refer to section "Change Source MAC Addresses - Gateway & VSX Mode - Gaia Scalable Platform R80.20SP, R80.30SP - Procedure".

 

Cheers

 

Alex_Shpilman
Collaborator

Thanks @eranas 

We came to the same conclusion ourselves and just noticed the updates here (very late)...

The issue we had is 2 SGs from different deployments connected with layer2 and have the same last 4th octet in their mgmt IP, therefore mac magic was the same as you suggested, changing one end fixed the problem.

Also, adding slaves into a bond with a different order would affect the master mac, this could be used as a temp workaround until a full outage is possible.

0 Kudos