Unknown MAC address used by Standby node in Cluster

[Nandhakumar]

Is this expected behaviour design that Checkpoint Standby node in cluster will access internet, IPS update, contacting internal servers etc., happens through Sync interface?

Pinged Google DNS 8.8.8.8 from Standby node and captured traffic on Standby firewall using tcpdump and observed traffic echo request packets on sync interface but there is no reply. Further observed there is difference in Source MAC address which leads to VMWare ESX host drops packet.

I have changed Forged transmits option to Accept from Reject in ESX portgroup fixed the issue. However, I would like to know from Checkpoint experts about below Unknown MAC addresses.

Below packet capture from Standby node sync interface

02:56:2e:00:00:01 (Unknow MAC) > 00:0c:29:XX:XX:XX (Active Node Sync interface MAC), ethertype IPv4 (0x0800), length 98: X.X.X.X (Standby Node internet interface IP) > 8.8.8.8: ICMP echo request, id 16914, seq 115, length 64

00:01:00:00:fd:01 (Unknown MAC) > 00:0c:29:YY:YY:YY (Standby Node Sync interface MAC), ethertype IPv4 (0x0800), length 98: 8.8.8.8 > X.X.X.X (Standby Node internet interface IP): ICMP echo reply, id 16914, seq 28, length 64

[Chris_Atkinson]

Standby members traffic via Sync is expected (R80.40) and higher, refer: sk167453

Note sk169154 (section 3.4) provides a mechanism to alter the behaviour if needed.

[Nandhakumar]

Thanks for that information. However, I would like to know Unknown MAC address belongs to Checkpoint or not? How can we ensure that?

There is no aymmetric connection issue here.

[the_rock]

Chris is correct it is indeed expected behavior. As far as unknown MAC, can you verify if that MAC is indeed showing either in cphaprob -a if or ifconfig -a command?

Andy

[Nandhakumar]

There is no such MAC address obderved by running suggested commands

[Chris_Atkinson]

Do you see the same mac-address when both cluster members are on the same ESX host?

Possibly correction layer mac-address TAC should be able to help confirm.

[Nandhakumar]

I dont think so this is specific to ESX host. I am getting same in Checkpoint appliance standby node as well.

[the_rock]

But your cluster works fine otherwise? No failover issue?

[Nandhakumar]

Cluster works absolutely fine. Only issue with Standby node unable to communicate outside world, that too fixed after allowed Forged transmits in ESX but we want to understand from Checkpoint why this behaviour and where these MAC address were getting generated?

Why it not forward packets with actual standby sync interface MAC address as Source MAC?

[the_rock]

I dont have any more ideas, sorry mate. Maybe open a TAC case and see what they say. Personally, I had never seen that before.

[Nandhakumar]

No issues @the_rock . Have already TAC case opened but no useful update from them. I have figured out all these things by myself. Let me see, what they will answer for this Unknown MAC.

Thanks for your response 🙂

[G_W_Albrecht]

One MAC is from EQUIP'TRANS (see above) - 02:56:2e:00:00:01 is not tied to a vendor. All CP appliances interface HW MACs are in the form: 

00:1C:7F:xx:yy:zz

[G_W_Albrecht]

MAC-Segment:	00:01:00:00:00:00 - 00:01:00:FF:FF:FF (MA-L)
Hersteller:	EQUIP'TRANS
Adresse:	31 rue Paul Cezanne LA ROCHETTE 77000 FR

[_Val_]

Those are MAC addresses used to forward traffic from a Standby member. Check why the standup member is actually receiving traffic that needs to be forwarded.

[Nandhakumar]

Not getting your point. @G_W_Albrecht @_Val_  Standby member uses same kind of MAC address to forward traffic to other hosts irrpespective of type of deployment (Deploy in Physical server such as HP or deploy as VM in esx host or Checkpoint appliance).

So, Checkpoint has to answer.

[_Val_]

Check Point is two words. 

What I see from your original post, you see traffic being forwarded from the standby member to the active member. This is normal for some limited scenarios, where a few connections may still be handled by a standby member after failover. These connections should be then forwarded to the active member and sent out.

To do that, a standby member is forwarding packets using an artificial MAC address. This is part of ClusterXL tech. The last octet of that artificial MAC address is a function of your cluster ID.

Depending on the version of your FWs (which you failed to mention) this forwarding can be done via sync or via production interfaces of your cluster. 

If you see a lot of forwarded traffic, you should investigate why production traffic hits the standby and not the active member, and fix the issue.

In your case, who is pinning 8.8.8.8 through standby member?

[Nandhakumar]

Cluster running in Gaia R81.10 version. I only ran ping 8.8.8.8 to test internet connectivity from standby firewall. Production traffic always hits active firewall not stanbdy. There is no issue with production application traffic.

Here, the issue that we are talking about connection initiated by standby member. Probably, I will wait for assigned engineer from Check Point to address my queries through remote session.

[the_rock]

Keep us posted, very interesting issue indeed.

[_Val_]

It is only interesting if one does not know how ClusterXl works, @the_rock  🙂

[the_rock]

@_Val_ lol, thats it : - )

[_Val_]

This is by design, see about forwarded traffic once mode. 

[Nandhakumar]

Still my question not answered. Where is the proof from Check Point, this is by design? Can you point to any SK article or public document please?

Sending traffic from Standby node to Active node for any communication by design is okay to accept but packets with unknown source MAC address is something I want to know and ensure that it belongs to checkpoint. Remember ESX blocking this traffic by MAC address Forging. 

[the_rock]

I agree with you 100%. Please keep us posted what TAC says.

[_Val_ · Accepted solution]

Dear @Nandhakumar 

@Chris_Atkinson has mentioned above sk167453, but indeed, it does not apply to your case directly, and the forwarding MAC is not described there. 

However, there are other documentation sources. Please look into the forwarding chapter of the older ClusterXL ATRG (page 41): https://supportcenter.checkpoint.com/supportcenter/portal/role/supportcenterUser/page/default.psml/m...

It explains how traffic is forwarded and how the special forwarding MAC address is formed. Look up fwha_mac_forward_magic parameter (page 42 of the document).

Note the ATRG is for R80.10 and below. In later documentation forwarding mechanism is no longer mentioned. Yet forwarding MAC still applies.

If you need an official answer, please open a case with TAC.

Hope this answers your question. If not, feel free to elaborate further.

[Nandhakumar]

Thanks so much @_Val_  for the information. I could see fwha_forward_mac_magic has decimal value of 253 which is equivalent to 'FD' in hexadecimal. Also, I assume that it takes 00 (as ID of target member) for member with highest priority and 01 (as ID of target member) for member with lower priority for version R81.10 atleast, I have verified with few cluster gateways and got same id in source MAC. Also Can you confirm in R81.10, the default forwarding Source MAC format is something like this "00 01(not sure what this value but i could see its same in all cluster members) 00 00 fd (forward mac magic id) 01/00 (id of target cluster member)" ?

Can you correct me if anything I misunderstand in above sentence?

I also would like to know actual source MAC address where connection initiated i.e., 02:56:2e:00:00:01 (Unknow MAC). How this was calculated?

[_Val_]

Both magic MAC and forward-magic MACs are function of a Cluster-ID, which is in your case 1. They differ in 1 bit.

The source forwarding MAC is the reversal of forward-magic MAC, i.e. 00:00:00:00:00:00 minus your forward_magic MAC.

Once more, these are very advanced settings that are not well documented. 

[the_rock]

By the way, @G_W_Albrecht provided the right info...not sure if he used same site to look it up, but below is what I used.

https://macaddress.io/mac-address-lookup/0V5glOjVkq

02:56:2e:00:00:01 MAC address details

Vendor details

None

Block details

The MAC address does not belong to any registered block.

00:01:00:00:fd:01 MAC address details

Vendor details

OUI00:01:00 

Is privateFalse

Company nameEquip'Trans

Company address31 rue Paul Cezanne LA ROCHETTE 77000 FR

Country codeFR

Block details

Is registeredTrue

Border left00:01:00:00:00:00

Border right00:01:00:FF:FF:FF

Block size16,777,216

Assignment block sizeMA-L 

Date created08 November 2000

Date updated26 September 2015

MAC address details

Is validTrue

Virtual MachineMicrosoft Virtual PC / Virtual Server 

Transmission typeUnicast 

Administration typeUAA 

Applications Not detected

Wireshark notes No details