ClusterXL with VMAC turned on - can the cluster VIP use the VMAC as the source MAC address

[BigLeBeauski24]

Hello!  

I have a 3200 ClusterXL HA pair of gateways at a client site.  This site uses Cox business broadband Internet.  We are having trouble getting the cluster to perform correctly.  In working with the engineer from Cox, he stated the device on his end of the network is looking for the reply traffic (after sending traffic to the VIP) to be sourced from the VMAC, not the physical MAC of the active member, which is what is happening now.  

Is there a way to force the cluster to respond to the VIP traffic with the VMAC address?  I have been looking at SKs and also posts here, but can't seem to find anything.  

Any help is appreciated!  

Thanks,

Dave

[Vladimir]

@BigLeBeauski24 , version and a few words on topology please.

If R77.30 or below, this may be your culprit:

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eve...

[BigLeBeauski24]

@Vladimir 

The cluster is running R80.30.

Topology:

Two gateways and Cox cable modem connected to small dumb switch.  The modem acts like a bridge in this setup and does not own the default gateway for the WAN.  Subnet is a /26 but there are only 6 useable IPs.  I have heard Cox reference a "CCAP" that owns the default gateway IP for the WAN.  Also Cox has said that their setup does not work well with gratuitous ARP, so we must use VMAC.

Hope that helps!

-Dave

[Vladimir]

Hi Dave,

Still sounds like the problem described in SK with the value-added "small dumb switch".

How dumb is that witch exactly? if the VMAC was configured recently and the ARP cache and MAC table on the COX side was not flashed, this may persist. I have seen relevant behavior (not the VMAC, but swap of one of the Cisco ASAs cluster members with Check Point), where this was required from the carrier to restore sanity to the network.

[BigLeBeauski24]

@Vladimir 

That switch is a 5-port Netgear.  "Dumb" as in it is not managed and is pretty simple and has no "configuration" that could be effecting anything.  The Cox side does have the VMAC in its ARP table (eventually, the ARP table on that device takes hours to refresh, which is why G-ARP does nothing).  

They have a problem with the way traffic from VIP is sourced with VIP and physical MAC of active member.  Sounds like their device drops traffic because it is expecting the VMAC as the source MAC.  I am working with the Cox engineer too, and hopefully he can adjust something on his end to get this working.  I was just throwing the behavior out here on the forum to see if there was any way to get the cluster to use the VMAC as the source MAC.  

-Dave

[Wolfgang]

@BigLeBeauski24 

this is normal behaviour with enabled vmac.

Have a look at How to enable ClusterXL Virtual MAC (VMAC) mode

### snip from sk####

Traffic originating from Active / Pivot cluster member is sent out in the following way:

• Layer 2 Source (MAC) address - physical MAC address of corresponding member's interface.
• Layer 3 Source (IP) address - Virtual IP address that was configured for corresponding cluster interfaces.

### snip from sk####

This is known with some LoadBalancers and some switch devices. ( sk102270, sk117412 )

As @Vladimir mentioned some more information about the devices on the others side will be helpful.

Wolfgang

[BigLeBeauski24]

@Wolfgang 

Thanks. I have read that SK about 5 times lol.  I realize that this is the "normal" behavior, but my question is asking if there is a way to configure this to behave differently.  Just as no VMAC is "normal" but you can still enable it.  

I do not have any information on the ISP side other than what I posted in my reply to @Vladimir .

-Dave

[Wolfgang]

Dave,

normally I prefer vmac. With this there is no change of the receiving MAC if the active clusternode change to the other.

If you want both MACs identical you can‘t use vmac. I think there is no way to change this behaviour.

Trigger @PhoneBoy , maybe something internal ?

Wolfgang

[BigLeBeauski24]

With VMAC, the traffic is sent to the VMAC but the reply is sourced from the physical MAC. That is the issue with this setup. The reply needs to come from the VMAC.

[Vladimir]

Given the constraints, can you spin-up another non-routed L3 VLAN on the switch connected to the internal interfaces of the CP cluster and use that instead of the netgear? With IP assigned to that VLAN, the vMAC should work.

[BigLeBeauski24]

We did have an Internet VLAN setup on the LAN switch originally.  Same results.  We moved to the "dumb" switch to simplify things and see if it made a difference.  

[Vladimir]

Looks like you'll have to route buffer the CP cluster using RFC1918 on its external interfaces, unless someone can suggest anything better.

P.S. When you've had the VLAN previously configured on the internal switch, did it have a VLAN interface with IP address?

[PhoneBoy]

Pretty sure with ClusterXL, we do that by design and there is no way to change it.
Wonder if VRRP has the same limitation?

[Wolfgang]

@PhoneBoy 

thanks Dameon. I thought too VRRP maybe is the solution but I think the behaviour is the same. With VRRP you can too specify special MAC for the VRRP IPs. But the sending traffic should be send with the physical MAC of the node.

Maybe someone here has running a VRRP-cluster.

One solution from the old times would be an IPSO-cluster in forwarding mode, but that‘s not supported today 😉

Wolfgang

[BigLeBeauski24]

Thanks everyone for the help and for jumping in on this.  

Had a 2.5 hour phone call with a CMTS engineer at Cox, and their device is definitely doing some funky stuff with ARP.  He took alot of notes and is going to escalate to Cisco.  

-Dave

[Vladimir]

Thanks Dave! Please keep this thread updated: I am sure this is not an isolated case and the solution may come in handy for some folks.

-Vladimir