Cluster VIP in Local Proxy ARP NET

[Fetakungen]

Hi, first off i'd like to say i'm writing this as a middle man, though i am a network/server tech i do not have the management over the products with the problem. I am an external consultant for a local company with is owned by a big one with central Firewall management.

Anyway to the point, I've installed a Checkpoint FW cluster which the remote guys have configured with 2 addresses and 1 cluster address. According the their policy communication MUST be done through the cluster ip.. and this i somewhere where it fails and as they do not manage to solve it and the ISP says nothing is wrong i feel kinda helpless as the middleman not in the power of any config..

The cluster IP does not manage to reach the GW for any other host except those in the local isp split switch(another brand fw). Nat or direct access to interface IP:s work which temporally allows for outbound connections but not for inbound as this is against policy and is forbidden...

The internet connection is provided in a City Net which uses Local proxy arp (means it answers on all IP:s in the subnet and then relay to the real target) which i think might have something to do with the problem.

Below is a traffic capture from a L2 swtich which sits between the ISP and the Firewalls.

FW1: 00:1c:7f:8d:75:14
FW2: 00:1c:7f:8e:30:8a
CLUSTER: 001c.7f00.2b0d

ISP: 00:00:5e:00:01:28 

Ping to Cluster IP

2:34:50.815365 70:f0:96:59:6e:c2 (oui Unknown) > 00:1c:7f:00:2b:0d (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-71.A163.corp.bahnhof.se > h-208-86.A163.corp.bahnhof.se: ICMP echo request, id 50, seq 3, length 80

Reply from Physical Mac.
12:34:50.815404 00:1c:7f:8e:30:8a (oui Unknown) > 00:00:5e:00:01:28 (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-86.A163.corp.bahnhof.se > h-208-71.A163.corp.bahnhof.se: ICMP echo reply, id 50, seq 3, length 80

Ping to Cluster IP

12:34:52.814614 70:f0:96:59:6e:c2 (oui Unknown) > 00:1c:7f:00:2b:0d (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-71.A163.corp.bahnhof.se > h-208-86.A163.corp.bahnhof.se: ICMP echo request, id 50, seq 4, length 80

Reply from Physical Mac.
12:34:52.814652 00:1c:7f:8e:30:8a (oui Unknown) > 00:00:5e:00:01:28 (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-86.A163.corp.bahnhof.se > h-208-71.A163.corp.bahnhof.se: ICMP echo reply, id 50, seq 4, length 80

Ping to Cluster IP
2:34:50.815365 70:f0:96:59:6e:c2 (oui Unknown) > 00:1c:7f:00:2b:0d (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-71.A163.corp.bahnhof.se > h-208-86.A163.corp.bahnhof.se: ICMP echo request, id 50, seq 3, length 80

Reply from Physical Mac.
12:34:50.815404 00:1c:7f:8e:30:8a (oui Unknown) > 00:00:5e:00:01:28 (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-86.A163.corp.bahnhof.se > h-208-71.A163.corp.bahnhof.se: ICMP echo reply, id 50, seq 3, length 80

Ping to Cluster IP
12:34:52.814614 70:f0:96:59:6e:c2 (oui Unknown) > 00:1c:7f:00:2b:0d (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-71.A163.corp.bahnhof.se > h-208-86.A163.corp.bahnhof.se: ICMP echo request, id 50, seq 4, length 80
Reply from Physical Mac.
12:34:52.814652 00:1c:7f:8e:30:8a (oui Unknown) > 00:00:5e:00:01:28 (oui Unknown), ethertype IPv4 (0x0800), length 114: h-208-86.A163.corp.bahnhof.se > h-208-71.A163.corp.bahnhof.se: ICMP echo reply, id 50, seq 4, length 80

I figured this could be were it goes wrong as it reply from the physical mac and not the cluster mac. (Normal VRRP behavior) The cluster is a XL cluster as far as i know. I figured this and local proxy arp might cause the problem?

The ISP has stated it will not remove local proxy arp for obvius reason as it's needed for private vlans peer to peer communication and also as this is a public /24 and where the customers only have 5 addresses.

Is it possible to change this behavior and or other solution ? Ideas ?

[JackPrendergast]

Hi.

Firstly, could you share or confirm the cluster configuration within Smart Console? Do you have the interfaces correctly defined with the cluster object and is the cluster vip set in there?

Whilst in the cluster settings, do you have clusterXL activated?

and going further than this, if you ran a ‘cpconfig’ on each firewalls, do you have an option to disable or enable cluster membership?

Thanks!

[Fetakungen]

Well first off i do not have access to the config as i'm "local it" in this case. The firewall of the bigger company manage this, but as there are about 200 other sites with same firewalls / config i doubt anything is wrong with the cluster config per say. Rather that something should be changed to be compatible with the ISP net. Regarding cluster mac / local proxy arp.

Traffic comes in on the interface when you try to reach the cluster but the answers / outgoing traffic gets dropped by the ISP GW.

Also how do i edit the first post ?...

[JackPrendergast]

When you say packets are dropped by the ISP GW, where do you see this? Where is packet dropped? If there is another firewall north of this cluster, you will need to make sure there is a rule to allow the cluster IP of this cluster out to the internet on the ISP GW.

[Fetakungen]

Well as there is no response within the subnet at all except from the other brand FW in the same L2 switch, meanwhile that other fw can ping atleast 5-6 other hosts in the subnet. Also packet from the external net reach the cluster but the responses does not pass ISP gw.

[Wolfgang]

@Fetakungen 

following your shown MAC addresses it looks like you are using vmac mode in ClusterXL.

It‘s normal behaviour that outgoing packets are send with the physical MAC of the active member.  How to enable ClusterXL Virtual MAC (VMAC) mode

In rare cases this will be problematic for attached devices if receiving and sending MAC differs. You can disable vmac and then both directions are using the physical MAC of the active member. But if the active node is changing, these MAC changes too. This will be possible problematic with your providers proxy arp entry.

Wolfgang

[Fetakungen]

Thank you, i have asked them to try to disable VMAC mode and we'll see if that solves it. But from your description it certainly should.