Incompatibility between CISCO ACI and VSX Cluster in ClusterXL Mode

[Uwe_Knoetsch]

Hello Community,

we have a two member (13500) cluster with vsx in the clusterXL mode.

Every day at 16:10 they are different connection losts between system's the are connected to cisco aci switch fabrick and the virtual firewall or systems behind the firewall.

What we see: Every day at 16:10 the vsx context standby member try to made a connect to Domaencontroller.
(maybe this is a part of ID-Awareness prozess).

What i have tryed:

Even when I take an ping from the standby member to an system behind the Cisco ACI the ping going out from the standby member with the Cluster vip IP from active vsx member and the mac adress from himself. And that is for the ACI Fabric a problem.

That is a dump from the ping from the standby vsx node to 172.27.100.243, the request going out with the vsx cluster vip.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond2.1062, link-type EN10MB (Ethernet), capture size 96 bytes
10:39:40.385678 IP 172.28.11.81 > 172.27.100.245: ICMP echo request, id 10002, seq 1, length 64
10:39:40.387316 IP 172.27.100.245 > 172.28.11.81: ICMP echo reply, id 10002, seq 1, length 64
10:39:40.491271 IP 172.27.100.245.56565 > 172.24.0.160.microsoft-ds: . ack 3496076707 win 63972 <nop,nop,sack 1 {0:1}>
10:39:41.384858 IP 172.28.11.81 > 172.27.100.245: ICMP echo request, id 10002, seq 2, length 64
10:39:41.386501 IP 172.27.100.245 > 172.28.11.81: ICMP echo reply, id 10002, seq 2, length 64
10:39:42.384879 IP 172.28.11.81 > 172.27.100.245: ICMP echo request, id 10002, seq 3, length 64
10:39:42.386521 IP 172.27.100.245 > 172.28.11.81: ICMP echo reply, id 10002, seq 3, length 64
10:39:42.506806 IP 172.27.100.245.56565 > 172.24.0.160.microsoft-ds: . ack 1 win 63972 <nop,nop,sack 1 {0:1}>
10:39:43.340163 IP 172.27.100.245 > 172.24.136.149: ICMP echo reply, id 29, seq 35061, length 40
10:39:43.360682 IP 172.27.100.245 > 172.24.136.149: ICMP echo reply, id 29, seq 35086, length 40
10:39:43.384916 IP 172.28.11.81 > 172.27.100.245: ICMP echo request, id 10002, seq 4, length 64
10:39:43.386552 IP 172.27.100.245 > 172.28.11.81: ICMP echo reply, id 10002, seq 4, length 64

The ACI Fabric in conversional learning mode is confusing that one IP adress (the Firewall VIP Adress) comming in the Fabric with different MAC Adresses on different ports. Following the connections are disrupted. The ACI Fabric send packets return on sida a (Cluster member A) or side b (Cluster member B) to both firewalls (the active and the standby mamber) is like a slotmaschine.

Has the community any idee?

VMAC is'nt an option, there is also a problem. See
VMAC Mode on R80.10
from
Alexander_Wilke

[Boris_Karnaukh]

As I understand, here problem is that you initiate LDAP traffic from standby node and this traffic comes out with VIP of VSX cluster.

I can see a number of workarounds for this specific problem:

1. You can deploy CheckPoint Identity Collector within your network so that Identity Collector will initiate connection to firewall cluster members. You can either make it work with each cluster member independently. If you have multiple  firewalls with Identity Awareness, Identity Collector deployment is advised by best practices guide (see sk88520).
2. You can tinker with table.def file on SmartCenter and make LDAP or LDAP-GC traffic part of no_hide_services_ports table as described in sk31832.

[Uwe_Knoetsch]

Thank you for the suggestion, first we try to use the sk94564. "ARP Requests sent from the Standby Virtual System cause switch to send traffic to the Standby Virtual System" I hope this is a good way to correct the problem with the ldap request.

Regards Uwe

[Wolfgang]

These are known problems with CISCOs ACI. 

Maybe ou can follow ClusterXL/VRRP Standby member communication issues when connected to a Cisco ACI and disable "Endpoint Dataplane Learning"

If you enable vmac mode you run into Traffic sent by ClusterXL configured in VMAC mode does not reach its destination when it passes thro...

sk article is for ACE LoadBalancer but also true for ACI.

It is a real problem with some network devices having different MACs for same IP for sending and receiving traffic...

Wolfgang

PS.: Did you configure LDAP for shared or private use? Meaning that the particular VS did the LDAP connection neither the VS0. https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Uwe_Knoetsch]

Hello Wolfgang,

thank you for the suggestion, first we try to use the sk94564. "ARP Requests sent from the Standby Virtual System cause switch to send traffic to the Standby Virtual System" I hope this is a good way to correct the problem with the ldap request. And then we wait of response from Cisco TAC, I hope there is also a solution inside the ACI.

Regards uwe

[Diego_Alvarez1]

Hello uwe,

Did you received any answer from Cisco TAC about this?

We really appreciate if you can share it with us.

Regards

[Uwe_Knoetsch]

Hi all,

yes, the problem between a vsx cluster (in cluster XL Mode) and the Cisco ACI is solved with th sk94564.

After I has configure the parameters inside the fwkern.conf file and I reboot the cluster member, all is fine.

Here the parameters:
fwha_arp_forward_standby=1
fwha_resend_arp_unicast=1

I can make a ICMP request (ping) from standby member and the request don't go directly in the network behind teh firewall. The request go to the active member and then in the ACI network. ACI sends the reply packet to the active member and the active member to the standby member. And the best thing is, the ACI isn't confused.

Thanks the support from our partner BRISTOL for the solution.

Regards Uwe

@ Checkpoint: please set a tag (m.e. Cisco ACI-VSX Cluster XL) on the sk94564 for this solution.

[Uwe_Knoetsch]

Hi all,

yes, the solution for this problem is sk94564.
(communication between VSX Cluster with ClusterXL and Cisco ACI)

After I have configured the following 2 lines in the fwkern.conf file and I reboot the cluster.

fwha_arp_forward_standby=1
fwha_resend_arp_unicast=1

Finaly, the communication are go out from standby member (I try it with a ping from standby member) to the active member and then out to the ACI-System.

The answer for this request going to the active member and then to the standby member. The rsult is the ACI Switch isn't confused. All is fine.

Thanks for the solution to our partner BRISTOL.

Regards Uwe